Most economic, social, and environmental situations and developments have a specific territorial dimension. When a statistic is computed to describe a population parameter, the computation is often carried out for a population within the boundaries of this territory.
Probably the most widely used and cited statistics are defined within the scope of national boundaries. The population of France relates to the counting of people living within the boundaries of France (unless we specifically count French citizens living outside these boundaries.)
Most national boundaries are stable, often throughout centuries. The number of national boundaries is approaching 200, which means that global datasets have boundary changes almost every year somewhere. Just the current European Union has witnessed several such changes in the last decades with the unification of Germany, the breakup of current member states Czechia and Slovakia, and the breakup of the former Yugoslavia to several current and prospective member states. When we are working with long-term data panels, obviously we have to make sure that we are comparing Czechia, Slovakia correctly with Czechoslovakia.
National boundary changes require complicated diplomatic negotiations and they are meant to be difficult, but changes of boundaries within a national boundary, i.e. the changes of sub-national or regional boundaries is very frequent. In fact, it is so frequent that there is no consistent way of describing them globally. Probably the most comprehensive international system, the NUTS of the European Union contains dozens of changes every three years. It describes currently 1845 territorial units, but far more are being used within the 27 member states of the EU. [The NUTS is de facto used in a number of non-EU member states, too, and much of these uses will be part of the NUTS2021 definition.]
Unless we work with shapefiles that contain the exact boundaries of territorial units, and we have access to microdata to replicate the statistical computation over varying boundaries, for example, before and after a regional border change, we cannot be sure that we are comparing exactly the same thing. The population of a region may have increased because more people were born or moved there than those who moved away or died, or simply because the boundaries were extended and new people who were earlier assigned to a different region became part of the count.
To make sure that we are working with the same territorial units, i.e. countries, provinces, (member or federal) states, administrative units, we must unambiguously attribute observation of our statistics to a territorial unit. This requires the use of controlled vocabularies as geographical identifiers.
Geographical names are not particularly useful identifiers in international datasets, because the names are subject to many changes in term of spelling, renaming, character codes or natural language that describes them and so on. The geographical names of the European Union use various extensions of the Latin alphabet, furthermore Cyril and Greek characters. In global datasets, we must deal with a lot more characters.
A machine-readable, controlled vocabulary, the geocode is used to describe the geographical territories with Latin alphanumerical combinations that are usually easy to use programmatically, too, as variable names.
The ISO standard for geographical territories is the
ISO-3166-1 standard for national boundaries and
ISO-3166-2 standard for sub-national divisions, such a
province in the Netherlands, states in the USA or Australia, or Land in
Germany. The ISO-3166-2 is rather often used for
statistical purposes; however, it is mainly a public administration
metadata that is very impractical for statistical use. Its greatest
disadvantage is that it is not well versioned. Because of the very
frequent changes before in sub-national boundaries, we can almost never
be sure if a certain ISO-3166-2 refers to the same
boundary. Usually, it refers to a same legal entity, for example, a
county with its own local regulations, budgets and contracts;
municipalities however switch country borders. For legal purposes, the
ISO-3166-2 is good. For one-time, cross sectional
comparison of data, the ISO-3166-2 is usable. For any
statistical computation that is periodically repeated, it is almost
useless.
The European Union uses its own nomenclature, the NUTS,
whose geocodes, and geographical names are controlled vocabularies with
about new editions every three years. When boundaries or names are
changing, usually the geocode is also changing for disambiguation.
Furthermore, the NUTS is a hierarchical system, which is very helpful
for missing data imputation. It is a good system to join statistical
data tables across time, for example, to compare the regional employment
data of the European Union in 2010 and 2020. Yet it is surprisingly
difficult to use.
A typical workflow is this:
Validate the geographical metadata (for which periods do you have data where you know which the territory was used for the statistical computation?)
Filter unchanged regions – these can be joined from data tables.
Recode regions that whose boundaries did not change to one single
controlled vocabulary, such as the current, 2016 edition of
NUTS; failing to do so will result in smaller data tables,
and loss of observations that were recoded.
Aggregate and disaggregate unambiguous changes to the units in
the NUTS2016 – in these cases all the statistical
information is present in different form, and any other imputation
method would result in loss of existing data.
Impute or estimate other regions, using the structural information about the remaining data. This will result in far more superior data than imputation techniques designed for non-aggregated, individual data points; usually a large part of the information is present in statistical tables that can be used for proximation.
When you are working with individual data, you are omitting steps 2-4. When working with national data, especially with small datasets, we can often join data tables by a country name or country code without the problems of 2-4, but even in these cases, imputation rules like using the median value for missing cases is obviously wrong. If the missing value represents a very small member state like Malta, which has a total population smaller than most capital cities in Europe, then using a median or mean country value will very much overestimate a count data for Malta.
There are two main international efforts are taking place to
standardize sub-national statistics: on the level of the European Union
and on the level of the OECD. Our programmatic solution aims to work
with both international data sources, and as much as possible, with any
country’s data that uses the ISO-3166-2 standard. However,
at this stage we have only included the typology used by the European
Union in helper functions and metadata (typological vocabulary)
files.
The European Union harmonises the national statistical systems of 27 members states, 4 EEA countries and several other potential member states. Its regional typology, the hierarchical NUTS system is the most complex international system, and the regions package is mainly developed to help working with European regional data. However, the tasks related to such regional typologies are not unique the European Union. They are present in almost any national statistical system, and they can be similarly complex in federal states like the United States or Australia.
The Methodological manual on territorial typologies 2018 edition introduces three types of typologies.
Regional typologies on NUTS1 level (states of Germany, provinces,
macroregions), NUTS2 level (regions) and NUTS3
level (small regions.) Similar typologies exist in the United States or
Australia.
Local Administrative Units (LAUs) which exist in almost all nation states that are sufficiently large, and have a territorially divided public administration and statistical system.
Grid typologies which are beyond the scope of
regions.
The availability of the sub-national statistical products is lower in terms of data table completeness or timeliness. This is often due to the fact that the creation of sub-national statistical products poses significant methodological challenges.
The changes of national or provincial boundaries are rare – within a single polity. However, when we work with global data panels, or data panels of many sub-national territorial units, boundary changes are rather frequent. On a national level, just within Europe we have witnessed in the last three decades the dissolution of federal states like the Soviet Union, Yugoslavia and Czechoslovakia and the unification of Germany. Especially the break-up of the former Yugoslavia took a long time, meaning that annual national statistics contain different configurations over the years for Serbia, Montenegro and Kosovo.
On sub-national levels, within Europe regional boundary changes are so frequent that they happen on average every three years. Of course, at one time, this procedure leaves most regional boundaries intact, and change only a relatively small number of boundaries. Whenever we work with European regional data, we still have to adjust panels of statistical aggregates every three years; similarly, we must validate any regional time series every three years.
In either case, producers of statistical products do not necessarily
re-cast historical statistical data for the new national or sub-national
boundaries. For example, when the regional boundary definition within
Central Hungary (HU1) changed in 2016, Eurostat did not
require the Hungarian national statistical authorities to re-cast
Hungary’s NUTS2 level data according to the new boundary
definition, i.e. creating separate observations for HU10
(Budapest) and HU12 (surrounding Pest county.)
Earlier HU101 (Budapest) on NUTS3
level became a NUTS2 region under the same name.
Furthermore, the former NUTS2 region
Közép-Magyarország or Central Hungary still exists on
NUTS1 level under the code HU1. Although you
will not find a history for the HU10 code before 2017, in
most cases, the very same data about Budapest can be found under the
code HU101 in NUTS3 datasets, or from national
statistical sources that follow Hungary’s ISO-3166-2
borders.
To put it another way, the HU-BP ISO-3166-2 code,
Budapest, did not change, it was just reassigned from the
NUTS3 typology to the NUTS2 typology, which is
reflecting the size of Budapest better. Budapest can be better compared
with NUTS2 regions than NUTS3 small
regions.
A very important and basic task is the validation of our data: which
data is valid in a certain typology? For example, if we have a data
source from 2014 that describes data within Europe in 2014 and we
currently download data from the Eurostat website in 2020, we may find a
discrepancy in how the regions are described. The dataset from 2014 is
likely to represent sub-national territories according to their
NUTS2013 boundary definition, and the new dataset according
to the currently valid NUTS2016 boundary definition. We may
not be able to correctly join the two datasets.
The validate function family helps in finding out to
conformity of your regional coding with various typologies. In the first
CRAN release, only Eurostat typologies are coded, but some of the
functions can be used with other typologies, for example, with the
ISO-3166-1 and ISO-3166-2 typologies. OECD
uses the NUTS typologies for the EU member states, so these typologies
are the most frequently used in international statistics.
Eurostat offers so-called correspondence tables to follow boundary changes, recoding and relabelling for all NUTS changes since the formalization of the NUTS typology. Unfortunately, these Excel tables do not conform with the requirements of tidy data, and their vocabulary for is not standardized, either. For example, recoding changes are often labelled as recoding, recoding and renaming, code change, Code change, etc.
The data-raw library contains these Excel tables and
very long data wrangling code that unifies the relevant vocabulary of
these Excel files and brings the tables into a single, tidy format ,
starting with the definition NUTS1999. The resulting data
file nuts_changes is included in the regions
package. It already contains the changes that will come into force in
2021.
The vignette Validating Your Typology gives a real-life example to the use of these functions. Validations are the starting points for Recoding and re-labelling, Aggregation And Re-Aggregation and Disaggregation And Projection operations.
In the European Union, the most frequent regional statistical change
is recoding and relabelling. Such changes do
not affect the statistical data, only the statistical metadata.
For example, the overseas department and region of France,
Réunion, was officially renamed La Réunion in 2010:
this is relabelling. In earlier sources, you may find data
about this region under the identifier Réunion, and later you
may find it extended with the French article La
Réunion. But the data from Réunion in 2009 and
La Réunion in 2012 refers to the same territory.
Furthermore, when France changed its administrative boundaries, it
changed their official code abbreviations. Although the typology of
La Réunion did not change, to indicate that a dataset is
comparable across French departments according to the new administrative
borders (which did change in many other regions) the geolocational code
of this region changed from FR94 to FRA4 in
2013 and to FRY4 in 2016. This is
recoding.
The geocodes of Eurostat are elements of a
controlled metadata vocabulary. To avoid confusion, when a
part of the vocabulary changes, for example, when France is changing
some elements in its NUTS regions and their coding, it often changes the
complete vocabulary, including for unchanged elements, for clarity. This
is why FR94 becomes FRA4 even though it is
describing the same overseas department and region,
La Réunion.
If you want to work with French or international data that include
any statistical information about the overseas departments, you must
make sure that observations coded with FR94,
FRA4, FRY4 are treated as a time-series of the
La Réunion and not as different entities.
Names are often dubious identifiers. La Réunion remains in
the ISO-3166-1 and ISO-3166-2 libraries
Réunion, which is no longer used as name in NUTS.
What makes working with ISO and NUTS
vocabularies is that while NUTS is “versioned”, ISO is not, so a
correspondence for a given year, for example, for the year 2013 is
almost impossible to create. This is why the use of Eurostats NUTS geo
codes, which are usually unique in a given year, should be preferred
over ISO.
Many of Eurostat’s regional datasets contain recoded geographical IDs
that violate the definition of tidy data, because under the same
geo column they may contain observations for
FR94, FRA4, FRY4. However, this
is not a tidy description of the territory. Like in our
nuts_changes data, they should be reported in three
different columns, code_2010, code_2013 and
code_2016. The descriptive metadata code, i.e. row
identifier FR94 is compatible with FR71 for
example, but not with FRY4.
If you are joining data from different tables, you must not join by
their geo code, but by the explicit code_2013
geo codes for the observation described or aggregated according to the
NUTS2013 regional boundaries and code_2016 for
the NUTS2016. In other words, you must make sure that you
join by valid elements in a given version of NUTS.
Another important note: if you join with the raw geo
data, you will often have seemingly missing data. You
may have data available for FR93, FR94,
FRY3, FRY4, with missingness for the first two
in 2016 and missingness for the latter two in 2010, but in fact, these
observations or related to two, unchanged territorial units
(Guyane and La Réunion.) You must choose one
consistent coding, either using only FRY4 for La
Réunion or FR93 for La Réunion. You can also
identify FR93, FRY3 explicitly with the name
Guyane. However, you must choose between the use of La
Réunion or Réunion.
data(nuts_changes)
overseas <- nuts_changes[
which(nuts_changes$geo_name_2016 %in% c("La Réunion", "Guyane")),
]
knitr::kable(
overseas[, c("code_2010", "code_2013", "code_2016",
"geo_name_2010", "geo_name_2013", "geo_name_2016")]
)| code_2010 | code_2013 | code_2016 | geo_name_2010 | geo_name_2013 | geo_name_2016 |
|---|---|---|---|---|---|
| FR93 | FRA3 | FRY3 | Guyane | Guyane | Guyane |
| FR94 | FRA4 | FRY4 | Réunion | La Réunion | La Réunion |
| FR930 | FRA30 | FRY30 | Guyane | Guyane | Guyane |
| FR940 | FRA40 | FRY40 | Réunion | La Réunion | La Réunion |
This is in fact a simpler recoding than the earlier example of
Budapest. La Réunion did change its name and code, so
you find its data under different labels, but it is still within the
NUTS2 typology. Its comparators are likely to be found in
the same statistical table. In the case of Budapest, the name
did not change, but the change of the geo label reflects the fact that
Budapest was moved from the NUTS3 typology to the
NUTS2 typology, because it can be much better compared with
NUTS2 regions. For data concerning the time period before
2017, you need to look for data about Budapest in datasets that follow
the NUTS3 typology, or in national data sources, but you
can fill in these historical data to pre-2017 cells of your
NUTS2 regional data. The size of Budapest did not change
that much - it should have been compared with NUTS2 regions
in 2014, like in 2020, and not with smaller NUTS3
units.
See in more detail the vignette Recoding & Relabelling for our programmatic solution.
R has many packages that implement imputation algorithms designed for non-aggregated, individual data. In almost all the cases, these methods are not adequate for territorially aggregated data. With individual data imputation, we usually take a typical individual data point (such as the median value) or hypothesize a probability distribution of unobserved data and pick a hypothetically likely observation. We must never use such methods when data among observations is differently aggregated. Furthermore, very often some elements of the information, or all the information is present in the dataset, and a far less biased estimate can be given to the missing data.
Often, we have missing data about smaller territorial units, and we
would like to impute their wider region’s representative data to a
higher density dataset. For example, we do not have data for
Schwabia, which is a region of the German federal state of
Bavaria, and therefore we would like to impute the Bavarian
(NUTS1) level data to this NUTS2 level missing
observation for Schwabia. This is a valid imputation technique for
non-aggregating descriptive statistics, such as mean or median values
derived from surveys.
A typical problem of missingness is that we are comparing
NUTS1 or NUTS2 level regions of Europe, and we
do not find information about the small members states of Cyprus,
Estonia, Luxembourg, Malta, because, due to their size, these countries
do not have such divisions. Because of their size and territorial
integrity, they can be well compared with Schwabia, for example, but the
data is missing in a NUTS2 data table. In this case, we may
want to impute the technical NUTS0 (country) level data
into the NUTS2 data table, creating the technical
NUTS2 data of CY00, EE00,
LU00 or MT00, which is identical to the
national CY, EE, LU,
MT observations. In this case, the imputation is valid to
any type of data, even to count data like population, because
EE = (NUTS0) EE0
(NUTS1) = EE00
(NUTS2).
The function impute_down creates a generic solution to
the problem when we want to project data from a larger territorial unit
to a smaller one. For example, some state level data is missing for
certain Australian or German federal states, and we want to use the
national (average) values for imputation.
The function impute_down_nuts gives and EU-specific
solution, with a single data frame as input. The
mixed_nuts_example data table shows a mixture of potential
problems in a small, fictional dataset with random numbers. For
simplicity, we take the fictional, country-level data from Malta, and
input it down to all potential sub-divisions.
Malta is a very transparent example, because the country has only 2
subdivisions on the lowest, NUTS3 level,
i.e. MT001 and MT002. You can find Malta among
countries (technical NUTS0 level) as MT, among
NUTS1 regions as MT0 (because there are no
subdivisions on this level), and among NUTS2 regions again
as MT00. For any data that cannot be assigned for a
geographical location or area, the technical codes ending with
Z are used.
So, what are our possibilities if we have only country-level
MTdata, but we would like to use it in a NUTS3
database and impute this variable to both MT001 and
MT002. (You can safely do this with average values taken
from a survey, but you should not do this with additive values such as
the population.)
data("mixed_nuts_example")
mixed_nuts_example %>%
dplyr::filter(substr(geo,1,2) == "MT") %>%
knitr::kable()| geo | values | method |
|---|---|---|
| MT | 93.3514 | actual |
Malta offers two technical imputation slots on NUTS1
level, MT0 for territorially attributed data, and
MTZ for non-territorial exceptions. On NUTS2 level,
MT00 and MTZZ. And at last, what is a real
imputation, you can impute the MT = MT0 =
MT00 values to MT001, should you have only
data for MT002, etc. (In this simple example, we impute the
MT values to both subdivisions, MT001 and
MT002.)
data("mixed_nuts_example")
impute_down_nuts(mixed_nuts_example) %>%
dplyr::filter(substr(geo,1,2) == "MT") %>%
knitr::kable ()| geo | values | method | typology | valid_2016 |
|---|---|---|---|---|
| MT | 93.3514 | country | TRUE | |
| MT0 | 93.3514 | imputed from country MT | nuts_level_1 | TRUE |
| MT00 | 93.3514 | imputed from country MT | nuts_level_2 | TRUE |
| MT001 | 93.3514 | imputed from country MT | nuts_level_3 | TRUE |
| MT002 | 93.3514 | imputed from country MT | nuts_level_3 | TRUE |
| MTZ | 93.3514 | imputed from country MT | nuts_level_1 | TRUE |
| MTZZ | 93.3514 | imputed from country MT | nuts_level_2 | TRUE |
| MTZZZ | 93.3514 | imputed from country MT | nuts_level_3 | TRUE |
The European Union sometimes publishes data on NUTS2
level regions, but not on the larger NUTS1 level macroregions or
provinces. In many cases, this information is fully present in the
NUTS2 datasets, and a simple aggregation can produce a
NUTS1 level dataset, because each NUTS2 region is strictly
assigned to one broader NUTS1 macroregion. For count-type data, we can
simply aggregate the constituent NUTS2 level data to
NUTS1 level.
A frequent problem is the re-assignment of a smaller territorial units, such as lower level administrative unit or a lower NUTS unit to another larger unit, or a pre-existing larger unit.
For example, LT02, i.e. Vidurio ir vakarų Lietuvos
regionas in Lithuania consists of the former LT00
(undivided) Lithuania minus LT00A, i.e. Vilnius.
LT01, or Sostinės regionas, i.e. the metropolitan
area of Vilnius is a separate region from 2016. If you have
national and NUTS3 or Vilnius municipality level data from
Lithuania, you can present it according to the undivided
NUTS2013 definition or the more granular
NUTS2016 definition.
The Southern region of Ireland, IE05 was
created in the same year from the earlier NUTS3 level
regions IE023, IE024, and IE025.
You can fill in the missing data for earlier years with the simple
additive formula IE05=IE023+IE024+IE025 for count/additive
variables, or you can create a re-weighted average for average
values.
This information can be found in the change_2016 of the
nuts_changes dataset.
data(nuts_changes)
southern <- nuts_changes[
which(nuts_changes$geo_name_2016 == "Southern"), ]
knitr::kable(
southern[, c("code_2013", "code_2016", "geo_name_2016", "change_2016")]
)| code_2013 | code_2016 | geo_name_2016 | change_2016 |
|---|---|---|---|
| NA | IE05 | Southern | new region, made from ex-ie023, ie024 and ie025 |
In some cases the boundary change cuts across all territorial typologies that re-configuration is impossible from aggregated data. Unless we have access to the individual level data (which is not within scope of the tasks helped by this package), it is not possible to create time-wise comparable data for these units. We must treat the observations on these units logically missing, which may or may not be imputed with any techniques. It is usually helpful to mark territorial observations in panels or time series that have such a structural change.
Because of the complexity of sub-national data, there are likely to be missing elements in time-wise comparisons. For example, we may find that some data is available for many years on a German federal level, and for most states, but a particular annual data is missing for Bavaria. It would be tempting to use a simple interpolation of the available data, but often a naïve interpolation is not the best method, because, as we have seen, often the data is available at a different aggregation level or under a different metadata label. It is imperative to first make sure that we exploit all information that is present in a different territorial structure for any given year, and then start inter-temporal imputation with interpolation, or carry forward/backward, forecasting and backcasting algorithms.
At last, we may go back to the cross-sectional data, and may fill in
some cross-sectional data gaps with interpolated or extrapolated data.
Aggregating an estimate for Bavaria from previous or next-years
constituent lower level (NUTS2 or NUTS3) data
is very likely to be a far better estimate than any imputation results,
for example, a simple substitution of a median value from all German or
EU NUTS1 units. The territorial aggregation structure gives plenty of
exploitable information.
Our package uses a number of boundary definitions that are easily expanded. Our functions are generic, i.e. they treat boundary information as an input variable. Well formatted metadata about such typologies can be used with any country.
The Online Browsing Platform (OBP) of the International Organization for Standardization contains all coding information, and metadata about recent changes for each country. (See, for example: ISO 3166 — Codes for the representation of names of countries and their subdivisions: NL - Netherlands (the))
It would be tempting to use ISO-3166-2 definitions for
sub-national boundaries, however, the ISO-3166-2 itself is
changing relatively often, and in time series or panel data we can only
refer to a certain version of the ISO-3166-2. All the
boundary definitions are valid for a certain time period and certain
boundaries. We currently incorporated only European metadata for the
last 20 years, and we are planning to continue with OECD metadata in the
forthcoming releases. However, we are welcoming any pull requests for
other countries or country-groups.