General project configuration
Each project instance hosted in the Scenario Services infrastructure requires a configuration file with a standardized format.
The following features can be accessed via a general project configuration file:
Import codelists and mappings from public GitHub repositories
Add all countries to the region codelist (details: A common list of country names)
Setup processors for region aggregation
These settings are then read into the project’s workflow script for scenario submission.
Tip
To learn more about workflow scripts, see Scenario workflow processing.
Configuration file format
The configuration file must be located on the same level as the mappings and definitions directories.
The file must be in YAML format and named nomenclature.yaml.
Specify dimensions to be read
The configuration file offers the possibility to set the dimensions which will be read by DataStructureDefinition, overriding the dimensions from the definitions sub-directories.
If no sub-directories exist (e.g.: when importing dimensions from external repositories), setting dimensions in the configuration file is mandatory.
In the below case we specify region, variable and scenario to be read and used for validation:
dimensions:
- region
- variable
- scenario
Importing from an external repository
In order to import from an external repository, the configuration file must define the repository.
repositories:
common-definitions:
url: https://github.com/IAMconsortium/common-definitions.git/
legacy-definitions:
url: https://github.com/IAMconsortium/legacy-definitions.git/
The repository has a name (in this example, common-definitions) and a URL. Optionally, a commit hash can be defined to pin the repository to a specific version. Multiple repositories can be used in a single configuration file.
In order for the import to work URL url must be given in the above format, i.e. with the leading https:// and the trailing .git/.
Information from an external repository can either be used for codelists (“definitions”) or model mappings, or both. For each definition dimension, i.e. region or variable, multiple external repositories can be used as the example below illustrates for variable:
repositories:
common-definitions:
url: https://github.com/IAMconsortium/common-definitions.git/
legacy-definitions:
url: https://github.com/IAMconsortium/legacy-definitions.git/
definitions:
region:
repository: common-definitions
variable:
repositories:
- common-definitions
- legacy-definitions
mappings:
repository: common-definitions
The value in definitions.region.repository can be a list or a single value.
For model mappings the process is analogous using mappings.repository.
Filter codelists from external repositories
Since importing the entirety of, for example, common-definitions is too much
for most projects, the list can be filtered using include and exclude keywords.
Under these keywords, lists of filters can be given that will be applied to the
codelist from the given repository.
The filtering can be done by any attribute:
repositories:
common-definitions:
url: https://github.com/IAMconsortium/common-definitions.git/
definitions:
variable:
repository:
name: common-definitions
include:
- name: [Primary Energy*, Final Energy*]
- name: "Population*"
tier: 1
exclude:
- name: "Final Energy|Industry*"
depth: 2
If a filter is being used for repositories, the name attribute must be used for the repository.
In the example above we are including:
All variables starting with Primary Energy or Final Energy
All variables starting with Population and with the tier attribute equal to 1
From this list we are then excluding all variables that match “Final Energy|Industry*” and have a depth of 2 (meaning that they contain two pipe “|” characters).
Filter model mappings from external repositories
We often only want to use a subset of models in a particular project (and not import all mappings), so there is an option to filter for specific model mappings. This works very similarly to the filtering for definitions.
repositories:
common-definitions:
url: https://github.com/IAMconsortium/common-definitions.git/
mappings:
repository:
name: common-definitions
include:
- MESSAGEix-GLOBIOM 2.1-M-R12
The above example retrieves only the model mapping for MESSAGEix-GLOBIOM 2.1-M-R12 from the common-definitions repository.
Adding countries to the region codelist
By setting definitions.region.country as true in the configuration file:
definitions:
region:
country: true
the nomenclature package will add all countries to the region codelist.
More details on the list of countries can be found here: A common list of country names.
Adding NUTS to the region codelist
By setting definitions.region.nuts in the configuration file:
definitions:
region:
nuts:
nuts-1: [ AT, BE, CZ ]
nuts-2: [ AT ]
nuts-3: true
the nomenclature package will add the selected NUTS regions to the region codelist.
In the example above, the package will add: NUTS 1 regions for Austria, Belgium and Czechia, NUTS 2 regions for Austria, NUTS 3 regions for all EU countries.
More details on the list of NUTS regions can be found here: NUTS classification.
Configuring processors
The processors section of nomenclature.yaml allows processors to be declared
directly in the configuration file, so they are applied automatically when calling
process without passing an explicit processor argument.
Region processor
Setting processors.region-processor as true will automatically create a
RegionProcessor from the project’s default mappings/ directory:
processors:
region-processor: true
This is equivalent to calling:
import pyam
from nomenclature.processor import RegionProcessor
df = pyam.IamDataFrame(data="path/to/file.csv")
dsd = DataStructureDefinition("definitions")
processor = RegionProcessor.from_directory("mappings", dsd)
aggregated_data = process(df, dsd)
NUTS processor
Setting processors.nuts to a list of model names will automatically create a
NutsProcessor and apply NUTS hierarchical aggregation (NUTS3 → NUTS2 →
NUTS1 → Country → EU27) for those models:
processors:
nuts-processor: [ Model A, Model B ]
More details on NUTS aggregation can be found here: NUTS classification.
Country processor
Setting processors.country-processor to a list of model names will
automatically create a CountryProcessor and aggregate country-level data
to common regional definitions such as R5, R9 and R10 for those models:
repositories:
common-definitions:
url: https://github.com/IAMconsortium/common-definitions.git/
definitions:
region:
country: true
repository:
name: common-definitions
include:
- hierarchy: [R5, R9, R10]
processors:
country-processor: [ Model A, Model B ]
The country processor reads the imported R5/R9/R10 definitions from the region codelist and aggregates directly from countries to those regions. If one of those hierarchy levels is not present in the codelist, it is skipped.
Tip
More details on the country processor API can be found here.