.. _nomenclature-config: General project configuration ============================= Each project instance hosted in the Scenario Services infrastructure requires a configuration file with a standardized format. The following features can be accessed via a general project configuration file: - Import codelists and mappings from public GitHub repositories - Add all countries to the region codelist (details: :ref:`countries`) - Setup processors for region aggregation These settings are then read into the project's workflow script for scenario submission. .. tip:: To learn more about workflow scripts, see :ref:`project-workflow`. Configuration file format ------------------------- The configuration file **must** be located on the same level as the *mappings* and *definitions* directories. The file **must** be in YAML format and named *nomenclature.yaml*. Specify dimensions to be read ----------------------------- The configuration file offers the possibility to set the dimensions which will be read by *DataStructureDefinition*, overriding the dimensions from the *definitions* sub-directories. If no sub-directories exist (e.g.: when importing dimensions from external repositories), setting dimensions in the configuration file is **mandatory**. In the below case we specify *region*, *variable* and *scenario* to be read and used for validation: .. code:: yaml dimensions: - region - variable - scenario .. _external-repos: Importing from an external repository ------------------------------------- In order to import from an external repository, the configuration file must define the repository. .. code:: yaml repositories: common-definitions: url: https://github.com/IAMconsortium/common-definitions.git/ legacy-definitions: url: https://github.com/IAMconsortium/legacy-definitions.git/ The repository has a **name** (in this example, *common-definitions*) and a **URL**. Optionally, a commit **hash** can be defined to pin the repository to a specific version. Multiple repositories can be used in a single configuration file. In order for the import to work URL url must be given in the above format, i.e. with the leading *https://* and the trailing *.git/*. Information from an external repository can either be used for codelists ("definitions") or model mappings, or both. For each definition dimension, i.e. *region* or *variable*, multiple external repositories can be used as the example below illustrates for *variable*: .. code:: yaml repositories: common-definitions: url: https://github.com/IAMconsortium/common-definitions.git/ legacy-definitions: url: https://github.com/IAMconsortium/legacy-definitions.git/ definitions: region: repository: common-definitions variable: repositories: - common-definitions - legacy-definitions mappings: repository: common-definitions The value in *definitions.region.repository* can be a list or a single value. For model mappings the process is analogous using *mappings.repository*. Filter codelists from external repositories ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Since importing the entirety of, for example, *common-definitions* is too much for most projects, the list can be filtered using ``include`` and ``exclude`` keywords. Under these keywords, lists of filters can be given that will be applied to the codelist from the given repository. The filtering can be done by any attribute: .. code:: yaml repositories: common-definitions: url: https://github.com/IAMconsortium/common-definitions.git/ definitions: variable: repository: name: common-definitions include: - name: [Primary Energy*, Final Energy*] - name: "Population*" tier: 1 exclude: - name: "Final Energy|Industry*" depth: 2 If a filter is being used for repositories, the *name* attribute **must** be used for the repository. In the example above we are including: 1. All variables starting with *Primary Energy* or *Final Energy* 2. All variables starting with *Population* **and** with the tier attribute equal to 1 From this list we are then **excluding** all variables that match "Final Energy|Industry\*" and have a depth of 2 (meaning that they contain two pipe "|" characters). Filter model mappings from external repositories ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ We often only want to use a subset of models in a particular project (and not import all mappings), so there is an option to filter for specific model mappings. This works very similarly to the filtering for definitions. .. code:: yaml repositories: common-definitions: url: https://github.com/IAMconsortium/common-definitions.git/ mappings: repository: name: common-definitions include: - MESSAGEix-GLOBIOM 2.1-M-R12 The above example retrieves only the model mapping for *MESSAGEix-GLOBIOM 2.1-M-R12* from the common-definitions repository. Adding countries to the region codelist --------------------------------------- By setting *definitions.region.country* as *true* in the configuration file: .. code:: yaml definitions: region: country: true the nomenclature package will add all countries to the *region* codelist. More details on the list of countries can be found here: :ref:`countries`. .. _adding-countries: Adding NUTS to the region codelist ---------------------------------- By setting *definitions.region.nuts* in the configuration file: .. code:: yaml definitions: region: nuts: nuts-1: [ AT, BE, CZ ] nuts-2: [ AT ] nuts-3: true the nomenclature package will add the selected NUTS regions to the *region* codelist. In the example above, the package will add: NUTS 1 regions for Austria, Belgium and Czechia, NUTS 2 regions for Austria, NUTS 3 regions for all EU countries. More details on the list of NUTS regions can be found here: :ref:`nuts`. Configuring processors ---------------------- The ``processors`` section of *nomenclature.yaml* allows processors to be declared directly in the configuration file, so they are applied automatically when calling ``process`` without passing an explicit ``processor`` argument. Region processor ^^^^^^^^^^^^^^^^ Setting *processors.region-processor* as *true* will automatically create a ``RegionProcessor`` from the project's default ``mappings/`` directory: .. code:: yaml processors: region-processor: true This is equivalent to calling: .. code:: python import pyam from nomenclature.processor import RegionProcessor df = pyam.IamDataFrame(data="path/to/file.csv") dsd = DataStructureDefinition("definitions") processor = RegionProcessor.from_directory("mappings", dsd) aggregated_data = process(df, dsd) NUTS processor ^^^^^^^^^^^^^^ Setting *processors.nuts* to a list of model names will automatically create a ``NutsProcessor`` and apply NUTS hierarchical aggregation (NUTS3 → NUTS2 → NUTS1 → Country → EU27) for those models: .. code:: yaml processors: nuts-processor: [ Model A, Model B ] More details on NUTS aggregation can be found here: :ref:`nuts`. Country processor ^^^^^^^^^^^^^^^^^ Setting *processors.country-processor* to a list of model names will automatically create a ``CountryProcessor`` and aggregate country-level data to common regional definitions such as R5, R9 and R10 for those models: .. code:: yaml repositories: common-definitions: url: https://github.com/IAMconsortium/common-definitions.git/ definitions: region: country: true repository: name: common-definitions include: - hierarchy: [R5, R9, R10] processors: country-processor: [ Model A, Model B ] The country processor reads the imported R5/R9/R10 definitions from the region codelist and aggregates directly from countries to those regions. If one of those hierarchy levels is not present in the codelist, it is skipped. .. tip:: More details on the country processor API can be found `here`_. .. _here: https://nomenclature-iamc.readthedocs.io/en/stable/api/regionprocessor.html#creating-a-country-to-region-processor