Retrieving Data

Retrieving Data#

Not all data dependencies are shipped with the git repository, since git is not suited for handling large changing files. Instead we use separate steps in the workflow (rules executed by snakemake) to download external data using the retrieve_<dataset> rules.

Data is generally retrieved in a version-controlled manner, enabling control over input data versions, reproducibility and consistency of modelling runs. The rules download data into subfolders in the data/ directory, following the structure data/{dataset}/{source}/{version}, e.g. data/jrc_idees/primary/March-2025-V1/. Which specific data version is retrieve can be controlled in the data configuration .

Below some specific retrieve_<dataset> rules are documented. For more information on the datasets retrieved, see the data sources and Data inventory section there in the documentation.

Rule retrieve_bidding_zones#

Retrieves bidding zone shape files from two sources. electricitymaps-contrib provides shape files for all the zones on a global level. entsoe-py provides country level shape files which are concatenated into one file. The electricitymaps-contrib data is preferred, but the Italian bidding zones from entsoe-py are more accurate.

Outputs#

  • data/busshapes/bidding_zones_electricitymaps.geojson:

  • data/busshapes/bidding_zones_entsoepy.geojson:

Rule retrieve_cutout#

https://zenodo.org/badge/DOI/10.5281/zenodo.6382570.svg

Cutouts are spatio-temporal subsets of the European weather data from the ECMWF ERA5 reanalysis dataset and the CMSAF SARAH-3 solar surface radiation dataset for the year 2013, 2019 or 2023. They have been prepared by and are for use with the atlite tool. You can either generate them yourself using the build_cutouts rule or retrieve them directly from zenodo through the rule retrieve_cutout. The Tutorial: Electricity-Only uses a smaller cutout than required for the full model (30 MB), which is also automatically downloaded.

Note

To download cutouts yourself from the ECMWF ERA5 you need to set up the CDS API.

Relevant Settings

tutorial:
enable:
    build_cutout:

See also

Documentation of the configuration file config/config.yaml at toplevel_cf

Outputs

  • cutouts/{cutout}: weather data from either the ERA5 reanalysis weather dataset and/or SARAH-3 satellite-based historic weather data.

See also

For details see build_cutout and read the atlite documentation.

Rule retrieve_cost_data#

This rule downloads techno-economic assumptions from the technology-data repository.

Relevant Settings

costs:
    year:

See also

Documentation of the configuration file config/config.yaml at costs

Outputs

  • data/costs/primary/{version}/costs_{year}.csv

Contents