# Configuration files ## Overview All the configuration for Autosubmit is provided as yaml configuration files (`*.yml` or `*.yaml`). Autosubmit parses these files and joins them to a common set that is used for a specific experiment. Some of the configuration is directly used by Autosubmit (e.g. which jobs to run where), other parts are passed to the [jobscripts](/Developers/Jobs.md#jobscript). In these jobscripts, Autosubmit placeholders will be substituted with the respective value of the configuration option. These placeholders are always encapsulated in percentage signs and support nested yaml sections, e.g. `%SEC.SUBSEC.PARAM%` will be replaced with the value of `PARAM` (`myval`), that is defined in subsection `SUBSEC` of section `SEC` (see [here](https://autosubmit.readthedocs.io/en/master/userguide/variables.html) for details): ```yaml SEC: SUBSEC: PARAM: 'myval' ``` As Autosubmit allows for a flexible, distributed configuration, the configuration for an _auto-icon_ experiment is distributed over many files. Following the general Autosubmit configuration structure, there are three/four places with configuration files, each one taking precedence over the preceding one: | File/location | Purpose | | ------------------------------- | ------- | | `/conf/minimal.yml` | sets a few configuration switches and general Autosubmit parameters. The user has to manipulate this file only in rare cases. | | `/proj/git_project/conf` | project subdirectory: **most of the project configuration, including templates for full experiments, is stored here** | | `/conf` | experiment-specific configuration | | `~/.asconf/platforms_user.yml` | User-specific platform configuration | :::{hint} Further, the [init script](/Introductory-guides/Step-by-step-guide.md#4-initialize-the-auto-icon-project) creates a directory `/` containing symlinks to all files relevant for most use cases, allowing you to have all your configuration in one place. ::: ### How to create your own configuration To create your own configuration for your specific experiment, you have two major options: 1. Create your own branch of _auto-icon_. In this branch, you can directly edit the config files, push changes and thus reuse your changes for several (similar) experiments. This has the additional advantage that changes there are tracked in the git repository. 2. Create configuration files in `/conf`. The entries in these files overwrite the values from the project directory and are thus favorable to be used when only a few changes have to be made for a specific experiment. These files can also be copied to a new experiment with Autosubmit directly, when creating a new experiment. Both methods can also be combined, e.g. if you have a set of similar experiments, you can create your own branch of _auto-icon_ with a default configuration to be used for all your experiments. You can then create several experiments from this branch and overwrite the necessary fields with values provided in files in `/conf`. ### Symlinks created by the init script The init script can create a couple of symlinks to the most used configuration files. These files will all be linked to the directory `/` and the links will be used for referencing in this documentation. The files are listed in the following table along with their link names and the command line options that have to be provided to create the link. | Target config file (relative to `/proj/git_project/conf/`) | Link name | Option | | ------------------------------------------- | ------------------------------- | ------ | | `.yml` | `.yml` | | | `/experiments/.yml` | `experiment.yml` | | | `/visualization/.yml` | `visualization.yml` | `-v` | | `common/build.yml` | `build.yml` | `-b` | | `common/jobs/wrappers.yml` | `wrappers.yml` | `-w` | | `common/jobs/prep-remap-ifs2icon.yml` | `prep-remap-ifs2icon.yml` | `--ifs2icon` | | `common/jobs/prep-remap-era2icon.yml` | `prep-remap-era2icon.yml` | `--era2icon` | | `common/jobs/prep-remap-era2icon-chunk.yml` | `prep-remap-era2icon-chunk.yml` | `--era2icon --rl chunk` | Further, for the art case you have the namelist (with suffix `nml` or `yml`) | Target namelist file (relative to `/proj/git_project/`) | Link name | | ---------------------------------- | ------------------- | | `namelists/art/.` | `namelist.` | and for the seamless case you have the namelists (with suffix `nml` or `yml`) | Target namelist file (relative to `/proj/git_project/`) | Link name | | ------------------------------------------- | ------------------ | | `namelists/seamless/_atm.` | `nml_atm.` | | `namelists/seamless/_lnd.` | `nml_lnd.` | | `namelists/seamless/_oce.` | `nml_oce.` | | `namelists/seamless/_cpl.yml` | `nml_cpl.yml` | ## Detailed description ### Minimal config The file `/conf/minimal.yml` defines the minimal required configuration for an Autosubmit experiment. General Autosubmit parameters are given, like the project type (_auto-icon_) and URL and Autosubmit version. Additionally, some basic configuration entries are provided like the __expid__, the __default HPC architecture__ and where __custom configuration files__ are located. ### project conf subdirectory This directory (`conf`, usually in `proj/git_project`) contains most of the configuration for the experiment. It is split in further subdirectories: ``` %PROJDIR%/conf/art -----> ICON case 'art' %PROJDIR%/conf/build_only -----> ICON case 'build_only' %PROJDIR%/conf/common -----> common files for all use cases %PROJDIR%/conf/real-from-dwd-ana -----> ICON case 'real-from-dwd-ana' %PROJDIR%/conf/real-from-ideal -----> ICON case 'real-from-ideal' %PROJDIR%/conf/seamless -----> ICON case 'seamless' ``` For each directory, there is an include file (`%PROJDIR%/conf/.yml`), that is specified in the `minimal.yml` file and includes all further required configuration files. #### common Basic configuration for general tasks of the workflow. Most general configuration for ICON-ART workflows is included here. Important files here are ``` %PROJDIR%/conf/common/build.yml -----> Options for building the ICON model and Python environment %PROJDIR%/conf/common/platforms.yml -----> Configuration of the (HPC) platforms %PROJDIR%/conf/common/platforms/.yml -----> Platform-specific configuration (e.g. pool directories, module names, ...) ``` #### ICON cases There are different ICON use cases (`ICON_CASE`) defined: - art - real-from-ideal - real-from-dwd-ana - build_only - seamless which are more or less self-explanatory. Each use case has its own subdirectory with specific configurations. The `art` case is specialized for real data runs with ICON and ICON-ART but is versatile enough to also handle most such cases. It is the most general use case and thus recommended. The `seamless` case is a very similar case but includes coupling to the land surface scheme JSBACH and optionally the ocean. :::{hint} For most applications, the versatile `art` case is best or - if you need coupling to the ocean - the `seamless` case. ::: For the `art` case, there are two additional configuration files (and likewise for the `seamless` case): ``` %PROJDIR%/conf/art/experiments/.yml %PROJDIR%/conf/art/simulation.yml ``` The `experiments/.yml` file (multiple ones for the different templates exist) contains all the experiment-specific configuration. The `simulation.yml` configuration holds some basic keys necessary for running an ART use case. Usually, it does not have to be modified by the user. :::{hint} Most of your configuration goes into the `experiments/.yml` file. ::: :::{seealso} [Important config options](/Userguide/Important-configuration.md) ::: ### User specific configuration This file contains some values that are specific to the user but not the experiment. It is intended to provide user and project information for the HPC systems. For this, it has to contain a section `PLATFORMS` and in this section a section for each HPC system in use with the `USER` and `PROJECT` keys. An example for it is the `examples/platforms_user.yml` file. We suggest to use this and modify it. On Levante, the `SCRATCH_PROJECT_DIR` field needs to be set to the first letter of your user/project (the one in `/home//`). :::{hint} Per default it is located in `~/.asconf/platform_user.yml`. Instead, the full path can be given in `conf/.yml` in the `DEFAULT.CUSTOM_CONFIG.POST` section. :::