Configuration files

Overview

All the configuration for Autosubmit is provided as yaml configuration files (*.yml or *.yaml). Autosubmit parses these files and joins them to a common set that is used for a specific experiment. Some of the configuration is directly used by Autosubmit (e.g. which jobs to run where), other parts are passed to the jobscripts. In these jobscripts, Autosubmit placeholders will be substituted with the respective value of the configuration option. These placeholders are always encapsulated in percentage signs and support nested yaml sections, e.g. %SEC.SUBSEC.PARAM% will be replaced with the value of PARAM (myval), that is defined in subsection SUBSEC of section SEC (see here for details):

SEC:
  SUBSEC:
    PARAM: 'myval'

As Autosubmit allows for a flexible, distributed configuration, the configuration for an auto-icon experiment is distributed over many files. Following the general Autosubmit configuration structure, there are three/four places with configuration files, each one taking precedence over the preceding one:

File/location	Purpose
<expid>/conf/minimal.yml	sets a few configuration switches and general Autosubmit parameters. The user has to manipulate this file only in rare cases.
<expid>/proj/git_project/conf	project subdirectory: most of the project configuration, including templates for full experiments, is stored here
<expid>/conf	experiment-specific configuration
~/.asconf/platforms_user.yml	User-specific platform configuration

Hint

Further, the init script creates a directory <expid>/<expname> containing symlinks to all files relevant for most use cases, allowing you to have all your configuration in one place.

How to create your own configuration

To create your own configuration for your specific experiment, you have two major options:

1. Create your own branch of auto-icon. In this branch, you can directly edit the config files, push changes and thus reuse your changes for several (similar) experiments. This has the additional advantage that changes there are tracked in the git repository.

2. Create configuration files in <expid>/conf. The entries in these files overwrite the values from the project directory and are thus favorable to be used when only a few changes have to be made for a specific experiment. These files can also be copied to a new experiment with Autosubmit directly, when creating a new experiment.

Both methods can also be combined, e.g. if you have a set of similar experiments, you can create your own branch of auto-icon with a default configuration to be used for all your experiments. You can then create several experiments from this branch and overwrite the necessary fields with values provided in files in <expid>/conf.

Symlinks created by the init script

The init script can create a couple of symlinks to the most used configuration files. These files will all be linked to the directory <expid>/<expname> and the links will be used for referencing in this documentation. The files are listed in the following table along with their link names and the command line options that have to be provided to create the link.

Target config file (relative to <expid>/proj/git_project/conf/)	Link name	Option
<icon_case>.yml	<icon_case>.yml
<icon_case>/experiments/<expname>.yml	experiment.yml
<icon_case>/visualization/<expname>.yml	visualization.yml	-v
common/build.yml	build.yml	-b
common/jobs/wrappers.yml	wrappers.yml	-w
common/jobs/prep-remap-ifs2icon.yml	prep-remap-ifs2icon.yml	--ifs2icon
common/jobs/prep-remap-era2icon.yml	prep-remap-era2icon.yml	--era2icon
common/jobs/prep-remap-era2icon-chunk.yml	prep-remap-era2icon-chunk.yml	--era2icon --rl chunk

Further, for the art case you have the namelist (with suffix nml or yml)

Target namelist file (relative to <expid>/proj/git_project/)	Link name
namelists/art/<expname>.<suffix>	namelist.<suffix>

and for the seamless case you have the namelists (with suffix nml or yml)

Target namelist file (relative to <expid>/proj/git_project/)	Link name
namelists/seamless/<expname>_atm.<suffix>	nml_atm.<suffix>
namelists/seamless/<expname>_lnd.<suffix>	nml_lnd.<suffix>
namelists/seamless/<expname>_oce.<suffix>	nml_oce.<suffix>
namelists/seamless/<expname>_cpl.yml	nml_cpl.yml

Detailed description

Minimal config

The file <expid>/conf/minimal.yml defines the minimal required configuration for an Autosubmit experiment. General Autosubmit parameters are given, like the project type (auto-icon) and URL and Autosubmit version. Additionally, some basic configuration entries are provided like the expid, the default HPC architecture and where custom configuration files are located.

project conf subdirectory

This directory (conf, usually in proj/git_project) contains most of the configuration for the experiment. It is split in further subdirectories:

%PROJDIR%/conf/art                   -----> ICON case 'art'
%PROJDIR%/conf/build_only            -----> ICON case 'build_only'
%PROJDIR%/conf/common                -----> common files for all use cases
%PROJDIR%/conf/real-from-dwd-ana     -----> ICON case 'real-from-dwd-ana'
%PROJDIR%/conf/real-from-ideal       -----> ICON case 'real-from-ideal'
%PROJDIR%/conf/seamless              -----> ICON case 'seamless'

For each directory, there is an include file (%PROJDIR%/conf/<icon_case>.yml), that is specified in the minimal.yml file and includes all further required configuration files.

common

Basic configuration for general tasks of the workflow. Most general configuration for ICON-ART workflows is included here. Important files here are

%PROJDIR%/conf/common/build.yml                   -----> Options for building the ICON model and Python environment
%PROJDIR%/conf/common/platforms.yml               -----> Configuration of the (HPC) platforms
%PROJDIR%/conf/common/platforms/<PLATFORM>.yml    -----> Platform-specific configuration (e.g. pool directories, module names, ...)

ICON cases

There are different ICON use cases (ICON_CASE) defined:

• art

• real-from-ideal

• real-from-dwd-ana

• build_only

• seamless

which are more or less self-explanatory. Each use case has its own subdirectory with specific configurations. The art case is specialized for real data runs with ICON and ICON-ART but is versatile enough to also handle most such cases. It is the most general use case and thus recommended. The seamless case is a very similar case but includes coupling to the land surface scheme JSBACH and optionally the ocean.

Hint

For most applications, the versatile art case is best or - if you need coupling to the ocean - the seamless case.

For the art case, there are two additional configuration files (and likewise for the seamless case):

%PROJDIR%/conf/art/experiments/<expname>.yml
%PROJDIR%/conf/art/simulation.yml

The experiments/<expname>.yml file (multiple ones for the different templates exist) contains all the experiment-specific configuration. The simulation.yml configuration holds some basic keys necessary for running an ART use case. Usually, it does not have to be modified by the user.

Hint

Most of your configuration goes into the experiments/<expname>.yml file.

See also

Important config options

User specific configuration

This file contains some values that are specific to the user but not the experiment. It is intended to provide user and project information for the HPC systems. For this, it has to contain a section PLATFORMS and in this section a section for each HPC system in use with the USER and PROJECT keys. An example for it is the examples/platforms_user.yml file. We suggest to use this and modify it. On Levante, the SCRATCH_PROJECT_DIR field needs to be set to the first letter of your user/project (the one in /home/<letter>/<user>).

Hint

Per default it is located in ~/.asconf/platform_user.yml. Instead, the full path can be given in conf/<ICON_CASE>.yml in the DEFAULT.CUSTOM_CONFIG.POST section.