Step by step guide#

Note

This guide assumes that auto-icon is already fully set up. You can follow the instructions on the respective setup page to do so.

It is an overview of the steps that have to be done. If you want to transform a runscript into an auto-icon configuration, you can also follow the instructions here and for the configuration steps (5. and 7.), refer to the detailed documentation on how to transform a runscript into an auto-icon configuration.

1. Add a new experiment#

Run the following command and substitute the placeholders <HPCARC> and <ICON_CASE>. <HPCARC> is the HPC system to be used. Currently predefined are ‘LEVANTE’ and ‘HOREKA’. <ICON_CASE> is the type of ICON use case to be used. It can be art (for a basic atmospheric run with or without ART), seamless (for a run coupled to JSBACH and/or the ocean) or one of real-from-ideal, real-from-dwd-ana or build-only. For the latter three, please refer to the Quick start guide.

$ autosubmit expid \
  -min \
  -repo "https://gitlab.dkrz.de/auto-icon/auto-icon.git" \
  -conf "conf/<ICON_CASE>.yml" \
  -b main \
  -H <HPCARC> \
  -d "My perfect but short experiment description."
Autosubmit is running with 4.0.84
The new experiment "a00j" has been registered.
Generating folder structure...
Experiment folder: /pool/experiments/a00j
Generating config files...
Experiment a00j created

2. Go to the experiment directory#

Change to the experiment directory (e.g. /pool/experiments/a00j). All paths from now on will be relative to this directory. Also remember the “expid” of the project (e.g. a00j). It has proven useful to write it down for later reference.

3. Initialize your experiment#

This will clone the auto-icon project in your experiment folder (in proj/git_project).

autosubmit create <expid> -np

Further, the command creates the first workflow. Lateron it will be used again to create the final workflow.

4. Initialize the auto-icon project#

Choose a name for your experiment (without spaces, ideally it is short but descriptive). If you want to run one of the templates, you should just use the name of the template. For complete beginners with ICON, we recommend NWP_LIFETIME (chemistry) or DUST_RAD (dust aerosol) standard cases of ICON-ART.

There are several different templates among which are also some of the ICON-ART standard cases (with input data publicly available). Further, for the following cases, input data is available additionally:

  • DUST_AERODYN: global dust simulation with multiple chunks. (input data)

  • ICON_ART_LES: basic ICON-ART-LES simulation with diverse soil data. (input data)

To run the templates, download the input data, extract it (tar -zxvf <archive name>) and move it to wherever you want your input directory to be (see below for specification of input directories).

Now run the following command:

./proj/git_project/auto-icon-init -e <EXPNAME> [-t <TEMPLATE>]

This step is optional but does severa convenience tasks. It creates a separate directory <EXPNAME> and symlinks from within this directory to relevant configuration files. Thus, you can omit the step and edit/create all the files directly. For the following guide, we suppose you did this step. The script also has a bunch of options, so you might want to check these out with ./proj/git_project/auto-icon-init -h.

You might want to add the option --share to share the ICON model and Python virtual environment between different experiments. It will be located in ~/auto-icon-share but can be changed by adding a path to the share option. Details can be found here.

If you want to run your own experiment, you now have to fill in the namelist at <EXPNAME>/namelist.nml (or .yml if you did choose this option). If you did not run the script, create/modify the namelist in proj/git_project/namelists/<ICON_CASE>.

Update the main configuration file (only if init script was not used)

Open proj/git_project/conf/<ICON_CASE>.yml and set the value of EXPNAME to your chosen experiment name <EXPNAME>.
Further, you can set several general options for your workflow in this file.

5. Modify the namelist#

In this step, you modify the namelist to make use of auto-icon’s functionality. If you have taken a template, this step can be skipped.

  • In ‘grid_nml’ remove dynamics_grid_filename and radiation_grid_filename. It will be added by auto-icon.

  • Substitute all bash variable placeholders (${...}, e.g. for filetype or EXPNAME):

    • Filetype can be deleted it will be added automatically.

    • EXPNAME can be substituted by the Autosubmit placeholder %EXPNAME%.

    • You should not use nsteps (number of total steps), ${DTIME} (timestep), "${OUTPUT_START}" and "${OUTPUT_END}", it will all be added by auto-icon.

  • In ‘output_nml’ you can remove output_interval if it is set to the default in your experiment.yml file.

  • If you are using a variable INDIR (‘art_nml’), replace this by %DIRECTORIES.INDIR%. Do the same for OUTDIR (if applicable).

6. Modify the configuration#

auto-icon will take care of compiling the model code. For this guide, the default settings should be fine. If you would like to run your own experiments with a specific model version, further information is provided on the page on Building ICON. Specifying the option -b at the init script will also create a link to the build configuration file.

Modify <EXPNAME>/experiment.yml, the main configuration file (pointing to proj/git_project/conf/<ICON_CASE>/experiments/<EXPNAME>.yml). If you have taken the template directly or adjusted a template configuration, it will be populated with the template configuration. Otherwise it will be a configuration file with all possible config options and extensive comments. The original version of this file can always be viewed for reference at proj/git_project/conf/<ICON_CASE>/experiments/template.yml.

The major configuration sections are:

  • JOBS.RUN_ICON.NODES:: specify the number of processors to be used for the model run.

  • JOBS.RUN_ICON.WALLCLOCK:: specify the walltime for the model run.

  • GRID section: specify the grid to use using the respective fields as documente

  • EXPERIMENT section:

    • The experiment will run for each startdate (DATELIST) and each MEMBER a set of NUMCHUNKS consecutive simulations. Hereby, each chunk is a separate job (on the HPC system) with a length of CHUNKSIZE*CHUNKSIZEUNIT, e.g. 3 days.

    • TIMESTEP: your experiment time step

    • CHECKPOINT_INTERVAL and OUTPUT_INTERVAL just relate to the respective ICON parameters

  • DIRECTORIES section (all directories are to be specified for the used HPC system):

    • INDIR is the directory, where all the input files are located (subdirectories are possible)

    • OUTDIR is the directory, that will be used by ICON as working directory

    • LINK_FILES: all files required for ICON can be listed here and will be linked to from OUTDIR

      • FILES: list of general files to be linked (relative to INDIR or some pool directories)

      • ICON_DATA: list of files to be linked relative to <ICON root directory>

      • ART: list of files to be linked relative to <ICON root directory>/externals/art

Further modifications are possible according to the configurations page. A detailed guide on specifying input and output directories can be found here. For all details on the treatment of the configuration by Autosubmit can be found in the Autosubmit documentation (simple, advanced).

7. Provide your account details#

Hint

This has to be done only once for the first experiment.

Copy the user’s platform information to a private location:

cp proj/git_project/examples/platforms_user.yml ~/.asconf/platforms_user.yml

Now fill in the example fields with your account details. The directory ~/.asconf already exists for each user on the Autosubmit server. If you still choose a different file location, also modify in <EXPNAME>/<ICON_CASE>.yml the DEFAULT.CUSTOM_CONFIG.POST section.

8. Create your workflow job list#

This now creates the actual list of jobs for your experiment. It will show you a plot of your workflow or use -np to omit the plot:

autosubmit create <expid> [-np]

9. Optional: create job scripts for manual inspection#

All job scripts will be located in tmp/:

autosubmit inspect <expid>

10. Run your experiment:#

Remember to enable password-less ssh access, especially unlocking the key with the ssh-agent (see here for details):

eval $(ssh-agent -s)
ssh-add ~/.ssh/<ssh_key_name>

Run the experiment in the foreground

autosubmit run <expid>

or in the background

nohup autosubmit run <expid> &

or put the already running instance to the background (with CRTL + Z and bg).

11. Monitor the progress of your experiment:#

autosubmit monitor <expid> [--txt]

With the option --txt, you can also only write a text file with the status, instead of creating a plot.

Hint

You can also monitor your progress via the Autosubmit GUI.

Now you are ready to run your own simulation with auto-icon. If you already know the ICON model, go on an transform your runscript into an auto-icon configuration. Otherwise, check out the other templates and learn more about the ICON model in the official guide.