Changelog#

This page provides the change logs of all the changes since the first test release. Most of these changes are first merged to the development branch and the listed date is the date of the merge to this branch.

Unreleased#

Features:

  • Included documentation in the repository, building and deploying it via Gitlab Pages (see JOSS review).

  • A job for creating grids is added.

  • A job for remapping input data with CDO is added.

  • Full support for limited area grids and nested domains in combination with radiation grids is included.

  • Visualization possible on native ICON grid, actually plotting triangles, not interpolating contours (#174).

  • Visualization uses dask (#172).

Improvements:

  • Improvements related to JOSS review:

    • Remove bash env variables from LEVANTE configuration (closing #169).

    • Update of the ICON use cases to the current version, including remarks on cases using GPL licensed ART code (closing #158, #171).

    • Included documentation in the repository, building and deploying it via Gitlab Pages.

    • Updated Readme and copyright information.

    • Several minor improvements for documentation, pipelines and ICON use cases.

  • Created symlinks between LOGDIR and OUTDIR.

  • File type switches for input files added (#148)

  • Update to the latest ICON version and fixing all ICON-ART standard cases.

  • Documentation fully revised.

  • Remapping of input data now works for all grid files (#181).

Fixes:

  • Bugfix in the init script: change only supplied parameters on update (closing #170).

  • Update of resource specifications to fix some minor issues.

  • Init script was not correctly using the template for the seamless case (#178).

  • Fixes for empty lists and other empty-like fields (closes #168, #178).

  • Some fixes for grid creation (closing #193, #194)

Internal:

  • Linting and Formatting (closes #107).

  • Reduction of test case resources.

Version 1.3#

Released on 17.01.2025. Release; Tag.

17.01.2025: Several minor improvements and bugfixes

  • Init script help message improved.

  • Init script will now always change the experiment name accordingly.

  • SAMOA job will not fail if all files are already checked.

23.11.2024: Coupling to ocean and Jsbach supported

  • Support for coupling to the ocean and Jsbach is added:
    A new ICON_CASE seamless is added. It is meant for these coupling tasks.
    Further information can be found in the seamless experiments/template files.

18.10.2024: minor fixes and updates to init script

  • Overworked init-script:

    • simulation.yml file added to links.

    • Visualization config file added to links also if only animation is used.

  • Specifying time step in namelist and not in EXPERIMENT section is now supported.

  • Several minor fixes.

17.10.2024: Fix: files could added twice to the list of linked files

  • If a file (e.g. DWDFG) was given both in the FILELIST section of the GRID section and a name was specifically assigned to it (e.g. via the FILENAMES section in a domain), the file was added twice.

  • This resulted in two files with different names (the default and the custom) to be searched for, which led to File not found errors.

01.10.2024: CDO post-processing job enhanced.

  • If multiple files are processed, one can use PASSEACH as OUTPUT to pass the result of each individual file to the next operator, which receives multiple input files or acts in parallel on multiple files.

  • Filename substitution is now possible also when selecting some variables form the input as well as when chaining with the PASSEACH option.

09.09.2024: CDO post-processing job added.

A general CDO post-processing job is added that runs CDO commands based on a config. Supported features:

  • Run general CDO command with an arbitrary operator and arbitrary arguments.

  • Support for operator chaining (implicit and explicit).

  • Single statement for splitting all output files per variable.

  • Support for retrieving output file name form input file name.

  • Input files can also be variable(s) selected from other file(s) (implicit operator chaining).

09.08.2024: Major rework of plotting scripts.

  • Added script to create animations

  • Modularization of plot scripts

  • Update of logging.

  • Added enhanced functionality for selection and reduction of levels and times.

  • Colorbar scale options updated.

  • Limits and color clipping added.

  • Unit treatment enhanced.

12.07.2024: Collection of several minor enhancements

  • Init script updated:

    • Added [no-*] options to disable tasks.

    • Added option to only link files.

    • Added option to update only specified config.

  • ERA5 data can be used for reinit runs with automatic remapping of input data for each chunk.

  • Several internal updates for the workflow with the new files/domains mechanism.

  • For non-reinit cases, only a single namelist will be created.

  • Preparation of the new chunk in reinit runs was greatly simplified and put into the RUN_ICON job, greatly improving performance of these cases (and allowing wrappers).

  • Added a namelist check for dates in filenames for reinit runs.

  • Added a job to clean up restart files every few chunks.

Version 1.2#

Released on 30.07.2024. Release; Tag.

30.07.2024: Further unification of LMU and KIT repositories

  • Unification of workflows

  • Cleanup

Version 1.1#

Released on 19.07.2024. Tag.

17.06.2024: internal rework

A major internal rework paves the way for several upcoming features and convenience updates.

06.05.2024: Fortran namelists support way more flexibility

  1. Fortran namelists are now also modified with the Python routine designed to manage the yaml namelists, supporting nearly all functionality for Fortran namelists. Only the change of namelist parameters for each member is still only available for yaml namelists. The documentation is updated accordingly, check out the namelist page for details.

  2. The PREPARE_DATE job is removed as it basically did nothing.

  3. a variable INSTITUTE is added that is passed on to the master namelist.

02.05.2024: Feature: use specific submodule branches or commits

A new subsection in the build.yml file allows to use specific submodule branches or commits. In the icon section, add a section SUBMODULES. It contains a list of entries, one per submodule, that should point to a non-default value. The syntax is <submodule>:<branch>:<commit>, where submodule is the submodules name, e.g. art, the branch is the branch to check out, e.g. master and commit is either a specific commit or tag or can be HEAD, or CURRENT. HEAD means to use the branches HEAD and CURRENT to use the current state of the repository (if it is not yet existing, use the branches HEAD). (cf. MR !51)

26.04.2024: Several bug fixes

Bug Fixes:

  • Files to be linked in the lists ICON_DATA and ART will now be linked from these directories, NOT from the INDIR.

  • NWP_GASPHASE working now properly.

  • Some issues could arise with file linking if radiation grids are used.

  • Fixed issues with downloading/unpacking a zip archive if the target directory is not empty.

  • Fixed the fallback option to clone a new directory with branch name CURRENT.

24.04.2024: Inclusion of ICON-ART standard cases and input feature updates

Feature updates:

  • Included the EXTPAR_TILES key in the grid parser.

  • Added the option to find/download a zip archive and use its contents as simulation input.

  • Reworked pool directories.

  • Added FILENAMES section in the GRID section that supports patterns.

  • Added FILELIST Details on using the updated features are as usual documented in the template config file conf/art/experiments/template.yml.

Bug fixes:

  • NWP_EXT_DATA.nml using old input data locations.

  • File downloads always succeeded.

  • grfinfo files were not found if provided locally.

16.04.2024: Yaml namelist renamed

The yaml namelist have been renamed again: the prefix nml- was removed, naming them now again <EXPNAME>.yml.

16.04.2024: initialization script added

A new initialization script auto-icon-init is added. Calling this script right after creating an experiment (i.e. after autosubmit expid ... and autosubmit create ...) allows for an easy and flexible initialization of a few main features and further creates a directory with symlinks to all important config files. This script is purely optional but aims at simplifying the start with a new experiment. For details, run the script with the help options -h|--help or consult the wiki page on running your own simulation.

11.04.2024: Feature added: build current state of model source code.

In case the model source code was changed in place on the HPC system, setting the option %ICON.BRANCH% to CURRENT will build this model version and not perform a pull to update to the latest version. If the repo does not exists, CURRENT will behave like HEAD, i.e. pull the latest version.

09.04.2024: Reinitialization is updated (again).

Reinitializations are now also capable of continuing with chemistry data. Further, the switches for reinitializations are renamed. Check MR !43 and the configuration files page for details.

22.03.2024: Reinitialization is updated.

For the Yaml namelist approach, all files required for the continuation of the run are provided automatically. Only the data for the reinitialized quantities has to be provided manually.

For the f90 namelist approach, further the output files have to be added to the namelist, but several minor improvements are made.

Extensive documentation is added to the wiki.

25.01.2024: Job for basic plotting included

A new job for basic plotting is included. See MR !34 for details.

18.01.2024: SAMOA is now available for checking simulation output

SAMOA

is a tool for performing sanity checks on the output of COSMO and ICON(-ART) runs. It is now available and included by default in runs with the *art* ICON case.

08.01.2024: Changing ICON namelists with members

The ICON namelist can now be changed for each member by adding the changes to the namelist file. Details can be found in the documentation.

21.12.2023: Incremental build supported

The ICON model is currently build each time from scratch in a temporary directory. If the model is already built, building could be skipped by setting the build job to successful. A few changes are implemented now:

  • The default is that the model is not build, if the ICON binary exists.

  • If it does not exist, a new build is run. The ICON source code will be stored and only updated to the latest version matching the desired branch and already compiled parts of the code will also be stored. This allows for an incremental build, saving a lot of time during model development.

To alter this behavior, two switches have been implemented (set to True or False):

  • ENSURE_BUILD: the model will be build, even if a binary exists (this is useful for model code development, where regular updates and re-builds of the model are necessary; here the incremental build makes building fast)

  • REBUILD_FROM_SCRATCH: build the model completely from scratch, without using existing (temporary) files

Version 1.0#

Released on 23.11.2023. Release; Tag.

22.11.2023: two important bug fixes

  • Fixed the ICON-ART-LES template (please note the renaming to caps)

  • Fixed prepare_member.sh script when using the f90 namelist approach.

For details see MR !26.

14.11.2023: added yaml config for grid info and support for nested grids

The grid info can now be specified in a yaml section (see here for details).

  • The yaml approach for grids allows the use of nested grids.

  • Changed default: Horeka default workspace name was changed from icon to auto-icon

  • Changed behavior: yaml namelists need to be of the form nml-<expname>.yml

  • CHECK_OUTPUT job: updated pattern to more specifically find icon output

  • Bugfix: EXPERIMENT.OUTPUT_FILE_INTERVAL was changed to EXPERIMENT.FILE_INTERVAL, which is the name of the variable used in the ICON namelist.

  • Bugfix: retrieving grids from official repositories now also downloads a grfinfo file if present.

31.10.2023: Implemented a yaml parser to create the namelist With the yaml parser several new features could be implemented. New features only for the yaml approach:

  • Output files required for the reinitialization are created automatically.

  • Physics timesteps are provided as multipliers to the timestep.

Further changes:

  • REINIT-CHEM is added to the pipeline.

  • Placeholders EXPERIMENT.OUTPUT_FILE_INTERVAL and EXPERIMENT.STEPS_PER_FILE added. In the yaml approach these are added to all output_nml groups that do specify one of these.

  • Added default email notification if a RUN_ICON job fails.

Version 0.2#

Released on 10.10.2023. Tag.

09.10.2023: Reinitialization support and updates on input data linking

  • Reinitialization support is added. Documentation on this can be found here.

  • Further a template REINIT-CHEM is added.

  • Linking of input data now supports using patterns and placeholders for e.g. startdates.

  • Detailed documentation on this is now compiled in the wiki page on input data linking.

15.08.2023: Input data retrieved from pool directories more flexibly Retrieving of input data (to be linked to the working directory) from pool directories is now more flexible:

  • If the file name starts with a slash ‘/’, it is treated as an absolute path to the file and searched for exactly this path.

  • If it starts with a protocol ‘https://’, ‘ftp://’, ‘http://’, it is treated as an URL, downloaded to INDIR and then linked.

  • Otherwise, it is searched for in INDIR, i.e. looking for INDIR/FILENAME.

  • Then location directly in the pool directory is checked, i.e. POOL/FILENAME.

  • Then the pool directories on Levante or Horeka are searched for, i.e. if a file FILENAME exists somewhere (regardless of the depth in the tree) within the pool directory.

  • A grid file is further searched for in the online repository.

Full documentation on this will be provided soon.

15.08.2023: CI pipelines for auto-icon project testing

CI pipelines were introduced for a smooth testing of the auto-icon project development. This included two minor changes:

  • The default build is currently set to commit ‘601d5c8ceddf482febd82a5854283ae32507e951’, which is before the merge of icon-2.6.7-rc. The build script for HoreKa has to be adapted first, as it does not build ICON for the latest commit.

  • In the ‘prepare_member’ job, icon --version is now executed and the output stored into the log and in the output directory in the file icon_version_<chunk>.txt with the number of the current chunk <chunk>.

Version 0.1b#

Released on 03.08.2023. Release; Tag.

03.08.2023: v0.1b: Test release The first actual test release is done. The main changes versus the training release are:

  • Updated the Wiki with instructions for the actual Autosubmit server.

  • Updated the Wiki content for setting up everything for the first use: Setting up auto-icon

  • Created tag v0.1b

27.07.2023 Training release The first internal test release for the training was released. It contains basic functionality for running on the Autosubmit test server.