New jobs#

To add a new job, you have to do a few steps as outlined below.

Jobscript#

A new jobscript for the job is required. These are the templates for the actual jobs executed by Autosubmit after Autosubmit placeholders have been substituted and header and footer have been added.

The scripts for all jobs (plain bash or python scripts) are collected in the templates directory. It should be placed there in the common subdirectory if it is general to all (or many) ICON cases or into the subdirectory of a single case if it is case-specific.

Additionally, much of the functionality is extracted into custom Python modules in templates/utils. These modules are transferred to the remote HPC system with the PREPARE_EXPERIMENT job.

See also

Autosubmit documentation

Configuration file#

Next, a job-specific configuration file is required, such that the job can be included in the workflow. This yaml configuration file should be placed in conf/common/jobs named just like the job template is named and should contain the Autosubmit configuration for a job. The other files in the directory can provide examples for such files. It looks just like this:

JOBS:
  POST_CDO:
    FILE: "templates/common/post-cdo.py"
    DEPENDENCIES:
      RUN_ICON:
      BUILD_PYTHON:
      POST_SAMOA:
    RUNNING: member
    PLATFORM: "%DEFAULT.HPCARCH%_LOGIN"
    TYPE: python
    EXECUTABLE: "/usr/bin/env -S PYTHONPATH=%PYTHON_ENVIRONMENT.MODULES_DIR%:${PYTHONPATH} %AISHAREDIR%/%PYTHON_ENVIRONMENT.FOLDER_NAME%/bin/python3"

If the job requires some additional files from the repository, Autosubmit can also copy those to the HPC system, by adding them like so:

JOBS:
  PRE_CREATE_GRID:
    FILE: "templates/common/pre-create_grid.py,namelists/%ICON_CASE%/pre-create_grid.nml"

See also

Autosubmit documentation

Include files#

You then have to update the include files to ease inclusion of jobs in the actual workflow. For this, add a line into each ICON case include file conf/<icon_case>.yml that can use this job as follows:

DEFAULT:
  CUSTOM_CONFIG:
    PRE:
      ...
      # - "%PROJDIR%/conf/common/jobs/my-new-job.yml"

Init script#

Last, you should update the jobscript to include an option for the new file. This needs a few modifications:

  • Add the command line argument with add_argument (main part)

  • If you include an external tool, add a license hint (just like licenseNotes.add("CDO") and add the tool and license in the function initLicenses)

  • If a new (config) file shall be linked to the common directory:

    • Add the file in the function initFiles

    • Include the file to the list with filesToLink.append("IFS2ICON") if job is activated

  • Add the job and the corresponding command line option to the dict configUpdate