CDO post processing#

A CDO post-processing job is available, to be activated with the --cdo option in the init script. This job is basically a wrapper job to run (nearly) arbitrary CDO jobs on the simulation output.

To configure these, add a CDO section in your conf/art/experiments/<expname>.yml file. A full description of the current syntax is available in the conf/art/experiments/template.yml file. In general, this looks as follows:

CDO:
  SELDUST:
    OPERATOR: selname
    ARGS: [dusta, dustb, dustc]
    FILES: 'icon-art-%EXPNAME%-aero*'
    OUTPUT: '/aero/dust/'
  ADD:
    OPERATOR: add
    BINARY: True
    FILES:
      'icon-art-%EXPNAME%-dust*.nc':
        name: [dusta, dustb, dustc]
    OUTPUT: PASSEACH
  VAR_RENAME:
    OPERATOR: chname
    ARGS: [dusta, dust_total]
    FILES: ADD
    OUTPUT: '/dust/all_dust_total/'

Each section stands for a different (set of) operator call(s) and contains as information

  • the operator to invoke

  • (optional) further arguments

  • input file(s)

  • output file(s)

As input, one can also specify a section (the output from this operator will be used as input via operator chaining), or a selection of specific variables from an input file (that will be selected with a chained sel<kind>.

As output, one can specify a specific file name, a substitution pattern (s.t. each input file will produce a single output file with substituted patterns) or the PASS or PASSEACH keywords that indicate that the output will be used as input for another operator.

From the above configuration, the SELDUST section, will result in operator calls like

cdo selname,dusta,dustb,dustc icon-art-MY_EXPNAME-aero-DOM01_ML_0001.nc icon-art-MY_EXPNAME-dust-DOM01_ML_0001.nc
cdo selname,dusta,dustb,dustc icon-art-MY_EXPNAME-aero-DOM01_ML_0002.nc icon-art-MY_EXPNAME-dust-DOM01_ML_0002.nc
cdo selname,dusta,dustb,dustc icon-art-MY_EXPNAME-aero-DOM01_ML_0003.nc icon-art-MY_EXPNAME-dust-DOM01_ML_0003.nc
...

The ADD and VAR_RENAME will result in more complex operator calls:

cdo chname,dusta,dust_total \
  -add -selname,dusta icon-art-MY_EXPNAME-dust-DOM01_ML_0001.nc \
  -add -selname,dustb icon-art-MY_EXPNAME-dust-DOM01_ML_0001.nc \
  -selname,dustc icon-art-MY_EXPNAME-dust-DOM01_ML_0001.nc \
  icon-art-MY_EXPNAME-all_dust_total-DOM01_ML_0001.nc
cdo chname,dusta,dust_total \
  -add -selname,dusta icon-art-MY_EXPNAME-dust-DOM01_ML_0002.nc \
  -add -selname,dustb icon-art-MY_EXPNAME-dust-DOM01_ML_0002.nc \
  -selname,dustc icon-art-MY_EXPNAME-dust-DOM01_ML_0002.nc \
  icon-art-MY_EXPNAME-all_dust_total-DOM01_ML_0002.nc
...

Important

Many CDO operators are I/O-bound and thus do not require substantial CPU or memory resources. Thus, per default the CDO job runs on the login node. If you plan to run a high amount of CDO post-processing or CPU/memory intensive tasks, you need to tell auto-icon to submit it to the job queue. To do so, you should change in conf/common/jobs/post-cdo.yml the field JOBS.POST_CDO.PLATFORM: to "%DEFAULT.HPCARCH%". You might also want to adjust the partition if there is a designated partition for such jobs (as the shared partition on Levante) with JOBS.POST_CDO.PARTITION: "<partition>".

This Page