# Getting Started Welcome to the documentation of the *auto-icon* project. For a quick start, check out the [Shortest guide to success](Introductory-guides/Shortest-guide-to-success.md). You should also consider [Setting up _auto-icon_](/Setup/Setting-up-auto-icon.md) and the [User guide](/Userguide/index.rst). If you have any questions, check out the [FAQs](/Organization/Frequently-asked-questions.md) or get in [Contact](/Organization/Contact.md) with us. ## ICON, Autosubmit and _auto-icon_ [Autosubmit](https://autosubmit.readthedocs.io/en/master/) is a general workflow manager designed to manage complex workflows on HPC platforms. It holds a database of _experiments_ that are managed with Autosubmit. In this database, very basic information on the experiments are stored, like the date of creation, the HPC architecture used and a short description. Each experiment also gets a unique identifier (the _expid_) and the full experiment _configuration_ is stored in a separate directory for this expid. The _configuration_ of an experiment contains information on (among others) - the workflow to run (i.e. which individual _jobs_ should be executed, see below) - which model to use for the simulation (i.e. ICON-ART, which version/branch/...) - which parameters to use for the model run (mostly set by a namelist file) - where to run, i.e. on which HPC _platform_ - where to obtain input data from - where to store output data Autosubmit takes care of running all the jobs on the specified _platform_, i.e. submitting them to the respective batch system, taking into account _dependencies_ between jobs (e.g. the model can only run after it has been compiled). While the experiment runs, Autosubmit can also provide information on the current progress (see [documentation](https://autosubmit.readthedocs.io/en/master/userguide/monitor_and_check/index.html#how-to-monitor-an-experiment)). To do so, it runs a (background) task, that regularly checks all the currently running jobs, submits new jobs if necessary and collects log files of finished jobs. This job runs on a designated server or a local laptop/workstation, not on the HPC _platform_. ## General modeling workflow A general _auto-icon_ workflow including preparation and post-processing tasks can be represented by the following scheme: Most of the ICON-ART workflows can be cast in the following workflow:  The workflow is created by Autosubmit and is resembled by the dependencies between tasks. The yellow tasks are described by _auto-icon_ and executed by Autosubmit, whereas the grey tasks are core functionality of Autosubmit. The purple/red colored actions are tasks manually done by the user. Parameters in the _configuration_ can be used to adjust the behavior of the jobs to the users need (e.g. directory names). The purpose of the _auto-icon_ project is to provide a set of scripts for the yellow [jobs](Userguide/Jobs.md) and the necessary [_configuration_](/Userguide/Important-configuration.md) to run these jobs with Autosubmit. ## Where is what? Autosubmit runs on a central server. This server has a common database for the experiments that is handled by Autosubmit internally. Further, all experiments have their own directory (at KIT in `/pool/experiments/`). This contains the following: - `conf`: general configuration for Autosubmit (see [configuration](/Userguide/Configuration-files.md)) - `pkl`: Autosubmit internal files for storing workflow information - `plot`: Visualization files of the workflow (e.g. from create, monitor, ...) - `proj`: project directory, i.e. the directory where _auto-icon_ is cloned - `status`: internal status files - `tmp`: log files for both Autosubmit internal logs and the logs of _all experiment jobs_ Each experiment runs on one default HPC system. Autosubmit creates a custom directory on this system, where it stores all job log files (before copying them back to the central Autosubmit server). Experiment input and output data **is not stored** on the central Autosubmit server and also not in this custom directory. However, the users should specify a custom directory for this (see [here](/Userguide/Important-configuration.md#specifying-directories) for details). ## Experiment, start dates, members and chunks A single _experiment_ can feature model runs at different _start dates_. Similarly, for each _start date_ one or multiple _members_ can run, e.g. for an ensemble of simulations (this is an Autosubmit ensemble, not an ICON ensemble even though they are quite similar in nature). Each _member_ starts a single model run, that can be split into several _chunks_ (e.g. of one month length each) and submitted to the HPC system as a single job. This has two purposes: to reduce the maximum walltime to fit to the HPC walltime and if requested, reinitialization is triggered between the chunks. Thus, a hierarchy of one experiment containing two start dates with two members each and each member running two chunks, will create the workflow below. All these jobs run within one workflow and Autosubmit submits jobs not depending on each other (e.g. from different start dates) in parallel while submitting the chunks consecutively. ``` experiment ├--- start date 1 | ├--- member 1 | | ├--- chunk 1 | | └--- chunk 2 | └--- member 2 | ├--- chunk 1 | └--- chunk 2 └--- start date 2 ├--- member 1 | ├--- chunk 1 | └--- chunk 2 └--- member 2 ├--- chunk 1 └--- chunk 2 ```