4. Developer quickstart guide

This section contains instructions for beginning to

4.1. Developer installation instructions

To download and install the framework for development, follow the instructions for end users given in Quickstart installation instructions, with the following developer-specific modifications:

Obtaining the source code

POD developers should work from the develop branch of the framework code. This is the “beta test” version, used for testing changes before releasing them to end users.

Developers may download the code from GitHub as described in Download the framework code and supporting data, but we strongly recommend that you clone the repo in order to keep up with changes in the develop branch, and to simplify submitting pull requests with your POD’s code. Instructions for how to do this are given in Git-based development workflow.

Installing dependencies via conda

Regardless of development language, we strongly recommend that developers use conda to manage their language and library versions. Note that Conda is not Python-specific, but allows coexisting versioned environments of most scripting languages, including, R, NCL, Ruby, PyFerret, and more.

We recommend that new PODs be written in Python 3. We provide a developer version of the python3_base environment (described below) that includes Jupyter and other developer-specific tools. This is not installed by default, and must be requested by passing the -all-dev flag to the conda setup script:

% cd $CODE_ROOT
% ./src/conda/conda_env_setup.sh --all-dev --conda_root $CONDA_ROOT --env_dir $CONDA_ENV_DIR

4.2. POD development using existing Conda environments

To prevent the proliferation of dependencies, we suggest that new POD development use existing Conda environments whenever possible, e.g., python3_base, NCL_base, and R_base for Python, NCL, and R, respectively.

In case you need any exotic third-party libraries, e.g., a storm tracker, consult with the lead team and create your own Conda environment following instructions below.

Python

The framework provides the _MDTF_python3_base Conda environment (recall the _MDTF prefix for framework-specific environment) as the generic Python environment, which you can install following the instructions. You can then activate this environment by running in a terminal:

% source activate $CONDA_ENV_DIR/_MDTF_python3_base

where $CONDA_ENV_DIR is the path you used to install the Conda environments. After you’ve finished working under this environment, run % conda deactivate or simply close the terminal.

Other languages

The framework also provides the _MDTF_NCL_base and _MDTF_R_base Conda environments as the generic NCL and R environments.

4.3. POD development using a new Conda environment

If your POD requires languages that aren’t available in an existing environment or third-party libraries unavailable through the common conda-forge and anaconda channels, we ask that you notify us (since this situation may be relevant to other developers) and submit a YAML (.yml) file that creates the environment needed for your POD.

  • The new YAML file should be added to src/conda/, where you can find templates for existing environments from which you can create your own.

  • The YAML filename should be env_$your_POD_short_name.yml.

  • The first entry of the YAML file, name of the environment, should be _MDTF_$your_POD_short_name.

  • We recommend listing conda-forge as the first channel to search, as it’s entirely open source and has the largest range of packages. Note that combining packages from different channels (in particular, conda-forge and anaconda channels) may create incompatibilities.

  • We recommend constructing the list of packages manually, by simply searching your POD’s code for import statements referencing third-party libraries. Please do not exporting your development environment with % conda env export, which gives platform-specific version information and will not be fully portable in all cases; it also does so for every package in the environment, not just the “top-level” ones you directly requested.

  • We recommend specifying versions as little as possible, out of consideration for end-users: if each POD specifies exact versions of all its dependencies, conda will need to install multiple versions of the same libraries. In general, specifying a version should only be needed in cases where backward compatibility was broken (e.g., Python 2 vs. 3) or a bug affecting your POD was fixed (e.g., postscript font rendering on Mac OS with older NCL). Conda installs the latest version of each package that’s consistent with all other dependencies.

Framework interaction with conda environments

As described in Run the MDTF framework on sample data, when you run the mdtf executable, among other things, it reads pod_list in default_tests.jsonc and executes POD codes accordingly. For a POD included in the list (referred to as $POD_NAME):

  1. The framework will first try to look for the YAML file src/conda/env_$POD_NAME.yml. If it exists, the framework will assume that the corresponding conda environment _MDTF_$POD_NAME has been installed under $CONDA_ENV_DIR, and will switch to this environment and run the POD.

  2. If not, the framework will then look into the POD’s settings.jsonc file in $CODE_ROOT/diagnostics/$POD_NAME/. The runtime_requirements section in settings.jsonc specifies the programming language(s) adopted by the POD:

    a). If purely Python 3, the framework will look for src/conda/env_python3_base.yml and check its content to determine whether the POD’s requirements are met, and then switch to _MDTF_python3_base and run the POD.

    b). Similarly, if NCL or R is used, then NCL_base or R_base.

Note that for the 6 existing PODs depending on NCL (EOF_500hPa, MJO_prop_amp, MJO_suite, MJO_teleconnection, precip_diurnal_cycle, and Wheeler_Kiladis), Python is also used but merely as a wrapper. Thus the framework will switch to _MDTF_NCL_base when seeing both NCL and Python in settings.jsonc.

The framework verifies PODs’ requirements via looking for the YAML files and their contents. Thus if you choose to selectively install conda environments using the --env flag (Install framework dependencies with conda), remember to install all the environments needed for the PODs you’re interested in, and that _MDTF_base is mandatory for the framework’s operation.

  • For instance, the minimal installation for running the EOF_500hPa and convective_transition_diag PODs requres _MDTF_base (mandatory), _MDTF_NCL_base (because of b), and _MDTF_convective_transition_diag (because of 1). These can be installed by passing base, NCL_base, and convective_transition_diag to the --env flag one at a time (Install framework dependencies with conda).

Testing with a new Conda environment

If you’ve updated an existing environment or created a new environment (with corresponding changes to the YAML file), verify that your POD works.

Recall how the framework finds a proper Conda environment for a POD. First, it searches for an environment matching the POD’s short name. If this fails, it then looks into the POD’s settings.jsonc and prepares a generic environment depending on the language(s). Therefore, no additional steps are needed to specify the environment if your new YAML file follows the naming conventions above (in case of a new environment) or your settings.jsonc correctly lists the language(s) (in case of updating an existing environment).

  • For an updated environment, first, uninstall it by deleting the corresponding directory under $CONDA_ENV_DIR.

  • Re-install the environment using the conda_env_setup.sh script as described in the installation instructions, or create the new environment for you POD:

    % cd $CODE_ROOT
% ./src/conda/conda_env_setup.sh --env $your_POD_short_name --conda_root $CONDA_ROOT --env_dir $CONDA_ENV_DIR

  • Have the framework run your POD on suitable test data.

    1. Add your POD’s short name to the pod_list section of the configuration input file (template: src/default_tests.jsonc).

    2. Prepare the test data as described in Framework configuration for user model data.