4. Developer quickstart guide¶
This section contains instructions for beginning to
4.1. Developer installation instructions¶
To download and install the framework for development, follow the instructions for end users given in Installation instructions, with the following developer-specific modifications:
Obtaining the source code¶
POD developers should create their branches from the main branch of the framework code
git checkout -b feature/[POD name] main
This is the “beta test” version, used for testing changes before releasing them to end users
Developers may download the code from GitHub as described in Obtaining the code, but we strongly recommend that you clone the repo in order to keep up with changes in the main branch, and to simplify submitting pull requests with your POD’s code. Instructions for how to do this are given in Git-based development workflow.
Installing dependencies via conda¶
Regardless of development language, we strongly recommend that developers use conda to manage their language and library versions. Note that Conda is not Python-specific, but allows coexisting versioned environments of most scripting languages, including, R, NCL, Ruby, PyFerret, and more.
Python-based PODs should be written in Python 3.11 or newer. We provide a developer version of the python3_base environment (described below) that includes Jupyter and other developer-specific tools. This is not installed by default, and must be requested by passing the
--all flag to the conda setup script:
If you are using Anaconda or miniconda to manage the conda environments, run: .. code-block:: console
% cd $CODE_ROOT % ./src/conda/conda_env_setup.sh –all –conda_root $CONDA_ROOT –env_dir $CONDA_ENV_DIR
Installing dependencies via Micromamba¶
Micromamba is a lightweight version of Anaconda. It is required to install the base and python3_base conda enviroments on macOS machines with Apple M-series chips. Installation instructions are available in the Micromamba Documentation, Once Micromamba is installed on your system, run the following to install all conda environments if you are NOT using an Apple M-series machine, where $MICROMAMBA_ROOT is the location of the micromamba installation, and MICROMAMBA_EXE is the path to the micromamba executable on your system:
% cd $CODE_ROOT
% ./src/conda/micromamba_env_setup.sh --all --conda_root $MICROMAMBA_ROOT --micromamba_exe $MICROMAMBA_EXE --env_dir $CONDA_ENV_DIR
If you are using an Apple M-series machine, you can install just the base and python3_base environments:
4.2. POD development using existing Conda environments¶
To prevent the proliferation of dependencies, we suggest that new POD development use existing Conda environments whenever possible, e.g., python3_base, NCL_base, and R_base for Python, NCL, and R, respectively.
In case you need any exotic third-party libraries, e.g., a storm tracker, consult with the lead team and create your own Conda environment following instructions below.
The framework provides the _MDTF_python3_base Conda environment (recall the
_MDTF prefix for framework-specific environment) as the generic Python environment, which you can install following the instructions. You can then activate this environment by running in a terminal:
% source activate $CONDA_ENV_DIR/_MDTF_python3_base
$CONDA_ENV_DIR is the path you used to install the Conda environments. After you’ve finished working under this environment, run
% conda deactivate or simply close the terminal.
4.3. POD development using a new Conda environment¶
If your POD requires languages that aren’t available in an existing environment or third-party libraries unavailable through the common conda-forge and anaconda channels, we ask that you notify us (since this situation may be relevant to other developers) and submit a YAML (.yml) file that creates the environment needed for your POD.
The new YAML file should be added to
src/conda/, where you can find templates for existing environments from which you can create your own.
The YAML filename should be
The first entry of the YAML file, name of the environment, should be
We recommend listing conda-forge as the first channel to search, as it’s entirely open source and has the largest range of packages. Note that combining packages from different channels (in particular, conda-forge and anaconda channels) may create incompatibilities.
We recommend constructing the list of packages manually, by simply searching your POD’s code for
importstatements referencing third-party libraries. Please do not exporting your development environment with
% conda env export, which gives platform-specific version information and will not be fully portable in all cases; it also does so for every package in the environment, not just the “top-level” ones you directly requested.
We recommend specifying versions as little as possible, out of consideration for end-users: if each POD specifies exact versions of all its dependencies, conda will need to install multiple versions of the same libraries. In general, specifying a version should only be needed in cases where backward compatibility was broken (e.g., Python 2 vs. 3) or a bug affecting your POD was fixed (e.g., postscript font rendering on Mac OS with older NCL). Conda installs the latest version of each package that’s consistent with all other dependencies.
Framework interaction with conda environments¶
As described in Running the package on sample model data, when you run the
mdtf executable, among other things, it reads
default_tests.jsonc and executes POD codes accordingly. For a POD included in the list (referred to as $POD_NAME):
The framework will first try to look for the YAML file
src/conda/env_$POD_NAME.yml. If it exists, the framework will assume that the corresponding conda environment
_MDTF_$POD_NAMEhas been installed under
$CONDA_ENV_DIR, and will switch to this environment and run the POD.
If not, the framework will then look into the POD’s
settings.jsoncspecifies the programming language(s) adopted by the POD:
a). If purely Python 3, the framework will look for
src/conda/env_python3_base.ymland check its content to determine whether the POD’s requirements are met, and then switch to
_MDTF_python3_baseand run the POD.
b). Similarly, if NCL or R is used, then
Note that for the 6 existing PODs depending on NCL (EOF_500hPa, MJO_prop_amp, MJO_suite, MJO_teleconnection, precip_diurnal_cycle, and Wheeler_Kiladis), Python is also used but merely as a wrapper. Thus the framework will switch to
_MDTF_NCL_base when seeing both NCL and Python in
The framework verifies PODs’ requirements via looking for the YAML files and their contents. Thus if you choose to selectively install conda environments using the
--env flag (Installing dependencies), remember to install all the environments needed for the PODs you’re interested in, and that
_MDTF_base is mandatory for the framework’s operation.
For instance, the minimal installation for running the
_MDTF_NCL_base(because of b), and
_MDTF_convective_transition_diag(because of 1). These can be installed by passing
--envflag one at a time (Installing dependencies).
Testing with a new Conda environment¶
If you’ve updated an existing environment or created a new environment (with corresponding changes to the YAML file), verify that your POD works.
Recall how the framework finds a proper Conda environment for a POD. First, it searches for an environment matching the POD’s short name. If this fails, it then looks into the POD’s
settings.jsonc and prepares a generic environment depending on the language(s). Therefore, no additional steps are needed to specify the environment if your new YAML file follows the naming conventions above (in case of a new environment) or your
settings.jsonc correctly lists the language(s) (in case of updating an existing environment).
For an updated environment, first, uninstall it by deleting the corresponding directory under
Re-install the environment using the
conda_env_setup.shscript as described in the installation instructions, or create the new environment for you POD:
% cd $CODE_ROOT % ./src/conda/conda_env_setup.sh --env $your_POD_short_name --conda_root $CONDA_ROOT --env_dir $CONDA_ENV_DIR
Or, if using micromamba:
% cd $CODE_ROOT % ./src/conda/conda_env_setup.sh --env $your_POD_short_name --micromamba_root $MICROMAMBA_ROOT --env_dir $CONDA_ENV_DIR
Have the framework run your POD on suitable test data.
Add your POD’s short name to the
pod_listsection of the configuration input file (template:
Prepare the test data as described in Running the package on your data.