.. role:: console(code) :language: console :class: highlight .. _ref-dev-start: Developer quickstart guide ========================== This section contains instructions for beginning to Developer installation instructions ----------------------------------- To download and install the framework for development, follow the instructions for end users given in :doc:`start_install`, with the following developer-specific modifications: Obtaining the source code ^^^^^^^^^^^^^^^^^^^^^^^^^ POD developers should create their branches from the `develop branch `__ of the framework code :: git checkout -b feature/[POD name] develop This is the "beta test" version, used for testing changes before releasing them to end users Developers may download the code from GitHub as described in :ref:`ref-download`, but we strongly recommend that you clone the repo in order to keep up with changes in the develop branch, and to simplify submitting pull requests with your POD's code. Instructions for how to do this are given in :doc:`dev_git_intro`. Installing dependencies via conda ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Regardless of development language, we strongly recommend that developers use conda to manage their language and library versions. Note that Conda is not Python-specific, but allows coexisting versioned environments of most scripting languages, including, `R `__, `NCL `__, `Ruby `__, `PyFerret `__, and more. We recommend that new PODs be written in Python 3. We provide a developer version of the python3_base environment (described below) that includes Jupyter and other developer-specific tools. This is not installed by default, and must be requested by passing the ``-all-dev`` flag to the conda setup script: .. code-block:: console % cd $CODE_ROOT % ./src/conda/conda_env_setup.sh --all-dev --conda_root $CONDA_ROOT --env_dir $CONDA_ENV_DIR POD development using existing Conda environments ------------------------------------------------- To prevent the proliferation of dependencies, we suggest that new POD development use existing Conda environments whenever possible, e.g., `python3_base `__, `NCL_base `__, and `R_base `__ for Python, NCL, and R, respectively. In case you need any exotic third-party libraries, e.g., a storm tracker, consult with the lead team and create your own Conda environment following :ref:`instructions ` below. Python ^^^^^^ The framework provides the `_MDTF_python3_base `__ Conda environment (recall the ``_MDTF`` prefix for framework-specific environment) as the generic Python environment, which you can install following the :ref:`instructions `. You can then activate this environment by running in a terminal: .. code-block:: console % source activate $CONDA_ENV_DIR/_MDTF_python3_base where ``$CONDA_ENV_DIR`` is the path you used to install the Conda environments. After you've finished working under this environment, run :console:`% conda deactivate` or simply close the terminal. Other languages ^^^^^^^^^^^^^^^ The framework also provides the `_MDTF_NCL_base `__ and `_MDTF_R_base `__ Conda environments as the generic NCL and R environments. .. _ref-create-conda-env: POD development using a new Conda environment --------------------------------------------- If your POD requires languages that aren't available in an existing environment or third-party libraries unavailable through the common `conda-forge `__ and `anaconda `__ channels, we ask that you notify us (since this situation may be relevant to other developers) and submit a `YAML (.yml) file `__ that creates the environment needed for your POD. - The new YAML file should be added to ``src/conda/``, where you can find templates for existing environments from which you can create your own. - The YAML filename should be ``env_$your_POD_short_name.yml``. - The first entry of the YAML file, name of the environment, should be ``_MDTF_$your_POD_short_name``. - We recommend listing conda-forge as the first channel to search, as it's entirely open source and has the largest range of packages. Note that combining packages from different channels (in particular, conda-forge and anaconda channels) may create incompatibilities. - We recommend constructing the list of packages manually, by simply searching your POD's code for ``import`` statements referencing third-party libraries. Please do *not* exporting your development environment with :console:`% conda env export`, which gives platform-specific version information and will not be fully portable in all cases; it also does so for every package in the environment, not just the "top-level" ones you directly requested. - We recommend specifying versions as little as possible, out of consideration for end-users: if each POD specifies exact versions of all its dependencies, conda will need to install multiple versions of the same libraries. In general, specifying a version should only be needed in cases where backward compatibility was broken (e.g., Python 2 vs. 3) or a bug affecting your POD was fixed (e.g., postscript font rendering on Mac OS with older NCL). Conda installs the latest version of each package that's consistent with all other dependencies. Framework interaction with conda environments ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ As described in :ref:`ref-execute`, when you run the ``mdtf`` executable, among other things, it reads ``pod_list`` in ``default_tests.jsonc`` and executes POD codes accordingly. For a POD included in the list (referred to as $POD_NAME): 1. The framework will first try to look for the YAML file ``src/conda/env_$POD_NAME.yml``. If it exists, the framework will assume that the corresponding conda environment ``_MDTF_$POD_NAME`` has been installed under ``$CONDA_ENV_DIR``, and will switch to this environment and run the POD. 2. If not, the framework will then look into the POD's ``settings.jsonc`` file in ``$CODE_ROOT/diagnostics/$POD_NAME/``. The ``runtime_requirements`` section in ``settings.jsonc`` specifies the programming language(s) adopted by the POD: a). If purely Python 3, the framework will look for ``src/conda/env_python3_base.yml`` and check its content to determine whether the POD's requirements are met, and then switch to ``_MDTF_python3_base`` and run the POD. b). Similarly, if NCL or R is used, then ``NCL_base`` or ``R_base``. Note that for the 6 existing PODs depending on NCL (EOF_500hPa, MJO_prop_amp, MJO_suite, MJO_teleconnection, precip_diurnal_cycle, and Wheeler_Kiladis), Python is also used but merely as a wrapper. Thus the framework will switch to ``_MDTF_NCL_base`` when seeing both NCL and Python in ``settings.jsonc``. The framework verifies PODs' requirements via looking for the YAML files and their contents. Thus if you choose to selectively install conda environments using the ``--env`` flag (:ref:`ref-conda-install`), remember to install all the environments needed for the PODs you're interested in, and that ``_MDTF_base`` is mandatory for the framework's operation. - For instance, the minimal installation for running the ``EOF_500hPa`` and ``convective_transition_diag PODs`` requres ``_MDTF_base`` (mandatory), ``_MDTF_NCL_base`` (because of b), and ``_MDTF_convective_transition_diag`` (because of 1). These can be installed by passing ``base``, ``NCL_base``, and ``convective_transition_diag`` to the ``--env`` flag one at a time (:ref:`ref-conda-install`). Testing with a new Conda environment ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ If you've updated an existing environment or created a new environment (with corresponding changes to the YAML file), verify that your POD works. Recall how the framework finds a proper Conda environment for a POD. First, it searches for an environment matching the POD's short name. If this fails, it then looks into the POD's ``settings.jsonc`` and prepares a generic environment depending on the language(s). Therefore, no additional steps are needed to specify the environment if your new YAML file follows the naming conventions above (in case of a new environment) or your ``settings.jsonc`` correctly lists the language(s) (in case of updating an existing environment). - For an updated environment, first, uninstall it by deleting the corresponding directory under ``$CONDA_ENV_DIR``. - Re-install the environment using the ``conda_env_setup.sh`` script as described in the :ref:`installation instructions `, or create the new environment for you POD: .. code-block:: console % cd $CODE_ROOT % ./src/conda/conda_env_setup.sh --env $your_POD_short_name --conda_root $CONDA_ROOT --env_dir $CONDA_ENV_DIR - Have the framework run your POD on suitable test data. 1. Add your POD's short name to the ``pod_list`` section of the configuration input file (template: ``src/default_tests.jsonc``). 2. Prepare the test data as described in :doc:`start_config`.