src.shared_diagnostic module

exception src.shared_diagnostic.PodRequirementFailure(pod, msg=None)[source]

Bases: Exception

Exception raised if POD doesn’t have required resoruces to run.

__init__(pod, msg=None)[source]

Initialize self. See help(type(self)) for accurate signature.

class src.shared_diagnostic.Diagnostic(pod_name, verbose=0)[source]

Bases: object

Class holding configuration for a diagnostic script.

This is the analogue of TestCase in the xUnit analogy.

Object attributes are read from entries in the settings section of the POD’s settings.json file upon initialization.

Attributes

  • driver (str) – Filename of the top-level driver script for the POD.

  • long_name (str) – POD’s name used for display purposes. May contain spaces.

  • description (str) – Short description of POD inserted by the link in the top-level index.html file.

  • required_programs (list of str, optional) – List of executables required by the POD (typically language interpreters). validate_environment.sh will make sure these are on the environment’s $PATH before the POD is run.

  • required_ncl_scripts (list of str, optional) – List of NCL scripts required by the POD, if any. validate_environment.sh will make sure these are on the environment’s $PATH before the POD is run.

__init__(pod_name, verbose=0)[source]

POD initializer. Given a POD name, we attempt to read a settings.json file in a subdirectory of /diagnostics by that name and parse the contents.

Parameters

  • pod_name (str) – Name of the POD to initialize.

  • verbose (int, optional) – Logging verbosity level. Default 0.

iter_vars_and_alts()[source]

Generator iterating over all variables and alternates in POD’s varlist.

_parse_pod_settings(settings, verbose=0)[source]

Private method called by __init__().

Parameters

  • settings (dict) – Contents of the settings portion of the POD’s settings.json file.

  • verbose (int, optional) – Logging verbosity level. Default 0.

Returns

Dict of parsed settings.

_parse_pod_varlist(varlist, verbose=0)[source]

Private method called by __init__().

Parameters

  • varlist (list of dict) – Contents of the varlist portion of the POD’s settings.json file.

  • verbose (int, optional) – Logging verbosity level. Default 0.

Returns

varlist

setUp(verbose=0)[source]

Perform filesystem operations and checks prior to running the POD.

In order, this 1) sets environment variables specific to the POD, 2) creates POD-specific working directories, and 3) checks for the existence of the POD’s driver script.

Note

The existence of data files is checked with data_manager.DataManager.fetchData() and the runtime environment is validated separately as a function of environment_manager.EnvironmentManager.run(). This is because each POD is run in a subprocess (due to the necessity of supporting multiple languages) so the validation must take place in that subprocess.

Raises: PodRequirementFailure if requirements

aren’t met. This is re-raised from the _check_pod_driver() and _check_for_varlist_files() subroutines.

_set_pod_env_vars(verbose=0)[source]

Private method called by setUp(). Sets all environment variables for POD.

Parameters

verbose (int, optional) – Logging verbosity level. Default 0.

_setup_pod_directories(verbose=0)[source]

Private method called by setUp().

Parameters

verbose (int, optional) – Logging verbosity level. Default 0.

_check_pod_driver(verbose=0)[source]

Private method called by setUp().

Parameters

verbose (int, optional) – Logging verbosity level. Default 0.

Raises: PodRequirementFailure if driver script

can’t be found.

_check_for_varlist_files(varlist, verbose=0)[source]

Verify that all data files needed by a POD exist locally.

Private method called by fetchData().

Parameters

  • varlist (list of dict) – Contents of the varlist portion of the POD’s settings.json file.

  • verbose (int, optional) – Logging verbosity level. Default 0.

Returns: tuple of found and missing file lists. Note that this is called

recursively.

run_commands()[source]

Produces the shell command(s) to run the POD. Called by environment_manager.EnvironmentManager.run().

Returns

Command-line invocation to run the POD.

Return type

(list of str)

validate_commands()[source]

Produces the shell command(s) to validate the POD’s runtime environment (ie, check for all requested third-party module dependencies.)

Called by environment_manager.EnvironmentManager.run(). Dependencies are passed as arguments to the shell script src/validate_environment.sh, which is invoked in the POD’s subprocess before the POD is run.

Returns

Command-line invocation to validate

the POD’s runtime environment.

Return type

(list of str)

tearDown(verbose=0)[source]

Performs cleanup tasks when the POD has finished running.

In order, this 1) creates the POD’s HTML output page from its included template, replacing CASENAME and other template variables with their current values, and adds a link to the POD’s page from the top-level HTML report; 2) converts the POD’s output plots (in PS or EPS vector format) to a bitmap format for webpage display; 3) Copies all requested files to the output directory and deletes temporary files.

Parameters

verbose (int, optional) – Logging verbosity level. Default 0.

make_pod_html()[source]

Perform templating on POD’s html results page(s).

A wrapper for append_html_template(). Looks for all html files in POD_CODE_DIR, templates them, and copies them to POD_WK_DIR, respecting subdirectory structure (see doc for recursive_copy()).

Update the top level index.html page with a link to this POD’s results.

This simply appends one of two html fragments to index.html: pod_result_snippet.html if the POD completed successfully, or pod_error_snippet.html if an exception was raised during the POD’s setup or execution.

Parameters

error (default None) – Exception object (if any) that was raised during POD’s attempted execution. If this is None, assume that POD ran successfully.

Check for missing files linked to from POD’s html page.

See documentation for LinkVerifier. This method calls LinkVerifier to check existence of all files linked to from the POD’s own top-level html page (after templating). If any files are missing, an error message listing them is written to the run’s index.html (located in src/html/pod_missing_snippet.html).

convert_pod_figures(src_subdir, dest_subdir)[source]

Convert all vector graphics in POD_WK_DIR/subdir to .png files using ghostscript.

All vector graphics files (identified by extension) in any subdirectory of POD_WK_DIR/src_subdir are converted to .png files by running ghostscript in a subprocess. Ghostscript is included in the _MDTF_base conda environment. Afterwards, any bitmap files (identified by extension) in any subdirectory of POD_WK_DIR/src_subdir are moved to POD_WK_DIR/dest_subdir, preserving and subdirectories (see doc for recursive_copy().)

Parameters

  • src_subdir – Subdirectory tree of POD_WK_DIR to search for vector graphics files.

  • dest_subdir – Subdirectory tree of POD_WK_DIR to move converted bitmap files to.

cleanup_pod_files()[source]

Copy and remove remaining files to POD_WK_DIR.

In order, this 1) copies .pdf documentation (if any) from POD_CODE_DIR/doc, 2) copies any bitmap figures in any subdirectory of POD_OBS_DATA to POD_WK_DIR/obs (needed for legacy PODs without digested observational data), 3) removes vector graphics if requested, 4) removes netCDF scratch files in POD_WK_DIR if requested.

Settings are set at runtime, when ConfigManager is initialized.