src.core module

Definition of the MDTF framework main loop and classes implementing basic, supporting functionality.

class src.core.ObjectStatus(value)

Bases: src.util.basic.MDTFEnum

util.MDTFEnum used to track the status of an object hierarchy object (child class of MDTFObjectBase):

  • NOTSET: the object hasn’t been fully initialized.

  • ACTIVE: the object is currently being processed by the framework.

  • INACTIVE: the object has been initialized, but isn’t being processed (e.g., alternate VarlistEntrys).

  • FAILED: processing of the object has encountered an error, and no further work will be done.

  • SUCCEEDED: Processing finished successfully.

NOTSET = 1
ACTIVE = 2
INACTIVE = 3
FAILED = 4
SUCCEEDED = 5
class src.core.MDTFObjectBase(*args, **kwargs)[source]

Bases: object

Base class providing shared functionality for the object hierarchy, which is:

  • The framework itself (MDTFFramework);

  • DataSourceBases belonging to a run of the package;

  • Diagnostics (PODs) belonging to a DataSourceBase;

  • VarlistEntrys (requested model variables) belonging to a Diagnostic.

name: str = sentinel.Mandatory
status: src.core.ObjectStatus = 1
__post_init__()[source]
property full_name
property failed
property active
iter_children(child_type=None, status=None, status_neq=None)[source]

Generator iterating over child objects associated with this object.

Parameters
  • status – None or ObjectStatus, default None. If None, iterates over all child objects, regardless of status. If a ObjectStatus value is passed, only iterates over child objects with that status.

  • status_neq – None or ObjectStatus, default None. If set, iterates over child objects which don’t have the given status. If status is set, this setting is ignored.

child_deactivation_handler(child, exc)[source]
child_status_update(exc=None)[source]
deactivate(exc, level=None)[source]
__init__(_id: src.util.basic.MDTF_ID = None, name: str = sentinel.Mandatory, _parent: Any = sentinel.Mandatory, status: src.core.ObjectStatus = <ObjectStatus.NOTSET>)None

Initialize self. See help(type(self)) for accurate signature.

class src.core.ConfigTuple(name, backup_filename, contents)

Bases: tuple

Class wrapping general structs used for configuration.

__init__()

Initialize self. See help(type(self)) for accurate signature.

property backup_filename
property contents
property name
class src.core.ConfigManager(*args, **kwargs)[source]

Bases: src.util.basic.Singleton, src.util.basic.NameSpace

__init__(cli_obj=None, pod_info_tuple=None, global_env_vars=None, case_d=None, log_config=None, unittest=False)[source]

Initialize self. See help(type(self)) for accurate signature.

backup_config(cli_obj, case_d)[source]

Copy serializable version of parsed settings, in order to write backup config file.

classmethod fromDict(x)

Recursively transforms a dictionary into a NameSpace via copy. (Note: as dicts are not hashable, they cannot be nested in sets/frozensets.)

fromkeys(value=None, /)

Create a new dictionary with keys from iterable and values set to value.

toDict()

Recursively converts a NameSpace back into a dictionary.

class src.core.PathManager(*args, **kwargs)[source]

Bases: src.util.basic.Singleton, src.util.basic.NameSpace

Singleton holding the root directories for all paths used by the code.

__init__(cli_obj=None, env_vars=None, unittest=False)[source]

Initialize self. See help(type(self)) for accurate signature.

model_paths(case, overwrite=False)[source]
pod_paths(pod, case)[source]
classmethod fromDict(x)

Recursively transforms a dictionary into a NameSpace via copy. (Note: as dicts are not hashable, they cannot be nested in sets/frozensets.)

fromkeys(value=None, /)

Create a new dictionary with keys from iterable and values set to value.

toDict()

Recursively converts a NameSpace back into a dictionary.

class src.core.TempDirManager(*args, **kwargs)[source]

Bases: src.util.basic.Singleton

__init__(temp_root=None, unittest=False)[source]

Initialize self. See help(type(self)) for accurate signature.

make_tempdir(hash_obj=None)[source]
rm_tempdir(path)[source]
cleanup()[source]
tempdir_cleanup_handler(signum=None, frame=None)[source]
class src.core.TranslatedVarlistEntry(coords: dataclasses.InitVar = None, name: str = sentinel.Mandatory, standard_name: str = sentinel.Mandatory, units: src.units.Units = sentinel.Mandatory, modifier: str = '', convention: str = sentinel.Mandatory, log: Any = sentinel.Mandatory)[source]

Bases: src.data_model.DMVariable

Class returned by VarlistTranslator.translate(). Marks some attributes inherited from DMVariable as being queryable in query_dataset().

convention: str = sentinel.Mandatory
name: str = sentinel.Mandatory
standard_name: str = sentinel.Mandatory
units: src.units.Units = sentinel.Mandatory
scalar_coords: list
log: Any = sentinel.Mandatory
property T

Return T axis dimension coordinate if defined, else None.

property X

Return X axis dimension coordinate if defined, else None.

property Y

Return Y axis dimension coordinate if defined, else None.

property Z

Return Z axis dimension coordinate if defined, else None.

__init__(coords: dataclasses.InitVar = None, name: str = sentinel.Mandatory, standard_name: str = sentinel.Mandatory, units: src.units.Units = sentinel.Mandatory, modifier: str = '', convention: str = sentinel.Mandatory, log: Any = sentinel.Mandatory)None

Initialize self. See help(type(self)) for accurate signature.

__post_init__(coords=None)
add_scalar(ax, ax_value, **kwargs)

Metadata operation corresponding to taking a slice of a higher-dimensional variable (extracting its values at axis ax = ax_value). The coordinate corresponding to ax is removed from the list of coordinate dimensions and added to the list of scalar coordinates.

property axes

Superset of the dim_axes() dict (whose values contain coordinate dimensions only) that includes axes corresponding to scalar coordinates.

property axes_set

Superset of the dim_axes_set() frozenset (which contains axes labels corresponding to coordinate dimensions only) that includes axes labels corresponding to scalar coordinates.

build_axes(*coords, verify=True)

Constructs a dict mapping axes labels to dimension coordinates (of type AbstractDMCoordinate.)

change_coord(ax_name, new_class=None, **kwargs)

Replace attributes on a given coordinate, but also optionally cast them to new classes.

Parameters
  • ax_name – Name of the coodinate to modify.

  • new_class (optional) – new class to cast the returned coordinate to.

  • kwargs – Set of attribute names and values to replace on the returned copy.

coords = None
property dim_axes

Retrun dict mapping axes labels (‘X’, ‘Y’, etc.) to corresponding dimension coordinate objects.

property dim_axes_set

Return frozenset of dimension coordinate axes labels.

property full_name

Object’s full name, to be used in logging and debugging. Preferred because it eliminates irrelevant information in repr(), which is lengthy.

get_scalar(ax_name)

If the axis label ax_name is a scalar coordinate, return the corresponding AbstractDMCoordinate object, otherwise return None.

property is_static

Whether the variable has time dependence (bool).

modifier = ''
remove_scalar(ax, position=- 1, **kwargs)

Metadata operation that’s the inverse of add_scalar(). Given an axis label ax that’s currently a scalar coordinate, remove the slice value and add it to the list of dimension coordinates at position (default end of the list.)

class src.core.FieldlistEntry(coords: dataclasses.InitVar = None, name: str = sentinel.Mandatory, standard_name: str = sentinel.Mandatory, units: src.units.Units = '', modifier: str = '', scalar_coord_templates: dict = <factory>)[source]

Bases: src.data_model.DMDependentVariable

Class corresponding to an entry in a fieldlist file.

scalar_coord_templates: dict
__post_init__(coords=None)[source]
classmethod from_struct(dims_d, name, **kwargs)[source]
scalar_name(old_coord, new_coord, log=<Logger>)[source]

Uses one of the scalar_coord_templates to construct the translated variable name for this variable on a scalar coordinate slice (eg. pressure level).

property T

Return T axis dimension coordinate if defined, else None.

property X

Return X axis dimension coordinate if defined, else None.

property Y

Return Y axis dimension coordinate if defined, else None.

property Z

Return Z axis dimension coordinate if defined, else None.

__init__(coords: dataclasses.InitVar = None, name: str = sentinel.Mandatory, standard_name: str = sentinel.Mandatory, units: src.units.Units = '', modifier: str = '', scalar_coord_templates: dict = <factory>)None

Initialize self. See help(type(self)) for accurate signature.

add_scalar(ax, ax_value, **kwargs)

Metadata operation corresponding to taking a slice of a higher-dimensional variable (extracting its values at axis ax = ax_value). The coordinate corresponding to ax is removed from the list of coordinate dimensions and added to the list of scalar coordinates.

property axes

Superset of the dim_axes() dict (whose values contain coordinate dimensions only) that includes axes corresponding to scalar coordinates.

property axes_set

Superset of the dim_axes_set() frozenset (which contains axes labels corresponding to coordinate dimensions only) that includes axes labels corresponding to scalar coordinates.

build_axes(*coords, verify=True)

Constructs a dict mapping axes labels to dimension coordinates (of type AbstractDMCoordinate.)

change_coord(ax_name, new_class=None, **kwargs)

Replace attributes on a given coordinate, but also optionally cast them to new classes.

Parameters
  • ax_name – Name of the coodinate to modify.

  • new_class (optional) – new class to cast the returned coordinate to.

  • kwargs – Set of attribute names and values to replace on the returned copy.

coords = None
property dim_axes

Retrun dict mapping axes labels (‘X’, ‘Y’, etc.) to corresponding dimension coordinate objects.

property dim_axes_set

Return frozenset of dimension coordinate axes labels.

property full_name

Object’s full name, to be used in logging and debugging. Preferred because it eliminates irrelevant information in repr(), which is lengthy.

get_scalar(ax_name)

If the axis label ax_name is a scalar coordinate, return the corresponding AbstractDMCoordinate object, otherwise return None.

property is_static

Whether the variable has time dependence (bool).

modifier = ''
name = sentinel.Mandatory
remove_scalar(ax, position=- 1, **kwargs)

Metadata operation that’s the inverse of add_scalar(). Given an axis label ax that’s currently a scalar coordinate, remove the slice value and add it to the list of dimension coordinates at position (default end of the list.)

standard_name = sentinel.Mandatory
units = ''
class src.core.Fieldlist(name: str = sentinel.Mandatory, axes: src.util.basic.WormDict = <factory>, axes_lut: src.util.basic.WormDict = <factory>, entries: src.util.basic.WormDict = <factory>, lut: src.util.basic.WormDict = <factory>, env_vars: dict = <factory>)[source]

Bases: object

Class corresponding to a single variable naming convention (single file in data/fieldlist_*.jsonc).

TODO: implement more robust indexing/lookup scheme. standard_name is not a unique identifier, but should include cell_methods, etc. as well as dimensionality.

name: str = sentinel.Mandatory
axes: util.WormDict
axes_lut: util.WormDict
entries: util.WormDict
lut: util.WormDict
env_vars: dict
classmethod from_struct(d)[source]
to_CF(var_or_name)[source]

Returns FieldlistEntry for the variable having the given name in this convention.

to_CF_name(var_or_name)[source]

Like to_CF(), but only return the CF standard name, given the name in this convention.

from_CF(var_or_name, modifier=None, num_dims=0)[source]

Look up FieldlistEntry corresponding to the given standard name, optionally providing a modifier to resolve ambiguity.

TODO: this is a hacky implementation; FieldlistEntry needs to be expanded with more ways to uniquely identify variable (eg cell methods).

from_CF_name(var_or_name, modifier=None)[source]

Like from_CF(), but only return the variable’s name in this convention.

translate_coord(coord, log=<Logger>)[source]

Given a DMCoordinate, look up the corresponding translated DMCoordinate in this convention.

translate(var)[source]

Returns TranslatedVarlistEntry instance, with populated coordinate axes. Units of scalar coord slices are translated to the units of the conventions’ coordinates. Includes logic to translate and rename scalar coords/slices, e.g. VarlistEntry for ‘ua’ (intrinsically 4D) @ 500mb could produce a TranslatedVarlistEntry for ‘u500’ (3D slice), depending on naming convention.

__init__(name: str = sentinel.Mandatory, axes: src.util.basic.WormDict = <factory>, axes_lut: src.util.basic.WormDict = <factory>, entries: src.util.basic.WormDict = <factory>, lut: src.util.basic.WormDict = <factory>, env_vars: dict = <factory>)None

Initialize self. See help(type(self)) for accurate signature.

__post_init__(*args, **kwargs)
class src.core.NoTranslationFieldlist(*args, **kwargs)[source]

Bases: src.util.basic.Singleton

Class which partially implements the Fieldlist interface but does no variable translation. VarlistEntry objects from the POD are passed through to create TranslatedVarlistEntry objects.

__init__()[source]

Initialize self. See help(type(self)) for accurate signature.

to_CF(var_or_name)[source]
to_CF_name(var_or_name)[source]
from_CF(var_or_name, modifier=None)[source]
from_CF_name(var_or_name, modifier=None)[source]
translate_coord(coord, log=<Logger>)[source]
translate(var)[source]

Returns TranslatedVarlistEntry instance, populated with contents of input VarlistEntry instance.

Note

We return a copy of the VarlistEntry because logic in DefaultDatasetParser alters the translation based on the file’s actual contents.

class src.core.VariableTranslator(*args, **kwargs)[source]

Bases: src.util.basic.Singleton

Singleton containing information for different variable naming conventions. These are defined in the data/fieldlist_*.jsonc files.

__init__(code_root=None, unittest=False)[source]

Initialize self. See help(type(self)) for accurate signature.

add_convention(d)[source]
read_conventions(code_root, unittest=False)[source]

Read in the conventions from the Fieldlists and populate the convention attribute.

get_convention_name(conv_name)[source]

Resolve the naming convention associated with a given Fieldlist object from among a set of possible aliases.

get_convention(conv_name)[source]

Return the Fieldlist object containing the variable name translation logic for a given convention name.

to_CF(conv_name, name)[source]
to_CF_name(conv_name, name)[source]
from_CF(conv_name, standard_name, modifier=None)[source]
from_CF_name(conv_name, standard_name, modifier=None)[source]
translate_coord(conv_name, coord, log=<Logger>)[source]
translate(conv_name, var)[source]
class src.core.MDTFFramework(*args, **kwargs)[source]

Bases: src.core.MDTFObjectBase

__init__(cli_obj)[source]

Initialize self. See help(type(self)) for accurate signature.

property full_name
configure(cli_obj, pod_info_tuple, log_config)[source]

Wrapper for all configuration done based on CLI arguments.

dispatch_classes(cli_obj)[source]
parse_mdtf_args(cli_obj, pod_info_tuple)[source]

Parse script options returned by the CLI. For greater customizability, most of the functionality is spun out into sub-methods.

parse_flags(cli_obj)[source]
parse_env_vars(cli_obj)[source]
parse_pod_list(pod_list, pod_info_tuple)[source]
set_case_pod_list(case, cli_obj, pod_info_tuple)[source]
parse_case(n, d, cli_obj, pod_info_tuple)[source]
parse_case_list(cli_obj, pod_info_tuple)[source]
verify_paths(config, p)[source]
__post_init__()
property active
child_deactivation_handler(child, exc)
child_status_update(exc=None)
deactivate(exc, level=None)
iter_children(child_type=None, status=None, status_neq=None)

Generator iterating over child objects associated with this object.

Parameters
  • status – None or ObjectStatus, default None. If None, iterates over all child objects, regardless of status. If a ObjectStatus value is passed, only iterates over child objects with that status.

  • status_neq – None or ObjectStatus, default None. If set, iterates over child objects which don’t have the given status. If status is set, this setting is ignored.

name: str = sentinel.Mandatory
status: src.core.ObjectStatus = 1
property failed

Overall success/failure of this run of the framework. Return True if any case or any POD has failed, else return False.

main()[source]
src.core.print_summary(fmwk)[source]