src.core module

Definition of the MDTF framework main loop and classes implementing basic, supporting functionality.

class src.core.ObjectStatus(value, names=None, *, module=None, qualname=None, type=None, start=1, boundary=None)

Bases: MDTFEnum

util.MDTFEnum used to track the status of an object hierarchy object (child class of MDTFObjectBase):

  • NOTSET: the object hasn’t been fully initialized.

  • ACTIVE: the object is currently being processed by the framework.

  • INACTIVE: the object has been initialized, but isn’t being processed (e.g., alternate VarlistEntrys).

  • FAILED: processing of the object has encountered an error, and no further work will be done.

  • SUCCEEDED: Processing finished successfully.

NOTSET = 1
ACTIVE = 2
INACTIVE = 3
FAILED = 4
SUCCEEDED = 5
class src.core.MDTFObjectBase(*args, **kwargs)[source]

Bases: object

Base class providing shared functionality for the object hierarchy, which is:

  • The framework itself (MDTFFramework);

  • DataSourceBases belonging to a run of the package;

  • Diagnostics (PODs) belonging to a DataSourceBase;

  • VarlistEntrys (requested model variables) belonging to a Diagnostic.

name: str = sentinel.Mandatory
status: ObjectStatus = 1
property full_name
property failed
property active
iter_children(child_type=None, status=None, status_neq=None)[source]

Generator iterating over child objects associated with this object.

Parameters:
  • child_type – None or Type type; default None. If None, iterates over

  • type (all child objects regardless of their) –

  • status – None or ObjectStatus, default None. If None, iterates over all child objects, regardless of status. If a ObjectStatus value is passed, only iterates over child objects with that status.

  • status_neq – None or ObjectStatus, default None. If set, iterates over child objects which don’t have the given status. If status is set, this setting is ignored.

child_deactivation_handler(child, exc)[source]
child_status_update(exc=None)[source]
deactivate(exc, level=None)[source]
class src.core.ConfigTuple(name, backup_filename, contents)

Bases: tuple

Class wrapping general structs used for configuration.

backup_filename
contents
name
class src.core.ConfigManager(*args, **kwargs)[source]

Bases: Singleton, NameSpace

backup_config(cli_obj, case_d)[source]

Copy serializable version of parsed settings, in order to write backup config file.

classmethod fromDict(x)

Recursively transforms a dictionary into a NameSpace via copy. (Note: as dicts are not hashable, they cannot be nested in sets/frozensets.)

fromkeys(value=None, /)

Create a new dictionary with keys from iterable and values set to value.

toDict()

Recursively converts a NameSpace back into a dictionary.

class src.core.PathManager(*args, **kwargs)[source]

Bases: Singleton, NameSpace

Singleton holding the root directories for all paths used by the code.

model_paths(case, overwrite=False)[source]
multirun_model_paths(pod, case)[source]
pod_paths(pod, case)[source]
classmethod fromDict(x)

Recursively transforms a dictionary into a NameSpace via copy. (Note: as dicts are not hashable, they cannot be nested in sets/frozensets.)

fromkeys(value=None, /)

Create a new dictionary with keys from iterable and values set to value.

toDict()

Recursively converts a NameSpace back into a dictionary.

class src.core.TempDirManager(*args, **kwargs)[source]

Bases: Singleton

make_tempdir(hash_obj=None)[source]
rm_tempdir(path)[source]
cleanup()[source]
tempdir_cleanup_handler(signum=None, frame=None)[source]
class src.core.TranslatedVarlistEntry(coords: InitVar = None, name: str = sentinel.Mandatory, standard_name: str = sentinel.Mandatory, units: Units = sentinel.Mandatory, modifier: str = '', component: str = '', associated_files: str = '', rename_coords: bool = True, convention: str = sentinel.Mandatory, log: Any = sentinel.Mandatory)[source]

Bases: DMVariable

Class returned by VarlistTranslator.translate(). Marks some attributes inherited from DMVariable as being queryable in query_dataset().

convention: str = sentinel.Mandatory
name: str = sentinel.Mandatory
standard_name: str = sentinel.Mandatory
units: Units = sentinel.Mandatory
scalar_coords: list
log: Any = sentinel.Mandatory
property T

Return T axis dimension coordinate if defined, else None.

property X

Return X axis dimension coordinate if defined, else None.

property Y

Return Y axis dimension coordinate if defined, else None.

property Z

Return Z axis dimension coordinate if defined, else None.

add_scalar(ax, ax_value, **kwargs)

Metadata operation corresponding to taking a slice of a higher-dimensional variable (extracting its values at axis ax = ax_value). The coordinate corresponding to ax is removed from the list of coordinate dimensions and added to the list of scalar coordinates.

associated_files: str = ''
property axes

Superset of the dim_axes() dict (whose values contain coordinate dimensions only) that includes axes corresponding to scalar coordinates.

property axes_set

Superset of the dim_axes_set() frozenset (which contains axes labels corresponding to coordinate dimensions only) that includes axes labels corresponding to scalar coordinates.

build_axes(*coords, verify=True)

Constructs a dict mapping axes labels to dimension coordinates (of type AbstractDMCoordinate.)

change_coord(ax_name, new_class=None, **kwargs)

Replace attributes on a given coordinate, but also optionally cast them to new classes.

Parameters:
  • ax_name – Name of the coordinate to modify.

  • new_class (optional) – new class to cast the returned coordinate to.

  • kwargs – Set of attribute names and values to replace on the returned copy.

component: str = ''
coords: InitVar = None
property dim_axes

Retrun dict mapping axes labels (‘X’, ‘Y’, etc.) to corresponding dimension coordinate objects.

property dim_axes_set

Return frozenset of dimension coordinate axes labels.

property full_name

Object’s full name, to be used in logging and debugging. Preferred because it eliminates irrelevant information in repr(), which is lengthy.

get_scalar(ax_name)

If the axis label ax_name is a scalar coordinate, return the corresponding AbstractDMCoordinate object, otherwise return None.

property is_static

Whether the variable has time dependence (bool).

modifier: str = ''
remove_scalar(ax, position=-1, **kwargs)

Metadata operation that’s the inverse of add_scalar(). Given an axis label ax that’s currently a scalar coordinate, remove the slice value and add it to the list of dimension coordinates at position (default end of the list.)

rename_coords: bool = True
dims: list
class src.core.FieldlistEntry(coords: ~dataclasses.InitVar = None, name: str = sentinel.Mandatory, standard_name: str = sentinel.Mandatory, units: ~src.units.Units = '', modifier: str = '', component: str = '', associated_files: str = '', rename_coords: bool = True, scalar_coord_templates: dict = <factory>)[source]

Bases: DMDependentVariable

Class corresponding to an entry in a fieldlist file.

scalar_coord_templates: dict
classmethod from_struct(dims_d, name, **kwargs)[source]
scalar_name(old_coord, new_coord, log=<Logger>)[source]

Uses one of the scalar_coord_templates to construct the translated variable name for this variable on a scalar coordinate slice (eg. pressure level).

property T

Return T axis dimension coordinate if defined, else None.

property X

Return X axis dimension coordinate if defined, else None.

property Y

Return Y axis dimension coordinate if defined, else None.

property Z

Return Z axis dimension coordinate if defined, else None.

add_scalar(ax, ax_value, **kwargs)

Metadata operation corresponding to taking a slice of a higher-dimensional variable (extracting its values at axis ax = ax_value). The coordinate corresponding to ax is removed from the list of coordinate dimensions and added to the list of scalar coordinates.

associated_files: str = ''
property axes

Superset of the dim_axes() dict (whose values contain coordinate dimensions only) that includes axes corresponding to scalar coordinates.

property axes_set

Superset of the dim_axes_set() frozenset (which contains axes labels corresponding to coordinate dimensions only) that includes axes labels corresponding to scalar coordinates.

build_axes(*coords, verify=True)

Constructs a dict mapping axes labels to dimension coordinates (of type AbstractDMCoordinate.)

change_coord(ax_name, new_class=None, **kwargs)

Replace attributes on a given coordinate, but also optionally cast them to new classes.

Parameters:
  • ax_name – Name of the coordinate to modify.

  • new_class (optional) – new class to cast the returned coordinate to.

  • kwargs – Set of attribute names and values to replace on the returned copy.

component: str = ''
coords: dc.InitVar = None
property dim_axes

Retrun dict mapping axes labels (‘X’, ‘Y’, etc.) to corresponding dimension coordinate objects.

property dim_axes_set

Return frozenset of dimension coordinate axes labels.

property full_name

Object’s full name, to be used in logging and debugging. Preferred because it eliminates irrelevant information in repr(), which is lengthy.

get_scalar(ax_name)

If the axis label ax_name is a scalar coordinate, return the corresponding AbstractDMCoordinate object, otherwise return None.

property is_static

Whether the variable has time dependence (bool).

modifier: str = ''
name: str = sentinel.Mandatory
remove_scalar(ax, position=-1, **kwargs)

Metadata operation that’s the inverse of add_scalar(). Given an axis label ax that’s currently a scalar coordinate, remove the slice value and add it to the list of dimension coordinates at position (default end of the list.)

rename_coords: bool = True
standard_name: str = sentinel.Mandatory
units: src.units.Units = ''
dims: list
scalar_coords: list
class src.core.Fieldlist(name: str = sentinel.Mandatory, axes: ~src.util.basic.WormDict = <factory>, axes_lut: ~src.util.basic.WormDict = <factory>, entries: ~src.util.basic.WormDict = <factory>, lut: ~src.util.basic.WormDict = <factory>, env_vars: dict = <factory>)[source]

Bases: object

Class corresponding to a single variable naming convention (single file in data/fieldlist_*.jsonc).

TODO: implement more robust indexing/lookup scheme. standard_name is not a unique identifier, but should include cell_methods, etc. as well as dimensionality.

name: str = sentinel.Mandatory
axes: WormDict
axes_lut: WormDict
entries: WormDict
lut: WormDict
env_vars: dict
classmethod from_struct(d)[source]
to_CF(var_or_name)[source]

Returns FieldlistEntry for the variable having the given name in this convention.

to_CF_name(var_or_name)[source]

Like to_CF(), but only return the CF standard name, given the name in this convention.

from_CF(var_or_name, modifier=None, num_dims=0, has_scalar_coords_att=False, name_only=False)[source]

Look up FieldlistEntry corresponding to the given standard name, optionally providing a modifier to resolve ambiguity.

TODO: this is a hacky implementation; FieldlistEntry needs to be expanded with more ways to uniquely identify variable (eg cell methods). :param var_or_name: variable or name of the variable :param modifier: optional string to distinguish a 3-D field from a 4-D field with :param the same var_or_name value: :param num_dims: number of dimensions of the POD variable corresponding to var_or_name :param has_scalar_coords_att: boolean indicating that the POD variable has a scalar_coords :param attribute: :param and therefore requires a level from a 4-D field: :param name_only: boolean indicating to not return a modifier–hacky way to accommodate :param a from_CF_name call that does not provide other metadata:

from_CF_name(var_or_name, modifier=None)[source]

Like from_CF(), but only return the variable’s name in this convention.

Parameters:
  • var_or_name – variable or name of the variable

  • modifier – optional string to distinguish a 3-D field from a 4-D field with

  • value (the same var_or_name) –

translate_coord(coord, log=<Logger>)[source]

Given a DMCoordinate, look up the corresponding translated DMCoordinate in this convention.

translate(var)[source]

Returns TranslatedVarlistEntry instance, with populated coordinate axes. Units of scalar coord slices are translated to the units of the conventions’ coordinates. Includes logic to translate and rename scalar coords/slices, e.g. VarlistEntry for ‘ua’ (intrinsically 4D) @ 500mb could produce a TranslatedVarlistEntry for ‘u500’ (3D slice), depending on naming convention.

class src.core.NoTranslationFieldlist(*args, **kwargs)[source]

Bases: Singleton

Class which partially implements the Fieldlist interface but does no variable translation. VarlistEntry objects from the POD are passed through to create TranslatedVarlistEntry objects.

to_CF(var_or_name)[source]
to_CF_name(var_or_name)[source]
from_CF(var_or_name, modifier=None)[source]
from_CF_name(var_or_name, modifier=None)[source]
translate_coord(coord, log=<Logger>)[source]
translate(var)[source]

Returns TranslatedVarlistEntry instance, populated with contents of input VarlistEntry instance.

Note

We return a copy of the VarlistEntry because logic in DefaultDatasetParser alters the translation based on the file’s actual contents.

class src.core.VariableTranslator(*args, **kwargs)[source]

Bases: Singleton

Singleton containing information for different variable naming conventions. These are defined in the data/fieldlist_*.jsonc files.

add_convention(d)[source]
read_conventions(code_root, unittest=False)[source]

Read in the conventions from the Fieldlists and populate the convention attribute.

get_convention_name(conv_name)[source]

Resolve the naming convention associated with a given Fieldlist object from among a set of possible aliases.

get_convention(conv_name)[source]

Return the Fieldlist object containing the variable name translation logic for a given convention name.

to_CF(conv_name, name)[source]
to_CF_name(conv_name, name)[source]
from_CF(conv_name, standard_name, modifier=None)[source]
from_CF_name(conv_name, standard_name, modifier=None)[source]
translate_coord(conv_name, coord, log=<Logger>)[source]
translate(conv_name, var)[source]
class src.core.MDTFFramework(*args, **kwargs)[source]

Bases: MDTFObjectBase

property full_name
configure(cli_obj, pod_info_tuple, log_config)[source]

Wrapper for all configuration done based on CLI arguments.

dispatch_classes(cli_obj)[source]
parse_mdtf_args(cli_obj, pod_info_tuple)[source]

Parse script options returned by the CLI. For greater customizability, most of the functionality is spun out into sub-methods.

parse_flags(cli_obj)[source]
parse_env_vars(cli_obj)[source]
parse_pod_list(pod_list, pod_info_tuple)[source]
set_case_pod_list(case, cli_obj, pod_info_tuple)[source]
parse_case(n, d, cli_obj, pod_info_tuple)[source]
parse_case_list(cli_obj, pod_info_tuple)[source]
verify_paths(config, p)[source]
property active
child_deactivation_handler(child, exc)
child_status_update(exc=None)
deactivate(exc, level=None)
iter_children(child_type=None, status=None, status_neq=None)

Generator iterating over child objects associated with this object.

Parameters:
  • child_type – None or Type type; default None. If None, iterates over

  • type (all child objects regardless of their) –

  • status – None or ObjectStatus, default None. If None, iterates over all child objects, regardless of status. If a ObjectStatus value is passed, only iterates over child objects with that status.

  • status_neq – None or ObjectStatus, default None. If set, iterates over child objects which don’t have the given status. If status is set, this setting is ignored.

name: str = sentinel.Mandatory
status: ObjectStatus = 1
property failed

Overall success/failure of this run of the framework. Return True if any case or any POD has failed, else return False.

main()[source]
src.core.print_summary(fmwk)[source]
src.core.print_multirun_summary(fmwk)[source]