src.core module

Definition of the MDTF framework main loop and classes implementing basic, supporting functionality.

class src.core.ObjectStatus(value, names=None, *, module=None, qualname=None, type=None, start=1, boundary=None)

Bases: MDTFEnum

util.MDTFEnum used to track the status of an object hierarchy object (child class of MDTFObjectBase):

• NOTSET: the object hasn’t been fully initialized.

• ACTIVE: the object is currently being processed by the framework.

• INACTIVE: the object has been initialized, but isn’t being processed (e.g., alternate VarlistEntrys).

• FAILED: processing of the object has encountered an error, and no further work will be done.

• SUCCEEDED: Processing finished successfully.

NOTSET = 1

ACTIVE = 2

INACTIVE = 3

FAILED = 4

SUCCEEDED = 5

class src.core.MDTFObjectBase(*args, **kwargs)

Bases: object

Base class providing shared functionality for the object hierarchy, which is:

• The framework itself (MDTFFramework);

• DataSourceBases belonging to a run of the package;

• Diagnostics (PODs) belonging to a DataSourceBase;

• VarlistEntrys (requested model variables) belonging to a Diagnostic.

name: str = sentinel.Mandatory

status: ObjectStatus = 1

property full_name

property failed

property active

iter_children(child_type=None, status=None, status_neq=None)

Generator iterating over child objects associated with this object.

Parameters:

• child_type – None or Type type; default None. If None, iterates over

• type (all child objects regardless of their) –

• status – None or ObjectStatus, default None. If None, iterates over all child objects, regardless of status. If a ObjectStatus value is passed, only iterates over child objects with that status.

• status_neq – None or ObjectStatus, default None. If set, iterates over child objects which don’t have the given status. If status is set, this setting is ignored.

child_deactivation_handler(child, exc)

child_status_update(exc=None)

deactivate(exc, level=None)

class src.core.ConfigTuple(name, backup_filename, contents)

Bases: tuple

Class wrapping general structs used for configuration.

backup_filename

contents

name

class src.core.ConfigManager(*args, **kwargs)

Bases: Singleton, NameSpace

backup_config(cli_obj, case_d)

Copy serializable version of parsed settings, in order to write backup config file.

classmethod fromDict(x)

Recursively transforms a dictionary into a NameSpace via copy. (Note: as dicts are not hashable, they cannot be nested in sets/frozensets.)

fromkeys(value=None, /)

Create a new dictionary with keys from iterable and values set to value.

toDict()

Recursively converts a NameSpace back into a dictionary.

class src.core.PathManager(*args, **kwargs)

Bases: Singleton, NameSpace

Singleton holding the root directories for all paths used by the code.

model_paths(case, overwrite=False)

multirun_model_paths(pod, case)

pod_paths(pod, case)

classmethod fromDict(x)

Recursively transforms a dictionary into a NameSpace via copy. (Note: as dicts are not hashable, they cannot be nested in sets/frozensets.)

fromkeys(value=None, /)

Create a new dictionary with keys from iterable and values set to value.

toDict()

Recursively converts a NameSpace back into a dictionary.

class src.core.TempDirManager(*args, **kwargs)

Bases: Singleton

make_tempdir(hash_obj=None)

rm_tempdir(path)

cleanup()

tempdir_cleanup_handler(signum=None, frame=None)

class src.core.TranslatedVarlistEntry(coords: InitVar = None, name: str = sentinel.Mandatory, standard_name: str = sentinel.Mandatory, units: Units = sentinel.Mandatory, modifier: str = '', component: str = '', associated_files: str = '', rename_coords: bool = True, convention: str = sentinel.Mandatory, log: Any = sentinel.Mandatory)

Bases: DMVariable

Class returned by VarlistTranslator.translate(). Marks some attributes inherited from DMVariable as being queryable in query_dataset().

convention: str = sentinel.Mandatory

name: str = sentinel.Mandatory

standard_name: str = sentinel.Mandatory

units: Units = sentinel.Mandatory

scalar_coords: list

log: Any = sentinel.Mandatory

property T

Return T axis dimension coordinate if defined, else None.

property X

Return X axis dimension coordinate if defined, else None.

property Y

Return Y axis dimension coordinate if defined, else None.

property Z

Return Z axis dimension coordinate if defined, else None.

add_scalar(ax, ax_value, **kwargs)

Metadata operation corresponding to taking a slice of a higher-dimensional variable (extracting its values at axis ax = ax_value). The coordinate corresponding to ax is removed from the list of coordinate dimensions and added to the list of scalar coordinates.

associated_files: str = ''

property axes

Superset of the dim_axes() dict (whose values contain coordinate dimensions only) that includes axes corresponding to scalar coordinates.

property axes_set

Superset of the dim_axes_set() frozenset (which contains axes labels corresponding to coordinate dimensions only) that includes axes labels corresponding to scalar coordinates.

build_axes(*coords, verify=True)

Constructs a dict mapping axes labels to dimension coordinates (of type AbstractDMCoordinate.)

change_coord(ax_name, new_class=None, **kwargs)

Replace attributes on a given coordinate, but also optionally cast them to new classes.

Parameters:

• ax_name – Name of the coordinate to modify.

• new_class (optional) – new class to cast the returned coordinate to.

• kwargs – Set of attribute names and values to replace on the returned copy.

component: str = ''

coords: InitVar = None

property dim_axes

Retrun dict mapping axes labels (‘X’, ‘Y’, etc.) to corresponding dimension coordinate objects.

property dim_axes_set

Return frozenset of dimension coordinate axes labels.

property full_name

Object’s full name, to be used in logging and debugging. Preferred because it eliminates irrelevant information in repr(), which is lengthy.

get_scalar(ax_name)

If the axis label ax_name is a scalar coordinate, return the corresponding AbstractDMCoordinate object, otherwise return None.

property is_static

Whether the variable has time dependence (bool).

modifier: str = ''

remove_scalar(ax, position=-1, **kwargs)

Metadata operation that’s the inverse of add_scalar(). Given an axis label ax that’s currently a scalar coordinate, remove the slice value and add it to the list of dimension coordinates at position (default end of the list.)

rename_coords: bool = True

dims: list

class src.core.FieldlistEntry(coords: ~dataclasses.InitVar = None, name: str = sentinel.Mandatory, standard_name: str = sentinel.Mandatory, units: ~src.units.Units = '', modifier: str = '', component: str = '', associated_files: str = '', rename_coords: bool = True, scalar_coord_templates: dict = <factory>)

Bases: DMDependentVariable

Class corresponding to an entry in a fieldlist file.

scalar_coord_templates: dict

classmethod from_struct(dims_d, name, **kwargs)

scalar_name(old_coord, new_coord, log=<Logger>)

Uses one of the scalar_coord_templates to construct the translated variable name for this variable on a scalar coordinate slice (eg. pressure level).

property T

Return T axis dimension coordinate if defined, else None.

property X

Return X axis dimension coordinate if defined, else None.

property Y

Return Y axis dimension coordinate if defined, else None.

property Z

Return Z axis dimension coordinate if defined, else None.

add_scalar(ax, ax_value, **kwargs)

Metadata operation corresponding to taking a slice of a higher-dimensional variable (extracting its values at axis ax = ax_value). The coordinate corresponding to ax is removed from the list of coordinate dimensions and added to the list of scalar coordinates.

associated_files: str = ''

property axes

Superset of the dim_axes() dict (whose values contain coordinate dimensions only) that includes axes corresponding to scalar coordinates.

property axes_set

Superset of the dim_axes_set() frozenset (which contains axes labels corresponding to coordinate dimensions only) that includes axes labels corresponding to scalar coordinates.

build_axes(*coords, verify=True)

Constructs a dict mapping axes labels to dimension coordinates (of type AbstractDMCoordinate.)

change_coord(ax_name, new_class=None, **kwargs)

Replace attributes on a given coordinate, but also optionally cast them to new classes.

Parameters:

• ax_name – Name of the coordinate to modify.

• new_class (optional) – new class to cast the returned coordinate to.

• kwargs – Set of attribute names and values to replace on the returned copy.

component: str = ''

coords: dc.InitVar = None

property dim_axes

Retrun dict mapping axes labels (‘X’, ‘Y’, etc.) to corresponding dimension coordinate objects.

property dim_axes_set

Return frozenset of dimension coordinate axes labels.

property full_name

Object’s full name, to be used in logging and debugging. Preferred because it eliminates irrelevant information in repr(), which is lengthy.

get_scalar(ax_name)

If the axis label ax_name is a scalar coordinate, return the corresponding AbstractDMCoordinate object, otherwise return None.

property is_static

Whether the variable has time dependence (bool).

modifier: str = ''

name: str = sentinel.Mandatory

remove_scalar(ax, position=-1, **kwargs)

Metadata operation that’s the inverse of add_scalar(). Given an axis label ax that’s currently a scalar coordinate, remove the slice value and add it to the list of dimension coordinates at position (default end of the list.)

rename_coords: bool = True

standard_name: str = sentinel.Mandatory

units: src.units.Units = ''

dims: list

scalar_coords: list

class src.core.Fieldlist(name: str = sentinel.Mandatory, axes: ~src.util.basic.WormDict = <factory>, axes_lut: ~src.util.basic.WormDict = <factory>, entries: ~src.util.basic.WormDict = <factory>, lut: ~src.util.basic.WormDict = <factory>, env_vars: dict = <factory>)

Bases: object

Class corresponding to a single variable naming convention (single file in data/fieldlist_*.jsonc).

TODO: implement more robust indexing/lookup scheme. standard_name is not a unique identifier, but should include cell_methods, etc. as well as dimensionality.

name: str = sentinel.Mandatory

axes: WormDict

axes_lut: WormDict

entries: WormDict

lut: WormDict

env_vars: dict

classmethod from_struct(d)

to_CF(var_or_name)

Returns FieldlistEntry for the variable having the given name in this convention.

to_CF_name(var_or_name)

Like to_CF(), but only return the CF standard name, given the name in this convention.

from_CF(var_or_name, modifier=None, num_dims=0, has_scalar_coords_att=False, name_only=False)

Look up FieldlistEntry corresponding to the given standard name, optionally providing a modifier to resolve ambiguity.

TODO: this is a hacky implementation; FieldlistEntry needs to be expanded with more ways to uniquely identify variable (eg cell methods). :param var_or_name: variable or name of the variable :param modifier: optional string to distinguish a 3-D field from a 4-D field with :param the same var_or_name value: :param num_dims: number of dimensions of the POD variable corresponding to var_or_name :param has_scalar_coords_att: boolean indicating that the POD variable has a scalar_coords :param attribute: :param and therefore requires a level from a 4-D field: :param name_only: boolean indicating to not return a modifier–hacky way to accommodate :param a from_CF_name call that does not provide other metadata:

from_CF_name(var_or_name, modifier=None)

Like from_CF(), but only return the variable’s name in this convention.

Parameters:

• var_or_name – variable or name of the variable

• modifier – optional string to distinguish a 3-D field from a 4-D field with

• value (the same var_or_name) –

translate_coord(coord, log=<Logger>)

Given a DMCoordinate, look up the corresponding translated DMCoordinate in this convention.

translate(var)

Returns TranslatedVarlistEntry instance, with populated coordinate axes. Units of scalar coord slices are translated to the units of the conventions’ coordinates. Includes logic to translate and rename scalar coords/slices, e.g. VarlistEntry for ‘ua’ (intrinsically 4D) @ 500mb could produce a TranslatedVarlistEntry for ‘u500’ (3D slice), depending on naming convention.

class src.core.NoTranslationFieldlist(*args, **kwargs)

Bases: Singleton

Class which partially implements the Fieldlist interface but does no variable translation. VarlistEntry objects from the POD are passed through to create TranslatedVarlistEntry objects.

to_CF(var_or_name)

to_CF_name(var_or_name)

from_CF(var_or_name, modifier=None)

from_CF_name(var_or_name, modifier=None)

translate_coord(coord, log=<Logger>)

translate(var)

Returns TranslatedVarlistEntry instance, populated with contents of input VarlistEntry instance.

Note

We return a copy of the VarlistEntry because logic in DefaultDatasetParser alters the translation based on the file’s actual contents.

class src.core.VariableTranslator(*args, **kwargs)

Bases: Singleton

Singleton containing information for different variable naming conventions. These are defined in the data/fieldlist_*.jsonc files.

add_convention(d)

read_conventions(code_root, unittest=False)

Read in the conventions from the Fieldlists and populate the convention attribute.

get_convention_name(conv_name)

Resolve the naming convention associated with a given Fieldlist object from among a set of possible aliases.

get_convention(conv_name)

Return the Fieldlist object containing the variable name translation logic for a given convention name.

to_CF(conv_name, name)

to_CF_name(conv_name, name)

from_CF(conv_name, standard_name, modifier=None)

from_CF_name(conv_name, standard_name, modifier=None)

translate_coord(conv_name, coord, log=<Logger>)

translate(conv_name, var)

class src.core.MDTFFramework(*args, **kwargs)

Bases: MDTFObjectBase

property full_name

configure(cli_obj, pod_info_tuple, log_config)

Wrapper for all configuration done based on CLI arguments.

dispatch_classes(cli_obj)

parse_mdtf_args(cli_obj, pod_info_tuple)

Parse script options returned by the CLI. For greater customizability, most of the functionality is spun out into sub-methods.

parse_flags(cli_obj)

parse_env_vars(cli_obj)

parse_pod_list(pod_list, pod_info_tuple)

set_case_pod_list(case, cli_obj, pod_info_tuple)

parse_case(n, d, cli_obj, pod_info_tuple)

parse_case_list(cli_obj, pod_info_tuple)

verify_paths(config, p)

property active

child_deactivation_handler(child, exc)

child_status_update(exc=None)

deactivate(exc, level=None)

iter_children(child_type=None, status=None, status_neq=None)

Generator iterating over child objects associated with this object.

Parameters:

• child_type – None or Type type; default None. If None, iterates over

• type (all child objects regardless of their) –

• status – None or ObjectStatus, default None. If None, iterates over all child objects, regardless of status. If a ObjectStatus value is passed, only iterates over child objects with that status.

• status_neq – None or ObjectStatus, default None. If set, iterates over child objects which don’t have the given status. If status is set, this setting is ignored.

name: str = sentinel.Mandatory

status: ObjectStatus = 1

property failed

Overall success/failure of this run of the framework. Return True if any case or any POD has failed, else return False.

main()

src.core.print_summary(fmwk)

src.core.print_multirun_summary(fmwk)