src.core module¶
Definition of the MDTF framework main loop and classes implementing basic, supporting functionality.
-
class
src.core.
ObjectStatus
(value)¶ Bases:
src.util.basic.MDTFEnum
util.MDTFEnum
used to track the status of an object hierarchy object (child class ofMDTFObjectBase
):NOTSET: the object hasn’t been fully initialized.
ACTIVE: the object is currently being processed by the framework.
INACTIVE: the object has been initialized, but isn’t being processed (e.g., alternate
VarlistEntry
s).FAILED: processing of the object has encountered an error, and no further work will be done.
SUCCEEDED: Processing finished successfully.
-
NOTSET
= 1¶
-
ACTIVE
= 2¶
-
INACTIVE
= 3¶
-
FAILED
= 4¶
-
SUCCEEDED
= 5¶
-
class
src.core.
MDTFObjectBase
(*args, **kwargs)[source]¶ Bases:
object
Base class providing shared functionality for the object hierarchy, which is:
The framework itself (
MDTFFramework
);DataSourceBase
s belonging to a run of the package;Diagnostic
s (PODs) belonging to aDataSourceBase
;VarlistEntry
s (requested model variables) belonging to aDiagnostic
.
-
status
: src.core.ObjectStatus = 1¶
-
property
full_name
¶
-
property
failed
¶
-
property
active
¶
-
iter_children
(child_type=None, status=None, status_neq=None)[source]¶ Generator iterating over child objects associated with this object.
- Parameters
status – None or
ObjectStatus
, default None. If None, iterates over all child objects, regardless of status. If aObjectStatus
value is passed, only iterates over child objects with that status.status_neq – None or
ObjectStatus
, default None. If set, iterates over child objects which don’t have the given status. If status is set, this setting is ignored.
-
class
src.core.
ConfigTuple
(name, backup_filename, contents)¶ Bases:
tuple
Class wrapping general structs used for configuration.
-
__init__
()¶ Initialize self. See help(type(self)) for accurate signature.
-
property
backup_filename
¶
-
property
contents
¶
-
property
name
¶
-
-
class
src.core.
ConfigManager
(*args, **kwargs)[source]¶ Bases:
src.util.basic.Singleton
,src.util.basic.NameSpace
-
__init__
(cli_obj=None, pod_info_tuple=None, global_env_vars=None, case_d=None, log_config=None, unittest=False)[source]¶ Initialize self. See help(type(self)) for accurate signature.
-
backup_config
(cli_obj, case_d)[source]¶ Copy serializable version of parsed settings, in order to write backup config file.
-
classmethod
fromDict
(x)¶ Recursively transforms a dictionary into a NameSpace via copy. (Note: as dicts are not hashable, they cannot be nested in sets/frozensets.)
-
fromkeys
(value=None, /)¶ Create a new dictionary with keys from iterable and values set to value.
-
toDict
()¶ Recursively converts a NameSpace back into a dictionary.
-
-
class
src.core.
PathManager
(*args, **kwargs)[source]¶ Bases:
src.util.basic.Singleton
,src.util.basic.NameSpace
Singleton
holding the root directories for all paths used by the code.-
__init__
(cli_obj=None, env_vars=None, unittest=False)[source]¶ Initialize self. See help(type(self)) for accurate signature.
-
classmethod
fromDict
(x)¶ Recursively transforms a dictionary into a NameSpace via copy. (Note: as dicts are not hashable, they cannot be nested in sets/frozensets.)
-
fromkeys
(value=None, /)¶ Create a new dictionary with keys from iterable and values set to value.
-
toDict
()¶ Recursively converts a NameSpace back into a dictionary.
-
-
class
src.core.
TempDirManager
(*args, **kwargs)[source]¶ Bases:
src.util.basic.Singleton
-
class
src.core.
TranslatedVarlistEntry
(coords: dataclasses.InitVar = None, name: str = sentinel.Mandatory, standard_name: str = sentinel.Mandatory, units: src.units.Units = sentinel.Mandatory, modifier: str = '', convention: str = sentinel.Mandatory, log: Any = sentinel.Mandatory)[source]¶ Bases:
src.data_model.DMVariable
Class returned by
VarlistTranslator.translate()
. Marks some attributes inherited fromDMVariable
as being queryable inquery_dataset()
.-
units
: src.units.Units = sentinel.Mandatory¶
-
log
: Any = sentinel.Mandatory¶
-
property
T
¶ Return T axis dimension coordinate if defined, else None.
-
property
X
¶ Return X axis dimension coordinate if defined, else None.
-
property
Y
¶ Return Y axis dimension coordinate if defined, else None.
-
property
Z
¶ Return Z axis dimension coordinate if defined, else None.
-
__init__
(coords: dataclasses.InitVar = None, name: str = sentinel.Mandatory, standard_name: str = sentinel.Mandatory, units: src.units.Units = sentinel.Mandatory, modifier: str = '', convention: str = sentinel.Mandatory, log: Any = sentinel.Mandatory) → None¶ Initialize self. See help(type(self)) for accurate signature.
-
__post_init__
(coords=None)¶
-
add_scalar
(ax, ax_value, **kwargs)¶ Metadata operation corresponding to taking a slice of a higher-dimensional variable (extracting its values at axis ax = ax_value). The coordinate corresponding to ax is removed from the list of coordinate dimensions and added to the list of scalar coordinates.
-
property
axes
¶ Superset of the
dim_axes()
dict (whose values contain coordinate dimensions only) that includes axes corresponding to scalar coordinates.
-
property
axes_set
¶ Superset of the
dim_axes_set()
frozenset (which contains axes labels corresponding to coordinate dimensions only) that includes axes labels corresponding to scalar coordinates.
-
build_axes
(*coords, verify=True)¶ Constructs a dict mapping axes labels to dimension coordinates (of type
AbstractDMCoordinate
.)
-
change_coord
(ax_name, new_class=None, **kwargs)¶ Replace attributes on a given coordinate, but also optionally cast them to new classes.
- Parameters
ax_name – Name of the coodinate to modify.
new_class (optional) – new class to cast the returned coordinate to.
kwargs – Set of attribute names and values to replace on the returned copy.
-
coords
= None¶
-
property
dim_axes
¶ Retrun dict mapping axes labels (‘X’, ‘Y’, etc.) to corresponding dimension coordinate objects.
-
property
dim_axes_set
¶ Return frozenset of dimension coordinate axes labels.
-
property
full_name
¶ Object’s full name, to be used in logging and debugging. Preferred because it eliminates irrelevant information in repr(), which is lengthy.
-
get_scalar
(ax_name)¶ If the axis label ax_name is a scalar coordinate, return the corresponding
AbstractDMCoordinate
object, otherwise return None.
-
property
is_static
¶ Whether the variable has time dependence (bool).
-
modifier
= ''¶
-
remove_scalar
(ax, position=- 1, **kwargs)¶ Metadata operation that’s the inverse of
add_scalar()
. Given an axis label ax that’s currently a scalar coordinate, remove the slice value and add it to the list of dimension coordinates at position (default end of the list.)
-
-
class
src.core.
FieldlistEntry
(coords: dataclasses.InitVar = None, name: str = sentinel.Mandatory, standard_name: str = sentinel.Mandatory, units: src.units.Units = '', modifier: str = '', scalar_coord_templates: dict = <factory>)[source]¶ Bases:
src.data_model.DMDependentVariable
Class corresponding to an entry in a fieldlist file.
-
scalar_name
(old_coord, new_coord, log=<Logger>)[source]¶ Uses one of the scalar_coord_templates to construct the translated variable name for this variable on a scalar coordinate slice (eg. pressure level).
-
property
T
¶ Return T axis dimension coordinate if defined, else None.
-
property
X
¶ Return X axis dimension coordinate if defined, else None.
-
property
Y
¶ Return Y axis dimension coordinate if defined, else None.
-
property
Z
¶ Return Z axis dimension coordinate if defined, else None.
-
__init__
(coords: dataclasses.InitVar = None, name: str = sentinel.Mandatory, standard_name: str = sentinel.Mandatory, units: src.units.Units = '', modifier: str = '', scalar_coord_templates: dict = <factory>) → None¶ Initialize self. See help(type(self)) for accurate signature.
-
add_scalar
(ax, ax_value, **kwargs)¶ Metadata operation corresponding to taking a slice of a higher-dimensional variable (extracting its values at axis ax = ax_value). The coordinate corresponding to ax is removed from the list of coordinate dimensions and added to the list of scalar coordinates.
-
property
axes
¶ Superset of the
dim_axes()
dict (whose values contain coordinate dimensions only) that includes axes corresponding to scalar coordinates.
-
property
axes_set
¶ Superset of the
dim_axes_set()
frozenset (which contains axes labels corresponding to coordinate dimensions only) that includes axes labels corresponding to scalar coordinates.
-
build_axes
(*coords, verify=True)¶ Constructs a dict mapping axes labels to dimension coordinates (of type
AbstractDMCoordinate
.)
-
change_coord
(ax_name, new_class=None, **kwargs)¶ Replace attributes on a given coordinate, but also optionally cast them to new classes.
- Parameters
ax_name – Name of the coodinate to modify.
new_class (optional) – new class to cast the returned coordinate to.
kwargs – Set of attribute names and values to replace on the returned copy.
-
coords
= None¶
-
property
dim_axes
¶ Retrun dict mapping axes labels (‘X’, ‘Y’, etc.) to corresponding dimension coordinate objects.
-
property
dim_axes_set
¶ Return frozenset of dimension coordinate axes labels.
-
property
full_name
¶ Object’s full name, to be used in logging and debugging. Preferred because it eliminates irrelevant information in repr(), which is lengthy.
-
get_scalar
(ax_name)¶ If the axis label ax_name is a scalar coordinate, return the corresponding
AbstractDMCoordinate
object, otherwise return None.
-
property
is_static
¶ Whether the variable has time dependence (bool).
-
modifier
= ''¶
-
name
= sentinel.Mandatory¶
-
remove_scalar
(ax, position=- 1, **kwargs)¶ Metadata operation that’s the inverse of
add_scalar()
. Given an axis label ax that’s currently a scalar coordinate, remove the slice value and add it to the list of dimension coordinates at position (default end of the list.)
-
standard_name
= sentinel.Mandatory¶
-
units
= ''¶
-
-
class
src.core.
Fieldlist
(name: str = sentinel.Mandatory, axes: src.util.basic.WormDict = <factory>, axes_lut: src.util.basic.WormDict = <factory>, entries: src.util.basic.WormDict = <factory>, lut: src.util.basic.WormDict = <factory>, env_vars: dict = <factory>)[source]¶ Bases:
object
Class corresponding to a single variable naming convention (single file in data/fieldlist_*.jsonc).
TODO: implement more robust indexing/lookup scheme. standard_name is not a unique identifier, but should include cell_methods, etc. as well as dimensionality.
-
axes
: util.WormDict¶
-
axes_lut
: util.WormDict¶
-
entries
: util.WormDict¶
-
lut
: util.WormDict¶
-
to_CF
(var_or_name)[source]¶ Returns
FieldlistEntry
for the variable having the given name in this convention.
-
to_CF_name
(var_or_name)[source]¶ Like
to_CF()
, but only return the CF standard name, given the name in this convention.
-
from_CF
(var_or_name, modifier=None, num_dims=0)[source]¶ Look up
FieldlistEntry
corresponding to the given standard name, optionally providing a modifier to resolve ambiguity.TODO: this is a hacky implementation; FieldlistEntry needs to be expanded with more ways to uniquely identify variable (eg cell methods).
-
from_CF_name
(var_or_name, modifier=None)[source]¶ Like
from_CF()
, but only return the variable’s name in this convention.
-
translate_coord
(coord, log=<Logger>)[source]¶ Given a
DMCoordinate
, look up the corresponding translatedDMCoordinate
in this convention.
-
translate
(var)[source]¶ Returns
TranslatedVarlistEntry
instance, with populated coordinate axes. Units of scalar coord slices are translated to the units of the conventions’ coordinates. Includes logic to translate and rename scalar coords/slices, e.g.VarlistEntry
for ‘ua’ (intrinsically 4D) @ 500mb could produce aTranslatedVarlistEntry
for ‘u500’ (3D slice), depending on naming convention.
-
__init__
(name: str = sentinel.Mandatory, axes: src.util.basic.WormDict = <factory>, axes_lut: src.util.basic.WormDict = <factory>, entries: src.util.basic.WormDict = <factory>, lut: src.util.basic.WormDict = <factory>, env_vars: dict = <factory>) → None¶ Initialize self. See help(type(self)) for accurate signature.
-
__post_init__
(*args, **kwargs)¶
-
-
class
src.core.
NoTranslationFieldlist
(*args, **kwargs)[source]¶ Bases:
src.util.basic.Singleton
Class which partially implements the
Fieldlist
interface but does no variable translation.VarlistEntry
objects from the POD are passed through to createTranslatedVarlistEntry
objects.-
translate
(var)[source]¶ Returns
TranslatedVarlistEntry
instance, populated with contents of inputVarlistEntry
instance.Note
We return a copy of the
VarlistEntry
because logic inDefaultDatasetParser
alters the translation based on the file’s actual contents.
-
-
class
src.core.
VariableTranslator
(*args, **kwargs)[source]¶ Bases:
src.util.basic.Singleton
Singleton
containing information for different variable naming conventions. These are defined in thedata/fieldlist_*.jsonc
files.-
__init__
(code_root=None, unittest=False)[source]¶ Initialize self. See help(type(self)) for accurate signature.
-
read_conventions
(code_root, unittest=False)[source]¶ Read in the conventions from the Fieldlists and populate the convention attribute.
-
get_convention_name
(conv_name)[source]¶ Resolve the naming convention associated with a given
Fieldlist
object from among a set of possible aliases.
-
-
class
src.core.
MDTFFramework
(*args, **kwargs)[source]¶ Bases:
src.core.MDTFObjectBase
-
property
full_name
¶
-
configure
(cli_obj, pod_info_tuple, log_config)[source]¶ Wrapper for all configuration done based on CLI arguments.
-
parse_mdtf_args
(cli_obj, pod_info_tuple)[source]¶ Parse script options returned by the CLI. For greater customizability, most of the functionality is spun out into sub-methods.
-
__post_init__
()¶
-
property
active
¶
-
child_deactivation_handler
(child, exc)¶
-
child_status_update
(exc=None)¶
-
deactivate
(exc, level=None)¶
-
iter_children
(child_type=None, status=None, status_neq=None)¶ Generator iterating over child objects associated with this object.
- Parameters
status – None or
ObjectStatus
, default None. If None, iterates over all child objects, regardless of status. If aObjectStatus
value is passed, only iterates over child objects with that status.status_neq – None or
ObjectStatus
, default None. If set, iterates over child objects which don’t have the given status. If status is set, this setting is ignored.
-
status
: src.core.ObjectStatus = 1¶
-
property
failed
¶ Overall success/failure of this run of the framework. Return True if any case or any POD has failed, else return False.
-
property