src.data_manager module

exception src.data_manager.DataQueryFailure(dataset, msg='')

Bases: Exception

Exception signaling a failure to find requested data in the remote location.

Raised by queryData() to signal failure of a data query. Should be caught properly in planData() or fetchData().

__init__(dataset, msg='')

Initialize self. See help(type(self)) for accurate signature.

exception src.data_manager.DataAccessError(dataset, msg='')

Bases: Exception

Exception signaling a failure to obtain data from the remote location.

__init__(dataset, msg='')

Initialize self. See help(type(self)) for accurate signature.

class src.data_manager.DataSet(*args, **kwargs)

Bases: src.util.NameSpace

Class to describe datasets.

https://stackoverflow.com/a/48806603 for implementation.

__init__(*args, **kwargs)

Initialize self. See help(type(self)) for accurate signature.

copy() → a shallow copy of D

classmethod from_pod_varlist(pod_convention, var, dm_args)

_freeze()

Return immutable representation of (current) attributes.

Exclude attributes starting with ‘_’ from the comparison, in case we want DataSets with different timestamps, temporary directories, etc. to compare as equal.

class src.data_manager.DataManager(case_dict, DateFreqMixin=None)

Bases: object

__init__(case_dict, DateFreqMixin=None)

Initialize self. See help(type(self)) for accurate signature.

iter_pods()

Generator iterating over all pods which haven’t been skipped due to requirement errors.

iter_vars()

Generator iterating over all variables in all pods which haven’t been skipped due to requirement errors.

setUp(verbose=0)

_setup_pod(pod)

static dataset_key(dataset)

Return immutable representation of DataSet. Two DataSets should have the same key

local_path(data_key)

Returns the absolute path of the local copy of the file for dataset.

This determines the local model data directory structure, which is $MODEL_DATA_ROOT/<CASENAME>/<freq>/<CASENAME>.<var name>.<freq>.nc’. Files not following this convention won’t be found.

_build_data_dicts()

fetch_data()

_fetch_exception_handler(exc)

_query_data()

_iter_populated_varlist(var_iter, pod_name)

Generator function yielding either a variable, its alternates if the variable was not found in the data query, or DataQueryFailure if the variable request can’t be satisfied with found data.

remote_data_list()

Process list of requested data to make data fetching efficient.

This is intended as a hook to be used by subclasses. Default behavior is to delete from the list duplicate datasets and datasets where a local copy of the data already exists and is current (as determined by local_data_is_current()).

Returns: collection of DataSet

objects.

_fetch_order_function = None

local_data_is_current(dataset)

Determine if local copy of data needs to be refreshed.

This is intended as a hook to be used by subclasses. Default is to always return False, ie always fetch remote data.

Returns: True if local copy of data exists and remote copy hasn’t been

updated.

plan_data_fetch_hook()

preprocess_local_data(*args, **kwargs)

abstract query_dataset(dataset)

abstract fetch_dataset(dataset)

tearDown()

_make_html(cleanup=True)

_backup_config_file(config)

Record settings in file variab_dir/config_save.json for rerunning

_make_tar_file(tar_dest_dir)

Make tar file of web/bitmap output.

_copy_to_output()

_abc_impl = <_abc_data object>

class src.data_manager.LocalfileDataManager(case_dict, DateFreqMixin=None)

Bases: src.data_manager.DataManager

_abc_impl = <_abc_data object>

class DataKey(name_in_model, date_freq)

Bases: tuple

_asdict()

Return a new OrderedDict which maps field names to their values.

_field_defaults = {}

_fields = ('name_in_model', 'date_freq')

_fields_defaults = {}

classmethod _make(iterable)

Make a new DataKey object from a sequence or iterable

_replace(**kwds)

Return a new DataKey object replacing specified fields with new values

property date_freq

Alias for field number 1

property name_in_model

Alias for field number 0

dataset_key(dataset)

Return immutable representation of DataSet. Two DataSets should have the same key

query_dataset(dataset)

local_data_is_current(dataset)

Determine if local copy of data needs to be refreshed.

This is intended as a hook to be used by subclasses. Default is to always return False, ie always fetch remote data.

Returns: True if local copy of data exists and remote copy hasn’t been

updated.

fetch_dataset(dataset)