src.gfdl module

class src.gfdl.ModuleManager(*args, **kwargs)

Bases: src.util.Singleton

_current_module_versions = {'anaconda': 'anaconda2/5.1', 'gcp': 'gcp/2.3', 'ncl': 'ncarg/6.5.0', 'netcdf': 'netcdf/4.2', 'python2': 'python/2.7.12', 'python3': 'python/3.4.3', 'r': 'R/3.4.4'}

__init__()

Initialize self. See help(type(self)) for accurate signature.

_module(*args)

_parse_names(*module_names)

load(*module_names)

Wrapper for module load.

load_commands(*module_names)

unload(*module_names)

Wrapper for module unload.

unload_commands(*module_names)

_list()

Wrapper for module list.

revert_state()

class src.gfdl.GfdlDiagnostic(pod_name, verbose=0)

Bases: src.shared_diagnostic.Diagnostic

Wrapper for Diagnostic that adds writing a placeholder directory to the output as a lockfile if we’re running in frepp cooperative mode.

__init__(pod_name, verbose=0)

POD initializer. Given a POD name, we attempt to read a settings.json file in a subdirectory of /diagnostics by that name and parse the contents.

Parameters

• pod_name (str) – Name of the POD to initialize.

• verbose (int, optional) – Logging verbosity level. Default 0.

setUp(verbose=0)

Perform filesystem operations and checks prior to running the POD.

In order, this 1) sets environment variables specific to the POD, 2) creates POD-specific working directories, and 3) checks for the existence of the POD’s driver script.

Note

The existence of data files is checked with data_manager.DataManager.fetchData() and the runtime environment is validated separately as a function of environment_manager.EnvironmentManager.run(). This is because each POD is run in a subprocess (due to the necessity of supporting multiple languages) so the validation must take place in that subprocess.

Raises: PodRequirementFailure if requirements

aren’t met. This is re-raised from the _check_pod_driver() and _check_for_varlist_files() subroutines.

tearDown(verbose=0)

Performs cleanup tasks when the POD has finished running.

In order, this 1) creates the POD’s HTML output page from its included template, replacing CASENAME and other template variables with their current values, and adds a link to the POD’s page from the top-level HTML report; 2) converts the POD’s output plots (in PS or EPS vector format) to a bitmap format for webpage display; 3) Copies all requested files to the output directory and deletes temporary files.

Parameters

verbose (int, optional) – Logging verbosity level. Default 0.

class src.gfdl.GfdlvirtualenvEnvironmentManager(verbose=0)

Bases: src.environment_manager.VirtualenvEnvironmentManager

__init__(verbose=0)

Initialize self. See help(type(self)) for accurate signature.

set_pod_env(pod)

_module_lookup = {'ncl': ['ncl'], 'ncl_MJO_suite': ['python', 'ncl'], 'py_convective_transition_diag': ['python', 'ncl'], 'py_default': ['python'], 'r_default': ['r']}

create_environment(env_name)

activate_env_commands(env_name)

deactivate_env_commands(env_name)

tearDown()

_abc_impl = <_abc_data object>

class src.gfdl.GfdlcondaEnvironmentManager(verbose=0)

Bases: src.environment_manager.CondaEnvironmentManager

_call_conda_create(env_name)

_abc_impl = <_abc_data object>

src.gfdl.GfdlautoDataManager(case_dict, DateFreqMixin=None)

Wrapper for dispatching DataManager based on inputs.

class src.gfdl.GfdlarchiveDataManager(case_dict, DateFreqMixin=None)

Bases: src.data_manager.DataManager

__init__(case_dict, DateFreqMixin=None)

Initialize self. See help(type(self)) for accurate signature.

class DataKey(name_in_model, date_freq)

Bases: tuple

_asdict()

Return a new OrderedDict which maps field names to their values.

_field_defaults = {}

_fields = ('name_in_model', 'date_freq')

_fields_defaults = {}

classmethod _make(iterable)

Make a new DataKey object from a sequence or iterable

_replace(**kwds)

Return a new DataKey object replacing specified fields with new values

property date_freq

Alias for field number 1

property name_in_model

Alias for field number 0

dataset_key(dataset)

Return immutable representation of DataSet. Two DataSets should have the same key

abstract undecided_key(dataset)

abstract parse_relative_path(subdir, filename)

_listdir(dir_)

_list_filtered_subdirs(dirs_in, subdir_filter=None)

abstract subdirectory_filters()

_query_data()

XXX UPDATE DOCSTRING Populate _remote_data attribute with list of candidate files.

Specifically, if a <component> and <chunk_freq> subdirectory has all the requested data, return paths to all files we would need in that subdirectory. The decision of which <component> and <chunk_freq> to use is made in plan_data_fetching() because it requires comparing the files found for all requested datasets.

query_dataset(dataset)

abstract _decide_allowed_components()

plan_data_fetch_hook()

Filter files on model component and chunk frequency.

local_data_is_current(dataset)

Test whether data is current based on filesystem modification dates.

Todo

• Throw an error if local copy has been modified after remote copy.

• Handle case where local data involves processing of remote data, like

  ncrcat’ing. Copy raw remote files to temp directory if we need to process?

• gcp –sync does this already.

remote_data_list()

Process list of requested data to make data fetching efficient.

_fetch_exception_handler(exc)

fetch_dataset(d_key, method='auto')

Copy files to temporary directory and combine chunks.

_determine_fetch_method(method='auto')

process_fetched_data_hook()

_make_html(cleanup=False)

_make_tar_file(tar_dest_dir)

Make tar file of web/bitmap output.

_copy_to_output()

_abc_impl = <_abc_data object>

class src.gfdl.GfdlppDataManager(case_dict, DateFreqMixin=None)

Bases: src.gfdl.GfdlarchiveDataManager

__init__(case_dict, DateFreqMixin=None)

Initialize self. See help(type(self)) for accurate signature.

UndecidedKey

alias of ComponentKey

undecided_key(dataset)

parse_relative_path(subdir, filename)

subdirectory_filters()

static _heuristic_component_tiebreaker(str_list)

Determine experiment component(s) from heuristics.

1. If we’re passed multiple components, select those containing ‘cmip’.

2. If that selects multiple components, break the tie by selecting the

   component with the fewest words (separated by ‘_’), or, failing that, the shortest overall name.

:param str_list (list of str: ): list of component names.

Returns: str: name of component that breaks the tie.

_decide_allowed_components()

_abc_impl = <_abc_data object>

class src.gfdl.Gfdlcmip6abcDataManager(case_dict, DateFreqMixin=None)

Bases: src.gfdl.GfdlarchiveDataManager

__init__(case_dict, DateFreqMixin=None)

Initialize self. See help(type(self)) for accurate signature.

abstract _cmip6_root()

class UndecidedKey(table_id, grid_label, version_date)

Bases: tuple

_asdict()

Return a new OrderedDict which maps field names to their values.

_field_defaults = {}

_fields = ('table_id', 'grid_label', 'version_date')

_fields_defaults = {}

classmethod _make(iterable)

Make a new UndecidedKey object from a sequence or iterable

_replace(**kwds)

Return a new UndecidedKey object replacing specified fields with new values

property grid_label

Alias for field number 1

property table_id

Alias for field number 0

property version_date

Alias for field number 2

undecided_key(dataset)

parse_relative_path(subdir, filename)

subdirectory_filters()

static _cmip6_table_tiebreaker(str_list)

static _cmip6_grid_tiebreaker(str_list)

_decide_allowed_components()

_abc_impl = <_abc_data object>

class src.gfdl.Gfdludacmip6DataManager(case_dict, DateFreqMixin=None)

Bases: src.gfdl.Gfdlcmip6abcDataManager

_cmip6_root = '/archive/pcmdi/repo/CMIP6'

_abc_impl = <_abc_data object>

class src.gfdl.Gfdldatacmip6DataManager(case_dict, DateFreqMixin=None)

Bases: src.gfdl.Gfdlcmip6abcDataManager

_cmip6_root = '/data_cmip6/CMIP6'

_abc_impl = <_abc_data object>

src.gfdl.gcp_wrapper(source_path, dest_dir, timeout=0, dry_run=False)

src.gfdl.make_remote_dir(dest_dir, timeout=None, dry_run=None)

src.gfdl.running_on_PPAN()

Return true if current host is in the PPAN cluster.

src.gfdl.is_on_tape_filesystem(path)

src.gfdl.frepp_freq(date_freq)

src.gfdl.parse_frepp_stub(frepp_stub)

Converts the frepp arguments to a Python dictionary.

See https://wiki.gfdl.noaa.gov/index.php/FRE_User_Documentation#Automated_creation_of_diagnostic_figures.

Returns: dict of frepp parameters.