src.environment_manager module

Classes which setup software dependencies for the PODs and which execute the PODs’ code.

class src.environment_manager.AbstractEnvironmentManager(log=<Logger>)

Bases: ABC

Abstract interface for EnvironmentManager classes. The EnvironmentManager is responsible for setting up the runtime environment for a POD’s third-party software dependencies (language versions, libraries, etc.) before execution and doing any associated cleanup after execution.

setup()

Performs any initialization tasks common to the EnvironmentManager as a whole (called once.)

abstract create_environment(env_name)

Install or otherwise create the POD runtime environment identified by env_name, which can be an arbitrary object describing the environment.

abstract get_pod_env(pod)

Assign an environment identifier (of the same type as the env_name arguments) to a pod based on the requirements and other declarative information from its settings.jsonc file.

Parameters:

var (Diagnostic) – POD object whose runtime environment needs to be determined.

Returns:

Environment identifier to be passed as env_name to other methods of the EnvironmentManager.

abstract activate_env_commands(env_name)

Generate shell commands needed to activate/set up the environment before the POD executes.

Parameters:

env_name – Identifier corresponding to the environment to activate.

Returns:

List of strings, one per command, corresponding to the shell commands needed to activate or set up the runtime environment within the POD’s execution environment (e.g., child subprocess) created by the RuntimeManager.

abstract deactivate_env_commands(env_name)

Generate shell commands needed to deactivate/clean up the environment after the POD has finished executing.

Parameters:

env_name – Identifier corresponding to the environment to deactivate.

Returns:

List of strings, one per command, corresponding to the shell commands needed to deactivate or tear down the runtime environment within the POD’s execution environment (e.g., child subprocess) created by the RuntimeManager.

abstract destroy_environment(env_name)

Uninstall or otherwise remove the POD runtime environment identified by env_name, which can be an arbitrary object describing the environment.

tear_down()

Performs any cleanup specific to the EnvironmentManager itself. Called once, after all PODs have executed.

class src.environment_manager.NullEnvironmentManager(log=<Logger>)

Bases: AbstractEnvironmentManager

EnvironmentManager class which does nothing; intended as a dummy setting for building framework test harnesses.

create_environment(env_name)

No-op.

destroy_environment(env_name)

No-op.

get_pod_env(pod)

No-op.

activate_env_commands(env_name)

No-op.

deactivate_env_commands(env_name)

No-op.

setup()

Performs any initialization tasks common to the EnvironmentManager as a whole (called once.)

tear_down()

Performs any cleanup specific to the EnvironmentManager itself. Called once, after all PODs have executed.

class src.environment_manager.VirtualenvEnvironmentManager(log=<Logger>)

Bases: AbstractEnvironmentManager

EnvironmentManager class which installs dependencies in python virtualenvs. Specifically,

• Assumes the needed versions of the scripting language executables are correctly set by default in the $PATH of the POD’s runtime environment;

• For python-based PODs, it uses pip and virtualenvs to install libraries requested by each POD. The installation location is set by the CLI flag --venv-root.

• For R-based PODs, it attempts to install needed libraries into the current user’s $PATH. The installation location is set by the CLI flag --r-lib-root.

• For other scripting languages (e.g. NCL), no library management is performed. All dependencies are assumed to have been pre-installed.

create_environment(env_key)

Install or otherwise create the POD runtime environment identified by env_name, which can be an arbitrary object describing the environment.

destroy_environment(env_key)

Uninstall or otherwise remove the POD runtime environment identified by env_name, which can be an arbitrary object describing the environment.

get_pod_env(pod)

Assign an environment identifier (of the same type as the env_name arguments) to a pod based on the requirements and other declarative information from its settings.jsonc file.

Parameters:

var (Diagnostic) – POD object whose runtime environment needs to be determined.

Returns:

Environment identifier to be passed as env_name to other methods of the EnvironmentManager.

activate_env_commands(env_key)

Generate shell commands needed to activate/set up the environment before the POD executes.

Parameters:

env_name – Identifier corresponding to the environment to activate.

Returns:

List of strings, one per command, corresponding to the shell commands needed to activate or set up the runtime environment within the POD’s execution environment (e.g., child subprocess) created by the RuntimeManager.

deactivate_env_commands(env_key)

Generate shell commands needed to deactivate/clean up the environment after the POD has finished executing.

Parameters:

env_name – Identifier corresponding to the environment to deactivate.

Returns:

List of strings, one per command, corresponding to the shell commands needed to deactivate or tear down the runtime environment within the POD’s execution environment (e.g., child subprocess) created by the RuntimeManager.

setup()

Performs any initialization tasks common to the EnvironmentManager as a whole (called once.)

tear_down()

Performs any cleanup specific to the EnvironmentManager itself. Called once, after all PODs have executed.

class src.environment_manager.CondaEnvironmentManager(log=<Logger>)

Bases: AbstractEnvironmentManager

AbstractEnvironmentManager that uses the conda package manager to define and switch runtime environments.

env_name_prefix = '_MDTF_'

create_environment(env_name)

Install or otherwise create the POD runtime environment identified by env_name, which can be an arbitrary object describing the environment.

create_all_environments()

destroy_environment(env_name)

Uninstall or otherwise remove the POD runtime environment identified by env_name, which can be an arbitrary object describing the environment.

get_pod_env(pod)

Assign an environment identifier (of the same type as the env_name arguments) to a pod based on the requirements and other declarative information from its settings.jsonc file.

Parameters:

var (Diagnostic) – POD object whose runtime environment needs to be determined.

Returns:

Environment identifier to be passed as env_name to other methods of the EnvironmentManager.

activate_env_commands(env_name)

Source conda_init.sh to set things that aren’t set b/c we aren’t in an interactive shell.

deactivate_env_commands(env_name)

Generate shell commands needed to deactivate/clean up the environment after the POD has finished executing.

Parameters:

env_name – Identifier corresponding to the environment to deactivate.

Returns:

List of strings, one per command, corresponding to the shell commands needed to deactivate or tear down the runtime environment within the POD’s execution environment (e.g., child subprocess) created by the RuntimeManager.

setup()

Performs any initialization tasks common to the EnvironmentManager as a whole (called once.)

tear_down()

Performs any cleanup specific to the EnvironmentManager itself. Called once, after all PODs have executed.

class src.environment_manager.AbstractRuntimeManager

Bases: ABC

Interface for RuntimeManager classes. The RuntimeManager is responsible for managing the actual execution of the PODs.

abstract setup()

Performs any initialization tasks

abstract run()

abstract tear_down()

class src.environment_manager.SubprocessRuntimePODWrapper(pod: ~typing.Any = sentinel.Mandatory, env: ~typing.Any = None, env_vars: dict = <factory>)

Bases: object

Wrapper for diagnostic.Diagnostic that adds fields and methods used by SubprocessRuntimeManager.

pod: Any = sentinel.Mandatory

env: Any = None

env_vars: dict

process: Any = None

pre_run_setup()

setup_env_vars()

setup_exception_handler(exc)

run_commands()

Produces the shell command(s) to run the POD.

run_msg()

Log message when execution starts.

validate_commands()

Produces the shell command(s) to validate the POD’s runtime environment (ie, check for all requested third-party module dependencies.) Dependencies are passed as arguments to the shell script src/validate_environment.sh, which is invoked in the POD’s subprocess before the POD is run.

Returns:

Command-line invocation to validate the POD’s

runtime environment.

Return type:

(str)

runtime_exception_handler(exc)

Handler which is called if an exception is raised during the POD’s execution (including setup and clean up).

tear_down(retcode=None)

class src.environment_manager.SubprocessRuntimeManager(case, EnvMgrClass)

Bases: AbstractRuntimeManager

RuntimeManager class that runs each POD in a child subprocess spawned on the local machine. Resource allocation is delegated to the local machine’s kernel’s scheduler.

iter_active_pods()

Generator iterating over all wrapped pods which are currently active, i.e. which haven’t been skipped due to requirement errors.

setup()

Performs any initialization tasks

spawn_subprocess(p, env_vars_base)

run()

tear_down()

runtime_terminate(signum=None, frame=None)

Handler called in the event that POD execution was halted abnormally, by receiving SIGINT or SIGTERM.

class src.environment_manager.MultirunSubprocessRuntimePODWrapper(pod: ~typing.Any = sentinel.Mandatory, env: ~typing.Any = None, env_vars: dict = <factory>)

Bases: object

Wrapper for diagnostic.multirunDiagnostic that adds fields and methods used by SubprocessRuntimeManager.

pod: Any = sentinel.Mandatory

env: Any = None

env_vars: dict

process: Any = None

pre_run_setup()

setup_env_vars()

write_case_env_file(case_list)

setup_exception_handler(exc)

run_commands()

Produces the shell command(s) to run the POD.

run_msg()

Log message when execution starts.

validate_commands()

Produces the shell command(s) to validate the POD’s runtime environment (ie, check for all requested third-party module dependencies.) Dependencies are passed as arguments to the shell script src/validate_environment.sh, which is invoked in the POD’s subprocess before the POD is run.

Returns:

Command-line invocation to validate the POD’s

runtime environment.

Return type:

(str)

runtime_exception_handler(exc)

Handler which is called if an exception is raised during the POD’s execution (including setup and clean up).

tear_down(retcode=None)

class src.environment_manager.MultirunSubprocessRuntimeManager(pod_dict, EnvMgrClass, parent)

Bases: SubprocessRuntimeManager

RuntimeManager class that runs each POD in a child subprocess spawned on the local machine. Resource allocation is delegated to the local machine’s kernel’s scheduler.

iter_active_pods()

Generator iterating over all wrapped pods which are currently active, i.e. which haven’t been skipped due to requirement errors.

setup()

Performs any initialization tasks

spawn_subprocess(p, env_vars_base)

run(parent)

tear_down()

runtime_terminate(signum=None, frame=None)

Handler called in the event that POD execution was halted abnormally, by receiving SIGINT or SIGTERM.