src.cli module

Classes which parse the framework’s command line interface configuration files and implement the dynamic CLI; see Framework configuration and parsing.

Familiarity with the python argparse module is recommended.

src.cli.canonical_arg_name(str_)

Convert a flag or other specification to a destination variable name. The destination variable name always has underscores, never hyphens, in accordance with PEP8.

E.g., canonical_arg_name('--GNU-style-flag') returns “GNU_style_flag”.

src.cli.plugin_key(plugin_name)

Convert user input for plugin options to string used to lookup plugin value from options defined in cli_plugins.jsonc files.

Ignores spaces and underscores in supplied choices for CLI plugins, and make matching of plugin names case-insensititve.

src.cli.word_wrap(str_)

Clean whitespace and perform 80-column word wrapping for multi-line help and description strings. Explicit paragraph breaks must be encoded as a double newline (\n\n).

src.cli.read_config_files(code_root, file_name, site='')

Utility function to read a pair of configuration files: one for the framework defaults, another optional one for site-specific configuration.

Parameters:

• code_root (str) – Code repo directory.

• file_name (str) – Name of file to search for. We search for the file in all subdirectories of CLIConfigManager.site_dir() and CLIConfigManager.framework_dir(), respectively.

• site (str) – Name of the site-specific directory (in /sites) to search.

Returns:

A tuple of the two files’ contents. First element is the site specific file (empty dict if that file isn’t found) and second is the framework file (if not found, fatal error and exit immediately.)

src.cli.read_config_file(code_root, file_name, site='')

Return the site’s config file if present, else the framework’s file. Wraps read_config_files().

Parameters:

• code_root (str) – Code repo directory.

• file_name (str) – Name of file to search for. We search for the file in all subdirectories of CLIConfigManager.site_dir() and CLIConfigManager.framework_dir(), respectively.

• site (str) – Name of the site-specific directory (in /sites) to search.

Returns:

Path to the configuration file.

class src.cli.CustomHelpFormatter(*args, **kwargs)

Bases: RawDescriptionHelpFormatter, ArgumentDefaultsHelpFormatter

Modify help text formatter to only display variable placeholder text (“metavar”) once, to save space. Taken from https://stackoverflow.com/a/16969505. Also inherit from argparse.RawDescriptionHelpFormatter in order to preserve line breaks in description only (https://stackoverflow.com/a/18462760).

class src.cli.RecordDefaultsAction(option_strings, dest, nargs=None, const=None, default=None, type=None, **kwargs)

Bases: Action

Argparse Action that adds a boolean to record if user actually set the argument’s value, or if we’re using the default value specified in the parser. From https://stackoverflow.com/a/50936474. This also re-implements the ‘store_true’ and ‘store_false’ actions, in order to give defaults information on boolean flags.

If the user specifies a value for an option named <option>, the __call__() method adds a variable named <option>_is_default_ to the returned argparse.Namespace. This information is used by MDTFArgParser.parse_args() to populate the is_default attribute of MDTFArgParser.

Subclasses of argparse.Action are only called on user-supplied values, not default values. If the call_on_defaults flag is set on a subclass, MDTFArgParser.parse_args() will also call the action on default values.

default_value_suffix = '_is_default_'

call_on_defaults = False

class src.cli.PathAction(option_strings, dest, nargs=None, const=None, default=None, type=None, **kwargs)

Bases: RecordDefaultsAction

Argparse Action that performs shell environment variable expansion and resolution of relative paths, using resolve_path(). Should be specified as the CLI action for every option taking paths as a value.

call_on_defaults = True

default_value_suffix = '_is_default_'

class src.cli.ClassImportAction(option_strings, dest, nargs=None, const=None, default=None, type=None, **kwargs)

Bases: RecordDefaultsAction

Argparse Action to import classes on demand. Values are looked up from the ‘cli_plugins.jsonc’ file. This is a placeholder used to trigger behavior when arguments are parsed.

call_on_defaults = False

default_value_suffix = '_is_default_'

class src.cli.PluginArgAction(option_strings, dest, nargs=None, const=None, default=None, type=None, **kwargs)

Bases: ClassImportAction

Argparse Action to invoke the CLI plugin functionality, specificially importing the plugin’s entry point via ClassImportAction. All CLI options which define a plugin should specify this as the action.

call_on_defaults = False

default_value_suffix = '_is_default_'

class src.cli.CLIArgument(name: str, action: str = None, nargs: str = None, const: Any = None, default: Any = None, type: Any = None, choices: Iterable = None, required: bool = False, help: str = None, metavar: str = None, dest: str = None, is_positional: bool = False, short_name: str = None, hidden: bool = False)

Bases: object

Class which stores configuration options for a single argument of an argparse.ArgumentParser, with several custom options to simplify the parsing of CLIs defined in JSON files. Attributes correspond to arguments to add_argument().

name: str

action: str = None

nargs: str = None

const: Any = None

default: Any = None

type: Any = None

choices: Iterable = None

required: bool = False

help: str = None

metavar: str = None

dest: str = None

arg_flags: list

is_positional: bool = False

short_name: str = None

hidden: bool = False

add(target_p)

Adds the CLI argument to the parser target_p. Wraps add_argument().

Parameters:

target_p – Parser object (or argument group, or subparser) to which the argument will be added.

class src.cli.CLIArgumentGroup(title: str, description: str = None, arguments: list = <factory>)

Bases: object

Class holding configuration options for an argparse.ArgumentParser argument group. Attributes correspond to arguments to add_argument_group().

title: str

description: str = None

arguments: list

classmethod from_dict(d)

Initialize an instance of this object from a nested dict d obtained from reading a JSON file.

add(target_p)

Adds the CLI argument group, as well as all arguments it contains, to the parser target_p. Wraps add_argument_group().

Parameters:

target_p – Parser object (or subparser) to which the argument group will be added.

class src.cli.CLIParser(prog: str = None, usage: str = None, description: str = None, epilog: str = None, arguments: list = <factory>, argument_groups: list = <factory>)

Bases: object

Class holding configuration options for an instance of argparse.ArgumentParser (or equivalently a subcommand parser or a CLI plugin). Attributes correspond to arguments given to the argparse.ArgumentParser constructor.

prog: str = None

usage: str = None

description: str = None

epilog: str = None

arguments: list

argument_groups: list

classmethod from_dict(d)

Initialize an instance of this object from a nested dict d obtained from reading a JSON file.

iter_args(filter_class=None)

Iterator over all CLIArgument objects associated with this parser; if filter_class is specified, only iterate over objects having (a subclass of) filter_class as their action.

configure(target_p)

Configures a parser object by setting top-level attributes and adding all arguments and argument groups.

Parameters:

target_p – Parser object to configure.

add_plugin_args(preparsed_d)

Revise arguments after we know what plugins are being used. This annotates the help string of the plugin selector argument and configures its choices attribute (which lists the allowed values for each option in the online help). It then inserts the plugin-specifc CLI arguments following that argument.

Parameters:

preparsed_d – dict of results of the preparsing operation. Keys are the destination strings of the plugin selector arguments (identified by having their action set to PluginArgAction), and values are the values assigned to them by preparsing.

class src.cli.CLICommand(name: str, entry_point: str, help: str = '', cli_file: str = None, cli: dict = None, code_root: InitVar = '')

Bases: object

Class holding configuration options for a subcommand (invoked via a subparser) or a plugin.

name: str

entry_point: str

help: str = ''

cli_file: str = None

cli: dict = None

parser: Any = None

code_root: InitVar = ''

import_target(data_type='')

Imports the function or class referred to by the entry_point attribute.

Parameters:

• data_type – single_run (default), or multi_run. This value is provided at runtime,

• for (and the default value is set in src/cli_template.jsonc. data_type is only queried) –

• classes ("_manager") –

• cli_plugins.jsonc. (that are defined as lists in) –

call(*args, **kwargs)

Imports the function or class referred to by the entry_point attribute, and calls it with the passed arguments.

class src.cli.DefaultsFileTypes(value, names=None, *, module=None, qualname=None, type=None, start=1, boundary=None)

Bases: MDTFEnum

MDTFEnum to distinguish the three different categories of input settings files. In order of precedence:

1. USER: Input settings read from a file supplied by the user.

2. SITE: Settings specific to the given site (set with the --site flag.)

3. GLOBAL: Settings applicable to all sites. The main intended use case of this file is to enable the user to configure a default site at install time.

USER = 1

SITE = 2

GLOBAL = 3

class src.cli.CLIConfigManager(*args, **kwargs)

Bases: Singleton

Singleton to handle search, loading and parsing of configuration files for the CLI and CLI default values. We encapsulate this functionality in its own class, instead of MDTFArgParser or its children, to try to make the code easier to understand (not out of necessity).

Note

This is initialized in MDTFTopLevelArgParser; as a Singleton, it must be properly initialized before being referenced by the classes in this module.

default_site = 'local'

Name of the default value for the --site option.

defaults_filename = 'defaults.jsonc'

Name of the JSONC file for site-specific default settings.

subcommands_filename = 'cli_subcommands.jsonc'

Name of the JSONC files defining site-specific and built-in CLI subcommands.

plugins_filename = 'cli_plugins.jsonc'

Name of the JSONC files defining site-specific and built-in CLI plugins.

property framework_dir

Absolute path to the framework code directory, <CODE_ROOT>/src.

property sites_dir

Absolute path to the directory for site-specific code, <CODE_ROOT>/sites.

property site_dir

Absolute path to the directory for the site-specific code for the chosen site, <CODE_ROOT>/sites/<site>.

read_defaults(def_type, path=None)

Populate one of the entries in self.defaults by reading from the appropriate defaults file.

Parameters:

• def_type (DefaultsFileTypes) – Type of defaults file to read.

• path (str, optional) – path of the file. Only used for user-specified defaults.

site_default_text()

Return text to be used in online help describing the paths to the defaults files that are being used.

read_subcommands()

Populates subcommands and subparser_kwargs attributes with contents of CLI plugin files for the framework and site. Site-specific subcommand definitions override those defined on the framework.

read_plugins()

Populates plugins attribute with contents of CLI plugin files for the framework and site.

get_plugin(plugin_name, choice_of_plugin=None)

Look up requested CLI plugin from plugins attribute, logging appropriate errors where KeyErrors would be raised.

Parameters:

• plugin_name (str) – Name of the plugin selected.

• choice_of_plugin (str, optional) – if provided, the name of the choice of plugin.

Returns:

CLICommand object corresponding to the requested plugin choice if both arguments are given, or a dict of recognized choices if only the first argument is given.

class src.cli.MDTFArgParser(*args, **kwargs)

Bases: ArgumentParser

Customized argparse.ArgumentParser. Added functionality:

• Configuring the parser from an external file (configure()).

• Customized help text formatting provided by CustomHelpFormatter.

• Recording whether the user specified each argument value, or whether the default was used, via RecordDefaultsAction.

• Better bookkeeping of argument groups, e.g. which arguments belong to which group.

static split_args(argv)

Wrapper for shlex.split().

iter_actions()

Iterator over Action objects associated with all user-defined arguments in parser, as well as those for any subcommands.

parse_known_args(args=None, namespace=None)

Wrapper for parse_known_args() which handles intermediate levels of default settings derived from the user’s settings files. These override defaults defined in the parser itself. The precedence order is:

1. Argument values explictly given by the user on the command line, as recorded in the is_default attribute of MDTFArgParser.

2. Argument values from a file the user gave via the -f flag. (CLIConfigManager.defaults[DefaultsFileTypes.USER]).

3. Argument values specified as the default values in the argument parser, which in turn are set with the following precedence order:

   1. Default values from a site-specfic file (defaults.jsonc), stored in CLIConfigManager.defaults[DefaultsFileTypes.SITE].

   2. Default values from a defaults.jsonc file in the /sites directory, stored in CLIConfigManager.defaults[DefaultsFileTypes.GLOBAL].

   3. Default values hard-coded in the CLI definition file itself.

Parameters:

• args (optional) – String or list of strings to parse. If a single string is passed, it’s split using split_args(). If not supplied, the default behavior parses sys.argv().

• namespace (optional) – An object to store the parsed arguments. The default is a new empty argparse.Namespace object.

Returns:

Tuple of 1) populated namespace containing parsed arguments and 2) unrecognized arguments, as with argparse.ArgumentParser.parse_known_args().

parse_args(args=None, namespace=None)

Subclassed implementation of parse_args() which wraps parse_known_args().

class src.cli.MDTFArgPreparser

Bases: MDTFArgParser

Parser class used to “preparse” plugin selector arguments, to determine what site and plugin values were set by the user. Plugin selector arguments are identified by having their action set to PluginArgAction.

parse_site(argv=None, default_site=None)

Wrapper for parse_known_args() used to determine what site to use.

parse_input_file(argv=None)

Wrapper for parse_known_args() used to determine what user input file to use.

parse_plugins(argv=None)

Wrapper for parse_known_args() used to parse the plugin selector arguments.

iter_actions()

Iterator over Action objects associated with all user-defined arguments in parser, as well as those for any subcommands.

parse_args(args=None, namespace=None)

Subclassed implementation of parse_args() which wraps parse_known_args().

parse_known_args(args=None, namespace=None)

Wrapper for parse_known_args() which handles intermediate levels of default settings derived from the user’s settings files. These override defaults defined in the parser itself. The precedence order is:

1. Argument values explictly given by the user on the command line, as recorded in the is_default attribute of MDTFArgParser.

2. Argument values from a file the user gave via the -f flag. (CLIConfigManager.defaults[DefaultsFileTypes.USER]).

3. Argument values specified as the default values in the argument parser, which in turn are set with the following precedence order:

   1. Default values from a site-specfic file (defaults.jsonc), stored in CLIConfigManager.defaults[DefaultsFileTypes.SITE].

   2. Default values from a defaults.jsonc file in the /sites directory, stored in CLIConfigManager.defaults[DefaultsFileTypes.GLOBAL].

   3. Default values hard-coded in the CLI definition file itself.

Parameters:

• args (optional) – String or list of strings to parse. If a single string is passed, it’s split using split_args(). If not supplied, the default behavior parses sys.argv().

• namespace (optional) – An object to store the parsed arguments. The default is a new empty argparse.Namespace object.

Returns:

Tuple of 1) populated namespace containing parsed arguments and 2) unrecognized arguments, as with argparse.ArgumentParser.parse_known_args().

static split_args(argv)

Wrapper for shlex.split().

class src.cli.MDTFTopLevelArgParser(code_root, skip_defaults=False, argv=None)

Bases: MDTFArgParser

Class for constructing the command-line interface, parsing the options, and handing off execution to the selected subcommand.

iter_arg_groups(subcommand=None)

Iterate over all arguments defined on the parser for the subcommand subcommand.

iter_group_actions(subcommand=None, group=None)

Iterate over all arguments defined on the parser group group for the subcommand subcommand.

add_input_file_arg(target_p)

Convenience method to add the flag to pass a user-designated defaults file to the parser target_p (either the top-level parser, or the preparser.)

init_user_defaults()

Set user defaults using values read in from a configuration file in one of two formats. The format is determined from context: either

1. A JSON/JSONC file of key-value pairs. This is parsed using parse_json().

2. A plain text file containing flags and arguments as they would be passed on the command line (except shell expansions are not performed). This is parsed by the MDTFArgParser.parse_args() method of the configured parser.

The file’s path is determined from the argument to the -f flag via MDTFArgPreparser.parse_input_file().

Raises:

ValueError – if the string cannot be parsed.

add_site_arg(target_p)

Convenience method to add the argument flag to select which site-specific code to use, to the parser target_p (either the top-level parser, or a preparser.)

init_site()

We allow site-specific installations to customize the CLI, so before we construct the CLI parser we need to determine what site to use. We do this by running a parser that only looks for the --site flag.

This sets the site attribute and populates the global and site-specific settings dicts.

add_contents(target_p)

Convenience method to fully configure a parser target_p (either the top-level parser, or a preparser), adding subparsers for all subcommands.

setup()

Method to wrap all configuration methods needed to configure the parser before calling parse_args(): reading the defaults files and configuring plugins based on existing values.

configure()

Method that assembles the top-level CLI parser; called at the end of setup(). Options specific to the script are hard-coded here; CLI options for each subcommand are given in jsonc configuration files for each command which are read in by setup(). See documentation for parse_known_args() for information on the configuration file mechanism.

parse_args(args=None, namespace=None)

Wrapper for parse_args() which handles intermediate levels of default settings. See documentation for parse_known_args().

parse_known_args(args=None, namespace=None)

Wrapper for parse_known_args() which handles intermediate levels of default settings; see documentation for parse_known_args().

dispatch(args=None)

Parse args, and call the subcommand that was selected.

iter_actions()

Iterator over Action objects associated with all user-defined arguments in parser, as well as those for any subcommands.

static split_args(argv)

Wrapper for shlex.split().

class src.cli.MDTFTopLevelSubcommandArgParser(code_root, skip_defaults=False, argv=None)

Bases: MDTFTopLevelArgParser

Extends MDTFTopLevelArgParser to add support for subcommands. Currently unused, intended for a future release.

add_input_file_arg(target_p)

Convenience method to add the flag to pass a user-designated defaults file to the parser target_p (either the top-level parser, or the preparser.)

add_site_arg(target_p)

Convenience method to add the argument flag to select which site-specific code to use, to the parser target_p (either the top-level parser, or a preparser.)

init_site()

We allow site-specific installations to customize the CLI, so before we construct the CLI parser we need to determine what site to use. We do this by running a parser that only looks for the --site flag.

This sets the site attribute and populates the global and site-specific settings dicts.

init_user_defaults()

Set user defaults using values read in from a configuration file in one of two formats. The format is determined from context: either

1. A JSON/JSONC file of key-value pairs. This is parsed using parse_json().

2. A plain text file containing flags and arguments as they would be passed on the command line (except shell expansions are not performed). This is parsed by the MDTFArgParser.parse_args() method of the configured parser.

The file’s path is determined from the argument to the -f flag via MDTFArgPreparser.parse_input_file().

Raises:

ValueError – if the string cannot be parsed.

iter_actions()

Iterator over Action objects associated with all user-defined arguments in parser, as well as those for any subcommands.

iter_arg_groups(subcommand=None)

Iterate over all arguments defined on the parser for the subcommand subcommand.

iter_group_actions(subcommand=None, group=None)

Iterate over all arguments defined on the parser group group for the subcommand subcommand.

parse_args(args=None, namespace=None)

Wrapper for parse_args() which handles intermediate levels of default settings. See documentation for parse_known_args().

parse_known_args(args=None, namespace=None)

Wrapper for parse_known_args() which handles intermediate levels of default settings; see documentation for parse_known_args().

setup()

Method to wrap all configuration methods needed to configure the parser before calling parse_args(): reading the defaults files and configuring plugins based on existing values.

static split_args(argv)

Wrapper for shlex.split().

add_contents(target_p)

Convenience method to fully configure a parser target_p (either the top-level parser, or a preparser), adding subparsers for all subcommands.

configure()

Method that assembles the top-level CLI parser. Options specific to the script are hard-coded here; CLI options for each subcommand are given in jsonc configuration files for each command which are read in here. See documentation for parse_known_args() for information on the configuration file mechanism.

dispatch()

Parse args, and call the subcommand that was selected.