src.util.filesystem module

Utility functions for interacting with the local filesystem and configuration files.

src.util.filesystem.abbreviate_path(path, old_base, new_base=None)

Express path as a path relative to old_base, optionally prepending new_base.

src.util.filesystem.resolve_path(path, root_path='', env=None, log=<Logger>)

Abbreviation to resolve relative paths, expanding environment variables if necessary.

Parameters:

• path (str) – Path to resolve.

• root_path (str) – Optional. Root path to resolve path with. If not given, resolves relative to os.getcwd().

Returns:

Absolute version of path.

Return type:

str

src.util.filesystem.recursive_copy(src_files, src_root, dest_root, copy_function=None, overwrite=False)

Copy src_files to dest_root, preserving relative subdirectory structure.

Copies a subset of files in a directory subtree rooted at src_root to an identical subtree structure rooted at dest_root, creating any subdirectories as needed. For example, recursive_copy('/A/B/C.txt', '/A', '/D') will first create the destination subdirectory /D/B and copy /A/B/C.txt to /D/B/C.txt.

Parameters:

• src_files (str or iterable) – Absolute path, or list of absolute paths, to files to copy.

• src_root (str) – Root subtree of all files in src_files.

• dest_root (str) – Destination directory in which to create the copied subtree.

• copy_function (function) – Function to use to copy individual files. Must take two arguments, the source and destination paths, respectively. Defaults to shutil.copy2().

• overwrite (bool) – Optional, default False. Determines whether to raise error if files would be overwritten.

Raises:

• ValueError – If all files in src_files are not contained in the src_root directory.

• OSError – If overwrite is False, raise if any destination files already exist, otherwise silently overwrite.

src.util.filesystem.check_executable(exec_name)

Tests if the executable exec_name is found on the current $PATH.

Parameters:

exec_name (str) – Name of the executable to search for.

src.util.filesystem.find_files(src_dirs, filename_globs, n_files=None)

Return list of files in src_dirs, or any subdirectories, matching any of filename_globs. Wraps Python glob.glob.

Parameters:

• src_dirs – Directory, or a list of directories, to search for files in. The function will also search all subdirectories.

• filename_globs – Glob, or a list of globs, for filenames to match. This is a shell globbing pattern, not a full regex.

• n_files (int) – Optional. Number of files expected to be found.

Raises:

MDTFFileNotFoundError – If n_files is supplied and the number of files found is different than this number.

Returns:

List of paths to files matching any of the criteria. If no files are found, the list is empty.

src.util.filesystem.check_dir(dir_, attr_name='', create=False)

Check existence of directories. No action is taken for directories that already exist; nonexistent directories either raise a MDTFFileNotFoundError or cause the creation of that directory.

Parameters:

• dir_ – If a string, the absolute path to check; otherwise, assume the path to check is given by the attr_name attribute on this object.

• attr_name – Name of the attribute being checked (used in log messages).

• create – (bool, default False): if True, nonexistent directories are created.

src.util.filesystem.bump_version(path, new_v=None, extra_dirs=None)

Append a version number to path, if necessary, so that it doesn’t conflict with existing files.

Parameters:

• path (str) – Path to test and append version number to.

• new_v (int) – Optional. Version number to begin incrementing at.

• extra_dirs (str or iterable) – Optional. If supplied, increment the version number of path so that it doesn’t conflict with pre-existing files at these locations either.

Returns:

path with a version number appended to it, if path exists. For files, the version number is appended before the extension. For example, repeated application would create a series of files file.txt, file.v1.txt, file.v2.txt, …

Return type:

str

src.util.filesystem.strip_comments(str_, delimiter=None)

Remove comments from str_. Comments are taken to start with an arbitrary delimiter and run to the end of the line.

src.util.filesystem.parse_json(str_)

Parse JSONC (JSON with //-comments) string str_ into a Python object. Comments are discarded. Wraps standard library json.loads().

Syntax errors in the input (JSONDecodeError) are passed through from the Python standard library parser. We correct the line numbers mentioned in the errors to refer to the original file (i.e., with comments.)

src.util.filesystem.read_json(file_path, log=<Logger>)

Reads a struct from a JSONC file at file_path.

Raises:

MDTFFileNotFoundError – If file not found at file_path.

Returns:

data contained in the file, as parsed by parse_json().

Return type:

dict

Execution exits with error code 1 on all other exceptions.

src.util.filesystem.find_json(dir_, file_name, exit_if_missing=True, log=<Logger>)

Reads a JSONC file file_name anywhere within the root directory dir_.

Parameters:

• dir_ (str) – Root directory to search (using find_files()).

• file_name (str) – Filename to search for.

• exit_if_missing (bool) – Optional, default True. Exit with error code 1 if file_name not found.

src.util.filesystem.write_json(struct, file_path, sort_keys=False, log=<Logger>)

Serializes struct to a JSON file at file_path.

Parameters:

• struct (dict) – Object to serialize.

• file_path (str) – path of the JSON file to write.

src.util.filesystem.pretty_print_json(struct, sort_keys=False)

Serialize struct to a pseudo-YAML string for human-readable debugging purposes only. Output is not valid JSON (or YAML).

src.util.filesystem.append_html_template(template_file, target_file, template_dict={}, create=True, append=True)

Perform substitutions on template_file and write result to target_file.

Variable substitutions are done with custom templating, replacing double curly bracket-delimited keys with their values in template_dict. For example, if template_dict is {'A': 'foo'}, all occurrences of the string {{A}} in template_file are replaced with the string foo. Spaces between the braces and variable names are ignored.

Double-curly-bracketed strings that don’t correspond to keys in template_dict are ignored (instead of raising a KeyError.)

Double curly brackets are chosen as the delimiter to match the default syntax of, e.g., jinja2. Using single curly braces would lead to conflicts with CSS syntax.

Parameters:

• template_file (str) – Path to template file.

• target_file (str) – Destination path for result.

• template_dict (dict) – Template name-value pairs. Both names and values must be strings.

• create (bool) – Optional, default True. If True, create target_file if it doesn’t exist, otherwise raise an OSError.

• append (bool) – Optional, default True. If target_file exists and this is True, append the substituted contents of template_file to it. If False, overwrite target_file with the substituted contents of template_file.