src.util.logs module

Utilities related to configuration and handling of framework logging.

class src.util.logs.MDTFConsoleHandler(stream=None)

Bases: StreamHandler

Dummy class to designate logging to stdout or stderr from the root logger.

__init__(stream=None)

Initialize the handler.

If stream is not specified, sys.stderr is used.

property name

terminator = '\n'

class src.util.logs.StringIOHandler

Bases: StreamHandler

StreamHandler instance that writes log entries to an internal StringIO buffer.

reset_buffer()

close()

Tidy up any resources used by the handler.

This version removes the handler from an internal map of handlers, _handlers, which is used for handler lookup by name. Subclasses should ensure that this gets called from overridden close() methods.

buffer_contents()

Return contents of buffer as a string.

property name

terminator = '\n'

class src.util.logs.MultiFlushMemoryHandler(capacity, flushLevel=40, target=None, flushOnClose=True)

Bases: MemoryHandler

Subclass MemoryHandler to enable flushing the contents of its log buffer to multiple targets. We do this to solve the chicken-and-egg problem of logging any events that happen before the log outputs are configured: those events are captured by an instance of this handler and then transfer()’ed to other handlers once they’re set up. See https://stackoverflow.com/a/12896092.

transfer(target_handler)

Transfer contents of buffer to target_handler.

Parameters:

target_handler (:py:class:`~logging.Handler`) – log handler to transfer contents of buffer to.

transfer_to_non_console(logger)

Transfer contents of buffer to all non-console-based handlers attached to logger (handlers that aren’t MDTFConsoleHandler.)

If no handlers are attached to the logger, a warning is printed and the buffer is transferred to the lastResort handler, i.e. printed to stderr.

Parameters:

logger (:py:class:`~logging.Logger`) – logger to transfer contents of buffer to.

__init__(capacity, flushLevel=40, target=None, flushOnClose=True)

Initialize the handler with the buffer size, the level at which flushing should occur and an optional target.

Note that without a target being set either here or via setTarget(), a MemoryHandler is no use to anyone!

The flushOnClose argument is True for backward compatibility reasons - the old behaviour is that when the handler is closed, the buffer is flushed, even if the flush level hasn’t been exceeded nor the capacity exceeded. To prevent this, set flushOnClose to False.

property name

class src.util.logs.HeaderFileHandler(filename, mode='a', encoding=None, delay=False, errors=None)

Bases: FileHandler

Subclass FileHandler to print system and git repo information in a header at the start of a log file without writing it to other loggers (e.g. the console.)

__init__(filename, mode='a', encoding=None, delay=False, errors=None)

Open the specified file and use it as the stream for logging.

property name

terminator = '\n'

class src.util.logs.MDTFHeaderFileHandler(filename, mode='a', encoding=None, delay=False, errors=None)

Bases: HeaderFileHandler

FileHandler which adds a header to log files with system information, git repo status etc. provided by mdtf_log_header().

__init__(filename, mode='a', encoding=None, delay=False, errors=None)

Open the specified file and use it as the stream for logging.

property name

terminator = '\n'

class src.util.logs.HangingIndentFormatter(fmt=None, datefmt=None, style='%', tabsize=0, header='', footer='')

Bases: Formatter

Formatter that applies a hanging indent, making it easier to tell where one entry stops and the next starts.

__init__(fmt=None, datefmt=None, style='%', tabsize=0, header='', footer='')

Initialize formatter with extra arguments.

Parameters:

• fmt (str) – format string, as in Formatter.

• datefmt (str) – date format string, as in Formatter or strftime.

• style (str) – string templating style, as in Formatter.

• tabsize (int) – Number of spaces to use for hanging indent.

• header (str) – Optional constant string to prepend to each log entry.

• footer (str) – Optional constant string to append to each log entry.

format(record)

Format the specified LogRecord as text, adding indentation and header/footer.

Parameters:

record (:py:class:`~logging.LogRecord`) – Logging record object to be formatted.

Returns:

String representation of the log entry.

This essentially repeats the method’s implementation in the python standard library. See comments there and the logging module documentation.

converter()

localtime([seconds]) -> (tm_year,tm_mon,tm_mday,tm_hour,tm_min,

tm_sec,tm_wday,tm_yday,tm_isdst)

Convert seconds since the Epoch to a time tuple expressing local time. When ‘seconds’ is not passed in, convert the current time instead.

default_msec_format = '%s,%03d'

default_time_format = '%Y-%m-%d %H:%M:%S'

class src.util.logs.GeqLevelFilter(name='', level=None)

Bases: _LevelFilterBase

Filter to include only log messages with a severity of level or worse. This is normally done by setting the level attribute on a Handler, but we need to add a filter when transferring records from another logger (the MultiFlushMemoryHandler cache), as shown in https://stackoverflow.com/a/24324246.

filter(record)

Determine if the specified record is to be logged.

Returns True if the record should be logged, or False otherwise. If deemed appropriate, the record may be modified in-place.

class src.util.logs.LtLevelFilter(name='', level=None)

Bases: _LevelFilterBase

Filter to include only log messages with a severity less than level.

filter(record)

Determine if the specified record is to be logged.

Returns True if the record should be logged, or False otherwise. If deemed appropriate, the record may be modified in-place.

class src.util.logs.EqLevelFilter(name='', level=None)

Bases: _LevelFilterBase

Filter to include only log messages with a severity equal to level.

filter(record)

Determine if the specified record is to be logged.

Returns True if the record should be logged, or False otherwise. If deemed appropriate, the record may be modified in-place.

class src.util.logs.NameMatchFilter(name='')

Bases: Filter

Filter that only accepts log events directed to it specifically, rejecting all events coming from child loggers.

Intended to be attached to a handler – the effect of attaching this to a logger is the same as setting propagate = False on it.

filter(record)

Determine if the specified record is to be logged.

Returns True if the record should be logged, or False otherwise. If deemed appropriate, the record may be modified in-place.

class src.util.logs.TagMatchFilter(name='', tags=None)

Bases: Filter

Filter which only accepts records having the designated combination of custom ‘tag’ attributes. These are assigned by the methods in MDTFObjectLogger or can be passed via the ‘extra’ kwarg on any logger (see discussion in entry for ~logging.Logger.)

filter(record)

Determine if the specified record is to be logged.

Returns True if the record should be logged, or False otherwise. If deemed appropriate, the record may be modified in-place.

class src.util.logs.ObjectLogTag(value, names=None, *, module=None, qualname=None, type=None, start=1, boundary=None)

Bases: MDTFEnum

Standardized values that the MDTF-defined tags attribute on LogRecord objects can take, and that TagMatchFilter can listen for. These specify different destinations for the logging events.

name = None

classmethod from_struct(str_)

Instantiate from string.

NC_HISTORY = 1

BANNER = 2

IN_FILE = 3

OUT_FILE = 4

class src.util.logs.MDTFObjectLogger(name)

Bases: Logger

This class wraps functionality for use by MDTFObjectLoggerMixin for log record-keeping by objects in the object hierarchy:

• A Logger to record events affecting the parent object only. This logger does not propagate events up the log hierarchy: the module-level logger should be used if that functionality is desired.

• A queue (_exceptions) for holding Exception objects received by the parent object.

log(level, msg, *args, **kw)

Log ‘msg % args’ with the integer severity ‘level’.

To pass exception information, use the keyword argument exc_info with a true value, e.g.

logger.log(level, “We have a %s”, “mysterious problem”, exc_info=True)

debug(msg, *args, **kw)

Log ‘msg % args’ with severity ‘DEBUG’.

To pass exception information, use the keyword argument exc_info with a true value, e.g.

logger.debug(“Houston, we have a %s”, “thorny problem”, exc_info=True)

info(msg, *args, **kw)

Log ‘msg % args’ with severity ‘INFO’.

To pass exception information, use the keyword argument exc_info with a true value, e.g.

logger.info(“Houston, we have a %s”, “interesting problem”, exc_info=True)

warning(msg, *args, **kw)

Log ‘msg % args’ with severity ‘WARNING’.

To pass exception information, use the keyword argument exc_info with a true value, e.g.

logger.warning(“Houston, we have a %s”, “bit of a problem”, exc_info=True)

error(msg, *args, **kw)

Log ‘msg % args’ with severity ‘ERROR’.

To pass exception information, use the keyword argument exc_info with a true value, e.g.

logger.error(“Houston, we have a %s”, “major problem”, exc_info=True)

critical(msg, *args, **kw)

Log ‘msg % args’ with severity ‘CRITICAL’.

To pass exception information, use the keyword argument exc_info with a true value, e.g.

logger.critical(“Houston, we have a %s”, “major disaster”, exc_info=True)

exception(msg, *args, exc_info=True, **kw)

Convenience method for logging an ERROR with exception information.

property has_exceptions

Return boolean corresponding to whether this object has received any exceptions (via store_exception().)

store_exception(exc)

Add an Exception object exc to the internal list.

classmethod get_logger(log_name) → <RootLogger root (WARNING)>

Workaround for setting the logger class, since logger objects have global state (calling getLogger with the same name returns the same object, like a Singleton.)

close_log_file(log=True)

manager = <logging.Manager object>

root = <RootLogger root (WARNING)>

class src.util.logs.MDTFObjectLoggerMixinBase

Bases: object

Dummy base class acting as a parent for all logging mixin classes for elements of the object hierarchy.

class src.util.logs.MDTFObjectLoggerMixin

Bases: MDTFObjectLoggerMixinBase

Base class to implement per-object logging for objects in the object hierarchy. Based on https://stackoverflow.com/q/57813096.

This wraps related functionalities:

• A Logger to record events affecting the object only. Log messages are cached in a StringIO buffer.

• A method format() for formatting the contents of the above into a string, along with log messages of any child objects. This is intended for preparing per-POD and per-case log files; logging intended for the console should use the module loggers.

init_log(fmt=None, log_dir=None)

Logger initialization. This is a mixin class, so we don’t define a __init__ method for simplicity.

init_extra_log_handlers()

Hook used by child classes to add class-specific log handlers.

property last_exception

Return most recent Exception received by the object.

format_log(children=True)

Return contents of log buffer, as well as that of any child objects in child_objs, as a formatted string.

class src.util.logs.VarlistEntryLoggerMixin

Bases: MDTFObjectLoggerMixin

Mixin providing per-object logging for VarlistEntry.

init_extra_log_handlers()

Hook used by child classes to add class-specific log handlers.

format_log(children=True)

Return contents of log buffer, as well as that of any child objects in child_objs, as a formatted string.

init_log(fmt=None, log_dir=None)

Logger initialization. This is a mixin class, so we don’t define a __init__ method for simplicity.

property last_exception

Return most recent Exception received by the object.

class src.util.logs.PODLoggerMixin

Bases: _CaseAndPODHandlerMixin, MDTFObjectLoggerMixin

Mixin providing per-object logging for Diagnostic (POD objects.)

format_log(children=True)

Return contents of log buffer, as well as that of any child objects in child_objs, as a formatted string.

init_extra_log_handlers()

Hook used by child classes to add class-specific log handlers.

init_log(fmt=None, log_dir=None)

Logger initialization. This is a mixin class, so we don’t define a __init__ method for simplicity.

property last_exception

Return most recent Exception received by the object.

class src.util.logs.CaseLoggerMixin

Bases: _CaseAndPODHandlerMixin, MDTFObjectLoggerMixin

Mixin providing per-object logging for DataSourceBase (case objects, corresponding to experiments.)

init_log(log_dir, fmt=None)

Logger initialization. This is a mixin class, so we don’t define a __init__ method for simplicity.

close_log_file(log=True)

format_log(children=True)

Return contents of log buffer, as well as that of any child objects in child_objs, as a formatted string.

init_extra_log_handlers()

Hook used by child classes to add class-specific log handlers.

property last_exception

Return most recent Exception received by the object.

src.util.logs.git_info()

Get the current git branch, hash, and list of uncommitted files, if available. Based on NumPy’s implementation: https://stackoverflow.com/a/40170206.

src.util.logs.mdtf_log_header(title)

Returns string of system debug information to use as log file header. Calls git_info() to get repo status.

src.util.logs.signal_logger(caller_name: str, signum=None, log=<Logger>)

Lookup signal name from number and write to log. Taken from https://stackoverflow.com/a/2549950.

Parameters:

• caller_name (str) – Calling function name, only used in log message.

• signum – Signal number of the signal we received.

• log – log file

src.util.logs.initial_log_config()

Configure the root logger for logging to console and to a cache provided by MultiFlushMemoryHandler.

This is temporary logging configuration, used to solve the chicken-and-egg problem of logging any problems that happen before we’ve set up logging according to the user’s log config files, which requires doing a full parse of the user input. Once this is set up, any messages in the cache are sent to the user-configured logs.

src.util.logs.transfer_log_cache(target_log=None, close=False)

Transfer the contents of the root log cache (MultiFlushMemoryHandler) to logs on newly-configured objects.

class src.util.logs.MDTFObjectBase(*args, **kwargs)

Bases: object

Base class providing shared functionality for the case object hierarchy, which is:

• :class:`~data_sources.DataSourceBase`s belonging to a run of the package;

• DataSourceBase;

• PodObject.

name: str = sentinel.Mandatory

status: ObjectStatus = 1

property full_name

property failed

property active

iter_children(child_type=None, status=None, status_neq=None)

Generator iterating over child objects associated with this object.

Parameters:

• child_type – None or Type type; default None. If None, iterates over

• all child objects regardless of their type

• status – None or ObjectStatus, default None. If None, iterates over all child objects, regardless of status. If a ObjectStatus value is passed, only iterates over child objects with that status.

• status_neq – None or ObjectStatus, default None. If set, iterates over child objects which don’t have the given status. If status is set, this setting is ignored.

child_deactivation_handler(child, exc)

child_status_update()

deactivate(exc, level=None)