src.util.logs module¶
Utilities related to configuration and handling of framework logging.
-
class
src.util.logs.
MDTFConsoleHandler
(stream=None)[source]¶ Bases:
logging.StreamHandler
Dummy class to designate logging to stdout or stderr from the root logger.
-
property
name
¶
-
terminator
= '\n'¶
-
property
-
class
src.util.logs.
StringIOHandler
[source]¶ Bases:
logging.StreamHandler
StreamHandler
instance that writes log entries to an internalStringIO
buffer.-
close
()[source]¶ Tidy up any resources used by the handler.
This version removes the handler from an internal map of handlers, _handlers, which is used for handler lookup by name. Subclasses should ensure that this gets called from overridden close() methods.
-
property
name
¶
-
terminator
= '\n'¶
-
-
class
src.util.logs.
MultiFlushMemoryHandler
(capacity, flushLevel=40, target=None, flushOnClose=True)[source]¶ Bases:
logging.handlers.MemoryHandler
Subclass
MemoryHandler
to enable flushing the contents of its log buffer to multiple targets. We do this to solve the chicken-and-egg problem of logging any events that happen before the log outputs are configured: those events are captured by an instance of this handler and then transfer()’ed to other handlers once they’re set up. See https://stackoverflow.com/a/12896092.-
transfer
(target_handler)[source]¶ Transfer contents of buffer to target_handler.
- Parameters
target_handler (
Handler
) – log handler to transfer contents of buffer to.
-
transfer_to_non_console
(logger)[source]¶ Transfer contents of buffer to all non-console-based handlers attached to logger (handlers that aren’t
MDTFConsoleHandler
.)If no handlers are attached to the logger, a warning is printed and the buffer is transferred to the
lastResort
handler, i.e. printed to stderr.- Parameters
logger (
Logger
) – logger to transfer contents of buffer to.
-
property
name
¶
-
-
class
src.util.logs.
HeaderFileHandler
(filename, mode='a', encoding=None, delay=False)[source]¶ Bases:
logging.FileHandler
Subclass
FileHandler
to print system and git repo information in a header at the start of a log file without writing it to other loggers (e.g. the console.)-
property
name
¶
-
terminator
= '\n'¶
-
property
-
class
src.util.logs.
MDTFHeaderFileHandler
(filename, mode='a', encoding=None, delay=False)[source]¶ Bases:
src.util.logs.HeaderFileHandler
FileHandler
which adds a header to log files with system information, git repo status etc. provided bymdtf_log_header()
.-
property
name
¶
-
terminator
= '\n'¶
-
property
-
class
src.util.logs.
HangingIndentFormatter
(fmt=None, datefmt=None, style='%', tabsize=0, header='', footer='')[source]¶ Bases:
logging.Formatter
Formatter
that applies a hanging indent, making it easier to tell where one entry stops and the next starts.-
__init__
(fmt=None, datefmt=None, style='%', tabsize=0, header='', footer='')[source]¶ Initialize formatter with extra arguments.
-
format
(record)[source]¶ Format the specified
LogRecord
as text, adding indentation and header/footer.- Parameters
record (
LogRecord
) – Logging record object to be formatted.- Returns
String representation of the log entry.
This essentially repeats the method’s implementation in the python standard library. See comments there and the logging module documentation.
-
converter
()¶ - localtime([seconds]) -> (tm_year,tm_mon,tm_mday,tm_hour,tm_min,
tm_sec,tm_wday,tm_yday,tm_isdst)
Convert seconds since the Epoch to a time tuple expressing local time. When ‘seconds’ is not passed in, convert the current time instead.
-
default_msec_format
= '%s,%03d'¶
-
default_time_format
= '%Y-%m-%d %H:%M:%S'¶
-
-
class
src.util.logs.
GeqLevelFilter
(name='', level=None)[source]¶ Bases:
src.util.logs._LevelFilterBase
Filter
to include only log messages with a severity of level or worse. This is normally done by setting the level attribute on aHandler
, but we need to add a filter when transferring records from another logger (theMultiFlushMemoryHandler
cache), as shown in https://stackoverflow.com/a/24324246.-
filter
(record)[source]¶ Determine if the specified record is to be logged.
Is the specified record to be logged? Returns 0 for no, nonzero for yes. If deemed appropriate, the record may be modified in-place.
-
__init__
(name='', level=None)¶ Initialize a filter.
Initialize with the name of the logger which, together with its children, will have its events allowed through the filter. If no name is specified, allow every event.
-
-
class
src.util.logs.
LtLevelFilter
(name='', level=None)[source]¶ Bases:
src.util.logs._LevelFilterBase
Filter
to include only log messages with a severity less than level.-
filter
(record)[source]¶ Determine if the specified record is to be logged.
Is the specified record to be logged? Returns 0 for no, nonzero for yes. If deemed appropriate, the record may be modified in-place.
-
__init__
(name='', level=None)¶ Initialize a filter.
Initialize with the name of the logger which, together with its children, will have its events allowed through the filter. If no name is specified, allow every event.
-
-
class
src.util.logs.
EqLevelFilter
(name='', level=None)[source]¶ Bases:
src.util.logs._LevelFilterBase
Filter
to include only log messages with a severity equal to level.-
filter
(record)[source]¶ Determine if the specified record is to be logged.
Is the specified record to be logged? Returns 0 for no, nonzero for yes. If deemed appropriate, the record may be modified in-place.
-
__init__
(name='', level=None)¶ Initialize a filter.
Initialize with the name of the logger which, together with its children, will have its events allowed through the filter. If no name is specified, allow every event.
-
-
class
src.util.logs.
NameMatchFilter
(name='')[source]¶ Bases:
logging.Filter
Filter
that only accepts log events directed to it specifically, rejecting all events coming from child loggers.Intended to be attached to a handler – the effect of attaching this to a logger is the same as setting propagate = False on it.
-
class
src.util.logs.
TagMatchFilter
(name='', tags=None)[source]¶ Bases:
logging.Filter
Filter
which only accepts records having the designated combination of custom ‘tag’ attributes. These are assigned by the methods inMDTFObjectLogger
or can be passed via the ‘extra’ kwarg on any logger (see discussion in entry for ~logging.Logger.)
-
class
src.util.logs.
ObjectLogTag
(value)¶ Bases:
src.util.basic.MDTFEnum
Standardized values that the MDTF-defined tags attribute on
LogRecord
objects can take, and thatTagMatchFilter
can listen for. These specify different destinations for the logging events.-
NC_HISTORY
= 1¶
-
BANNER
= 2¶
-
IN_FILE
= 3¶
-
OUT_FILE
= 4¶
-
-
class
src.util.logs.
MDTFObjectLogger
(name)[source]¶ Bases:
logging.Logger
This class wraps functionality for use by
MDTFObjectLoggerMixin
for log record-keeping by objects in the object hierarchy:A
Logger
to record events affecting the parent object only. This logger does not propagate events up the log hierarchy: the module-level logger should be used if that functionality is desired.A queue (_exceptions) for holding
Exception
objects received by the parent object.
-
log
(level, msg, *args, **kw)[source]¶ Log ‘msg % args’ with the integer severity ‘level’.
To pass exception information, use the keyword argument exc_info with a true value, e.g.
logger.log(level, “We have a %s”, “mysterious problem”, exc_info=1)
-
debug
(msg, *args, **kw)[source]¶ Log ‘msg % args’ with severity ‘DEBUG’.
To pass exception information, use the keyword argument exc_info with a true value, e.g.
logger.debug(“Houston, we have a %s”, “thorny problem”, exc_info=1)
-
info
(msg, *args, **kw)[source]¶ Log ‘msg % args’ with severity ‘INFO’.
To pass exception information, use the keyword argument exc_info with a true value, e.g.
logger.info(“Houston, we have a %s”, “interesting problem”, exc_info=1)
-
warning
(msg, *args, **kw)[source]¶ Log ‘msg % args’ with severity ‘WARNING’.
To pass exception information, use the keyword argument exc_info with a true value, e.g.
logger.warning(“Houston, we have a %s”, “bit of a problem”, exc_info=1)
-
error
(msg, *args, **kw)[source]¶ Log ‘msg % args’ with severity ‘ERROR’.
To pass exception information, use the keyword argument exc_info with a true value, e.g.
logger.error(“Houston, we have a %s”, “major problem”, exc_info=1)
-
critical
(msg, *args, **kw)[source]¶ Log ‘msg % args’ with severity ‘CRITICAL’.
To pass exception information, use the keyword argument exc_info with a true value, e.g.
logger.critical(“Houston, we have a %s”, “major disaster”, exc_info=1)
-
exception
(msg, *args, exc_info=True, **kw)[source]¶ Convenience method for logging an ERROR with exception information.
-
property
has_exceptions
¶ Return boolean corresponding to whether this object has received any exceptions (via
store_exception()
.)
-
classmethod
get_logger
(log_name)[source]¶ Workaround for setting the logger class, since logger objects have global state (calling getLogger with the same name returns the same object, like a Singleton.)
-
manager
= <logging.Manager object>¶
-
root
= <RootLogger root (WARNING)>¶
-
class
src.util.logs.
MDTFObjectLoggerMixinBase
[source]¶ Bases:
object
Dummy base class acting as a parent for all logging mixin classes for elements of the object hierarchy.
-
__init__
()¶ Initialize self. See help(type(self)) for accurate signature.
-
-
class
src.util.logs.
MDTFObjectLoggerMixin
[source]¶ Bases:
src.util.logs.MDTFObjectLoggerMixinBase
Base class to implement per-object logging for objects in the object hierarchy. Based on https://stackoverflow.com/q/57813096.
This wraps related functionalities:
A
Logger
to record events affecting the object only. Log messages are cached in aStringIO
buffer.A method
format()
for formatting the contents of the above into a string, along with log messages of any child objects. This is intended for preparing per-POD and per-case log files; logging intended for the console should use the module loggers.
-
init_log
(fmt=None)[source]¶ Logger initialization. This is a mixin class, so we don’t define a
__init__
method for simplicity.
-
property
last_exception
¶ Return most recent Exception received by the object.
-
format_log
(children=True)[source]¶ Return contents of log buffer, as well as that of any child objects in child_objs, as a formatted string.
-
__init__
()¶ Initialize self. See help(type(self)) for accurate signature.
-
class
src.util.logs.
VarlistEntryLoggerMixin
[source]¶ Bases:
src.util.logs.MDTFObjectLoggerMixin
Mixin providing per-object logging for
VarlistEntry
.-
__init__
()¶ Initialize self. See help(type(self)) for accurate signature.
-
format_log
(children=True)¶ Return contents of log buffer, as well as that of any child objects in child_objs, as a formatted string.
-
init_log
(fmt=None)¶ Logger initialization. This is a mixin class, so we don’t define a
__init__
method for simplicity.
-
property
last_exception
¶ Return most recent Exception received by the object.
-
-
class
src.util.logs.
PODLoggerMixin
[source]¶ Bases:
src.util.logs._CaseAndPODHandlerMixin
,src.util.logs.MDTFObjectLoggerMixin
Mixin providing per-object logging for
Diagnostic
(POD objects.)-
__init__
()¶ Initialize self. See help(type(self)) for accurate signature.
-
format_log
(children=True)¶ Return contents of log buffer, as well as that of any child objects in child_objs, as a formatted string.
-
init_extra_log_handlers
()¶ Hook used by child classes to add class-specific log handlers.
-
init_log
(fmt=None)¶ Logger initialization. This is a mixin class, so we don’t define a
__init__
method for simplicity.
-
property
last_exception
¶ Return most recent Exception received by the object.
-
-
class
src.util.logs.
CaseLoggerMixin
[source]¶ Bases:
src.util.logs._CaseAndPODHandlerMixin
,src.util.logs.MDTFObjectLoggerMixinBase
Mixin providing per-object logging for
DataSourceBase
(case objects, corresponding to experiments.)-
__init__
()¶ Initialize self. See help(type(self)) for accurate signature.
-
init_extra_log_handlers
()¶
-
-
src.util.logs.
git_info
()[source]¶ Get the current git branch, hash, and list of uncommitted files, if available. Based on NumPy’s implementation: https://stackoverflow.com/a/40170206.
-
src.util.logs.
mdtf_log_header
(title)[source]¶ Returns string of system debug information to use as log file header. Calls
git_info()
to get repo status.
-
src.util.logs.
signal_logger
(caller_name, signum=None, frame=None, log=<Logger>)[source]¶ Lookup signal name from number and write to log. Taken from https://stackoverflow.com/a/2549950.
- Parameters
caller_name (str) – Calling function name, only used in log message.
signum – Signal number of the signal we recieved.
frame – Parameters of the signal we recieved.
-
src.util.logs.
initial_log_config
()[source]¶ Configure the root logger for logging to console and to a cache provided by
MultiFlushMemoryHandler
.This is temporary logging configuration, used to solve the chicken-and-egg problem of logging any problems that happen before we’ve set up logging according to the user’s log config files, which requires doing a full parse of the user input. Once this is set up, any messages in the cache are sent to the user-configured logs.
-
src.util.logs.
transfer_log_cache
(target_log=None, close=False)[source]¶ Transfer the contents of the root log cache (
MultiFlushMemoryHandler
) to logs on newly-configured objects.