src.verify_links module¶

Checks html links in the output of the files returned by a run of the MDTF package and verifies that all linked files exist.

This is called by default at the end of each run, to determine if any PODs have failed without raising errors.

Based on test_website by Dani Coleman, bundy@ucar.edu.

class src.verify_links.Link(origin, target)¶

Bases: tuple

Class representing individual links, to simplify bookkeeping.

Attributes
  • origin (str) – URL of the document containing the link.

  • target (str) – URL referred to by the link.

__init__()¶

Initialize self. See help(type(self)) for accurate signature.

property origin¶
property target¶
class src.verify_links.LinkParser(*, convert_charrefs=True)[source]¶

Bases: html.parser.HTMLParser

Custom subclass of HTMLParser which constructs an iterable over each <a> tag. Adapted from https://stackoverflow.com/a/41663924.

reset()[source]¶

Reset this instance. Loses all unprocessed data.

handle_starttag(tag, attrs)[source]¶

Custom code for this subclass that extracts contents of <a> tags.

CDATA_CONTENT_ELEMENTS = ('script', 'style')¶
class src.verify_links.LinkVerifier(root, rel_path_root=None, verbose=False, log=None)[source]¶

Bases: object

__init__(root, rel_path_root=None, verbose=False, log=None)[source]¶

Initialize search for broken links.

Parameters
  • root (str) – Either a URL or path on the local filesystem. Location of the top-level html file to begin the search from.

  • rel_path_root (str, optional) – Either a URL or path on the local filesystem. If given, used as the path that relative paths to missing files are given relative to. Defaults to root (if root is a directory) or the directory containing root (if root is a file.)

  • verbose (bool, default False) – Set to True to print each file examined.

static gen_links(f, parser)[source]¶

Generator which parses the contents of an HTML file f and yields targets of all the links it contains. Adapted from https://stackoverflow.com/a/41663924.

Parameters
  • f – urllib.respose object of the form returned by urlopen(): either HTTPResponse for http or https, or addinfourl for files.

  • parser – instance of LinkParser.

Yields

Contents of the href attribute of each <a> tag of f, as extracted by parser.

check_one_url(link)[source]¶

Get list of URLs linked to from the current URL (in link.target).

Parameters

link (Link) – Link to check. Only the URL in link.target is examined.

Returns

Either 1) None if link.target can’t be opened, 2) the empty list if link.target is not an html document, or 3) a list of links contained in link.target, expressed as Link objects.

breadth_first(root_url)[source]¶

Breadth-first search of all files linked from an initial root_url.

The search correctly handles cycles (ie, A.html links to B.html and B.html links to A.html) and only examines files in subdirectories of root_url’s directory, so that links to external sites are ignored, rather than trying to trace the link structure of the whole internet.

Parameters

root_url (str) – URL of an html file to start the search at.

Returns

List of Link objects where the file referenced in link.target couldn’t be found.

group_relative_links(missing)[source]¶

Format paths to missing linked files as relative paths, grouped by POD.

Parameters

missing (list) – List of Link objects found by breadth_first(), whose targets correspond to missing files.

Returns

Dict, with keys given by the short names of PODs with missing files and values given by a list of the files that POD is missing. Missing files are listed by their path relative to the POD’s output directory.

verify_pod_links(pod_name)[source]¶

Perform search for missing linked files that were supposed to have been output by pod_name.

Parameters

pod_name – Name of the POD to check for missing files.

Returns

A list of the files that POD is missing. Missing files are listed by their path relative to the POD’s output directory.

verify_all_links()[source]¶

Perform search for any missing linked files from a run of the MDTF framework and collect them by POD.

Returns

Dict, with keys given by the short names of PODs with missing files and values given by a list of the files that POD is missing. Missing files are listed by their path relative to the POD’s output directory.

MDTF Diagnostics

Navigation

  • Getting started
    • 1. Overview
    • 2. Installation instructions
    • 3. Running the package on your data
  • Framework reference
    • Command-line options
    • Model data sources
    • Conventions for variable names and units
    • Model data format
    • Runtime configuration
  • Diagnostics reference
    • Summary of MDTF process-oriented diagnostics
    • Example Diagnostic Documentation
    • Convective Transition Diagnostic Package
    • EOF of Geopotential Height Diagnostic Module From NCAR
    • Mixed layer depth
    • MJO Propagation and Amplitude Diagnostic Package
    • MJO Suite Diagnostic Module From NCAR
    • MJO Teleconnection Diagnostic Package
    • Ocean Surface Latent Heat Flux Diagnostic Documentation
    • Phase and Amplitude of Precipitation Diurnal Cycle
    • Soil moisture-Evapotranspiration Coupling Diagnostic Package
    • Surface Temperature Extremes and Distribution Shape Package
    • Tropical Pacific Sea Level Diagnostic Documentation
    • Wavenumber Frequency Spectra Diagnostic Module From NCAR
  • Site-specific documentation
    • Customizing your installation with the ‘local’ site
    • GFDL-specific information
  • Developer information
    • 1. Introduction for POD developers
    • 2. Migration from framework v2.0
    • 3. POD development checklist
    • 4. Developer quickstart guide
    • 5. POD development guidelines
    • 6. POD settings file summary
    • 7. Walkthrough of framework operation
    • 8. POD coding best practices
    • 9. Git-based development workflow
  • Developer reference
    • Diagnostic settings file format
    • MDTF Environment variables
  • Internal code documentation
    • Internal API documentation
    • Module Index

  • Getting Started [PDF]
  • Developer's Walkthough [PDF]
  • Full documentation [PDF]

Related Topics

  • Documentation overview

Quick search

©2021, Model Diagnostics Task Force. | Powered by Sphinx 3.1.2 & Alabaster 0.7.12 | Page source