Skip to content

pudl_diff.defaults

What the tool knows about PUDL, without depending on it.

Everything here works when the PUDL package isn't installed, by falling back on constants and on PUDL's published outputs. When PUDL is importable, as it is when this runs inside a PUDL environment, its own definitions are used instead, so that they can't drift from these fallbacks. PUDL is only imported when it's needed, never when this module is, so PUDL can itself depend on this tool.

NIGHTLY_ROOT = f's3://{PUDL_CATALYST_COOP_HOST}/nightly/' module-attribute

The last nightly build's outputs, the reference point most diffs are measured against.

PUDL_CATALYST_COOP_DESCRIPTOR_NAME = 'pudl_parquet_datapackage.json' module-attribute

What the published outputs at PUDL_CATALYST_COOP_HOST call their datapackage descriptor, instead of datapackage.json.

PUDL_CATALYST_COOP_HOST = 'pudl.catalyst.coop' module-attribute

The bucket of PUDL's published outputs, on both S3 and GCS.

default_right_root()

The Parquet outputs of the local PUDL build: $PUDL_OUTPUT/parquet.

Raises:

Type Description
RuntimeError

If PUDL's output directory can't be determined.
Source code in src/pudl_diff/defaults.py 
50
51
52
53
54
55
56
57
58
59
60
61
62
63
def default_right_root() -> Path:
    """The Parquet outputs of the local PUDL build: `$PUDL_OUTPUT/parquet`.

    Raises:
        RuntimeError: If PUDL's output directory can't be determined.
    """
    if (paths := _import_pudl("pudl.workspace.setup")) is not None:
        return paths.PudlPaths().parquet_path()
    if not (pudl_output := os.environ.get("PUDL_OUTPUT")):
        raise RuntimeError(
            "There's no default for the right dataset unless PUDL_OUTPUT is set, "
            "so give its root explicitly."
        )
    return Path(pudl_output) / "parquet"

fallback_primary_key(table_name)

A table's primary key, when its own dataset can't say.

Tries PUDL's metadata if PUDL is installed, and then the datapackage of the last nightly build. The latter is only fetched if needed, and only once.

Returns:

Type Description
list[str] | None

The primary key, an empty list if the table has none, or None if it
list[str] | None

couldn't be found anywhere.
Source code in src/pudl_diff/defaults.py 
101
102
103
104
105
106
107
108
109
110
111
112
113
114
def fallback_primary_key(table_name: str) -> list[str] | None:
    """A table's primary key, when its own dataset can't say.

    Tries PUDL's metadata if PUDL is installed, and then the datapackage of the last
    nightly build. The latter is only fetched if needed, and only once.

    Returns:
        The primary key, an empty list if the table has none, or `None` if it
        couldn't be found anywhere.
    """
    for source in (_pudl_package_primary_key, _nightly_primary_key):
        if (primary_key := source(table_name)) is not None:
            return primary_key
    return None

nightly_root()

Where the last nightly build's outputs are, readable anonymously.

Source code in src/pudl_diff/defaults.py 
43
44
45
46
47
def nightly_root() -> UPath:
    """Where the last nightly build's outputs are, readable anonymously."""
    if (pudl := _import_pudl()) is not None:
        return pudl.PUDL_NIGHTLY_BUILDS_BASE_PATH
    return UPath(NIGHTLY_ROOT, anon=True)