pudl.dagster.assets.raw.ferc_to_sqlite#

Dagster asset definitions for granular FERC-to-SQLite extraction.

This module defines the prerequisite assets that build the FERC DBF and XBRL derived SQLite databases used elsewhere in the PUDL pipeline. It should contain asset factories, resource requirements, and materialization metadata specific to those prerequisite databases, rather than the downstream transforms that consume them.

Attributes#

`NETWORK_ERRORS`
`logger`
`raw_ferc1_dbf__sqlite`
`raw_ferc2_dbf__sqlite`
`raw_ferc6_dbf__sqlite`
`raw_ferc60_dbf__sqlite`
`raw_ferc1_xbrl__sqlite`
`raw_ferc2_xbrl__sqlite`
`raw_ferc6_xbrl__sqlite`
`raw_ferc60_xbrl__sqlite`
`raw_ferc714_xbrl__sqlite`

Classes#

FercPaths

Helper class to get paths to various FERC paths both local and remote.

Functions#

`_download_zipped_outputs`(paths, output_format)	Download nightly zipfile containing sqlite or parquet outputs and extract to local cache.
`_download_nightly_outputs`(→ None)	Download `ferc_to_sqlite` outputs from s3.
`_check_for_cached_db_w_compatible_provenance`(...)	Check to see if there is a compatible outputs either locally, or in nightly builds.
`ferc_to_sqlite_asset_factory`(→ dagster.AssetsDefinition)	Create a FERC-to-SQLite prerequisite asset for a specific FERC dataset.

Module Contents#

pudl.dagster.assets.raw.ferc_to_sqlite.NETWORK_ERRORS[source]#

pudl.dagster.assets.raw.ferc_to_sqlite.logger[source]#

class pudl.dagster.assets.raw.ferc_to_sqlite.FercPaths[source]#

Helper class to get paths to various FERC paths both local and remote.

data_format: Literal['dbf', 'xbrl'][source]#

local_datapackage_path: pathlib.Path[source]#

nightly_datapackage_path: upath.UPath[source]#

local_sqlite_path: pathlib.Path[source]#

nightly_sqlite_path: upath.UPath[source]#

local_duckdb_path: pathlib.Path | None = None[source]#

nightly_duckdb_path: upath.UPath | None = None[source]#

local_taxonomy_json_path: pathlib.Path | None = None[source]#

nightly_taxonomy_json_path: upath.UPath | None = None[source]#

local_parquet_path: pathlib.Path | None = None[source]#

nightly_parquet_path: upath.UPath | None = None[source]#

delete_local_outputs()[source]#: Helper function to delete local outputs before starting extraction.

classmethod from_dataset_format(dataset: pudl.settings.FercForm, data_format: Literal['dbf', 'xbrl'], paths: pudl.workspace.setup.PudlPaths) → FercPaths[source]#: Initialize class based on dataset and data_format.

pudl.dagster.assets.raw.ferc_to_sqlite._download_zipped_outputs(paths: FercPaths, output_format: Literal['sqlite', 'parquet'])[source]#: Download nightly zipfile containing sqlite or parquet outputs and extract to local cache.

pudl.dagster.assets.raw.ferc_to_sqlite._download_nightly_outputs(data_format: Literal['dbf', 'xbrl'], paths: FercPaths) → None[source]#

Download ferc_to_sqlite outputs from s3.

This will download all outputs produced by the ferc_to_sqlite process for the provided dataset and data_format. For the ‘DBF’ format, this includes the SQLite db and a datapackage JSON file, while ‘XBRL’ will include both of these plus a DuckDB file, parquet files, and the taxonomy JSON file.

pudl.dagster.assets.raw.ferc_to_sqlite._check_for_cached_db_w_compatible_provenance(dataset: pudl.settings.FercForm, data_format: Literal['dbf', 'xbrl'], zenodo_doi: str, paths: FercPaths, ferc_to_sqlite: pudl.settings.FercToSqliteDataConfig) → pudl.dagster.provenance.FercSqliteProvenanceRecord | None[source]#

Check to see if there is a compatible outputs either locally, or in nightly builds.

This function will first check the local datapackage for the specified dataset and data_format to see if it contains a FercSqliteProvenanceRecord that is compatible with the requirements of the current run. If the local datapckage doesn’t exist or contains an incompatible record, it will then download the datapackage produced by the most recent nightly build and perform the same check. If the nightly outputs are found to be compatible with the current run, then it will download all associated outputs from that run. For DBF outputs, this includes the SQLite file and the datapackage JSON file, while XBRL outputs also include a duckdb file, parquet files, and a taxonomy JSON file.

If the environment variable, PUDL_FORCE_FERC_TO_SQLITE, is set to true, then this function will immediately return None, triggering the normal extraction.

Returns:: Compatible FercSqliteProvenanceRecord if one is found, otherwise None.

pudl.dagster.assets.raw.ferc_to_sqlite.ferc_to_sqlite_asset_factory(*, dataset: pudl.settings.FercForm, data_format: Literal['dbf', 'xbrl'], extract_function: collections.abc.Callable[[dagster.AssetExecutionContext], None], op_tags: dict | None = None) → dagster.AssetsDefinition[source]#: Create a FERC-to-SQLite prerequisite asset for a specific FERC dataset.

pudl.dagster.assets.raw.ferc_to_sqlite.raw_ferc1_dbf__sqlite[source]#

pudl.dagster.assets.raw.ferc_to_sqlite.raw_ferc2_dbf__sqlite[source]#

pudl.dagster.assets.raw.ferc_to_sqlite.raw_ferc6_dbf__sqlite[source]#

pudl.dagster.assets.raw.ferc_to_sqlite.raw_ferc60_dbf__sqlite[source]#

pudl.dagster.assets.raw.ferc_to_sqlite.raw_ferc1_xbrl__sqlite[source]#

pudl.dagster.assets.raw.ferc_to_sqlite.raw_ferc2_xbrl__sqlite[source]#

pudl.dagster.assets.raw.ferc_to_sqlite.raw_ferc6_xbrl__sqlite[source]#

pudl.dagster.assets.raw.ferc_to_sqlite.raw_ferc60_xbrl__sqlite[source]#

pudl.dagster.assets.raw.ferc_to_sqlite.raw_ferc714_xbrl__sqlite[source]#