pudl.ferc_to_sqlite

Dagster definitions for the FERC to SQLite process.

Submodules

• pudl.ferc_to_sqlite.cli

Attributes

ALL_DBF_EXTRACTORS
logger
default_resources_defs
ferc_to_sqlite_full_settings
ferc_to_sqlite_full
ferc_to_sqlite_fast_settings
ferc_to_sqlite_fast
defs	A collection of dagster assets, resources, IO managers, and jobs for the FERC to

Classes

RuntimeSettings	Encodes runtime settings for the ferc_to_sqlite graphs.
EtlSettings	Main settings validation class.
XbrlFormNumber	Contains full list of supported FERC XBRL forms.

Functions

xbrl2sqlite_op_factory(→ collections.abc.Callable)	Generates xbrl2sqlite op for a given FERC form.
datastore(→ pudl.workspace.datastore.Datastore)	Dagster resource to interact with Zenodo archives.
ferc_to_sqlite_settings(...)	Dagster resource for parameterizing the ferc_to_sqlite graph.
ferc_to_sqlite()	Clone the FERC FoxPro databases and XBRL filings into SQLite.

Package Contents

pudl.ferc_to_sqlite.ALL_DBF_EXTRACTORS

pudl.ferc_to_sqlite.xbrl2sqlite_op_factory(form: pudl.settings.XbrlFormNumber) → collections.abc.Callable

Generates xbrl2sqlite op for a given FERC form.

class pudl.ferc_to_sqlite.RuntimeSettings

Bases: dagster.ConfigurableResource

Encodes runtime settings for the ferc_to_sqlite graphs.

xbrl_num_workers: None | int = None

xbrl_batch_size: int = 50

pudl.ferc_to_sqlite.datastore(init_context) → pudl.workspace.datastore.Datastore

Dagster resource to interact with Zenodo archives.

pudl.ferc_to_sqlite.ferc_to_sqlite_settings(init_context) → pudl.settings.FercToSqliteSettings

Dagster resource for parameterizing the ferc_to_sqlite graph.

This resource allows us to specify the years we want to process for each datasource in the Dagit UI.

class pudl.ferc_to_sqlite.EtlSettings(_case_sensitive: bool | None = None, _nested_model_default_partial_update: bool | None = None, _env_prefix: str | None = None, _env_file: pydantic_settings.sources.DotenvType | None = ENV_FILE_SENTINEL, _env_file_encoding: str | None = None, _env_ignore_empty: bool | None = None, _env_nested_delimiter: str | None = None, _env_nested_max_split: int | None = None, _env_parse_none_str: str | None = None, _env_parse_enums: bool | None = None, _cli_prog_name: str | None = None, _cli_parse_args: bool | list[str] | tuple[str, Ellipsis] | None = None, _cli_settings_source: pydantic_settings.sources.CliSettingsSource[Any] | None = None, _cli_parse_none_str: str | None = None, _cli_hide_none_type: bool | None = None, _cli_avoid_json: bool | None = None, _cli_enforce_required: bool | None = None, _cli_use_class_docs_for_groups: bool | None = None, _cli_exit_on_error: bool | None = None, _cli_prefix: str | None = None, _cli_flag_prefix_char: str | None = None, _cli_implicit_flags: bool | None = None, _cli_ignore_unknown_args: bool | None = None, _cli_kebab_case: bool | Literal['all', 'no_enums'] | None = None, _cli_shortcuts: collections.abc.Mapping[str, str | list[str]] | None = None, _secrets_dir: pydantic_settings.sources.PathType | None = None, **values: Any)

Bases: pydantic_settings.BaseSettings

Main settings validation class.

ferc_to_sqlite_settings: FercToSqliteSettings | None = None

datasets: DatasetsSettings | None = None

name: str | None = None

title: str | None = None

description: str | None = None

version: str | None = None

publish_destinations: list[str] = []

This is list of fsspec compatible paths to publish the output datasets to.

classmethod from_yaml(path: str) → EtlSettings

Create an EtlSettings instance from a yaml_file path.

Parameters:

path – path to a yaml file; this could be remote.

Returns:

An ETL settings object.

validate_xbrl_years()

Ensure the XBRL years in DatasetsSettings align with FercToSqliteSettings.

For each of the FERC forms that we are processing in PUDL, check to ensure that the years we are trying to process in the PUDL ETL are included in the XBRL to SQLite settings.

class pudl.ferc_to_sqlite.XbrlFormNumber(*args, **kwds)

Bases: enum.Enum

Contains full list of supported FERC XBRL forms.

FORM1 = 1

FORM2 = 2

FORM6 = 6

FORM60 = 60

FORM714 = 714

pudl.ferc_to_sqlite.logger

pudl.ferc_to_sqlite.ferc_to_sqlite()

Clone the FERC FoxPro databases and XBRL filings into SQLite.

pudl.ferc_to_sqlite.default_resources_defs

pudl.ferc_to_sqlite.ferc_to_sqlite_full_settings

pudl.ferc_to_sqlite.ferc_to_sqlite_full

pudl.ferc_to_sqlite.ferc_to_sqlite_fast_settings

pudl.ferc_to_sqlite.ferc_to_sqlite_fast

pudl.ferc_to_sqlite.defs: dagster.Definitions

A collection of dagster assets, resources, IO managers, and jobs for the FERC to SQLite ETL.