pudl.settings#

Module for validating pudl ETL data configurations.

Attributes#

logger

Classes#

`FercForm`	Contains full list of supported FERC forms.
`FrozenBaseModel`	BaseModel with global configuration.
`GenericDataConfig`	An abstract pydantic model for generic datasets.
`Ferc1DataConfig`	An immutable pydantic model to validate Ferc1DataConfig.
`Ferc714DataConfig`	An immutable pydantic model to validate Ferc714DataConfig.
`EpaCemsDataConfig`	An immutable pydantic model to validate EPA CEMS data configuration.
`EpaMatsDataConfig`	An immutable pydantic model to validate EPA MATS data configuration.
`PhmsaGasDataConfig`	An immutable pydantic model to validate PHMSA data configuration.
`Sec10kDataConfig`	An immutable pydantic model to validate SEC 10-K data configuration.
`NrelAtbDataConfig`	An immutable pydantic model to validate NREL ATB data configuration.
`Eia923DataConfig`	An immutable pydantic model to validate EIA 923 data configuration.
`Eia930DataConfig`	An immutable pydantic model to validate EIA 930 data configuration.
`Eia861DataConfig`	An immutable pydantic model to validate EIA 861 data configuration.
`Eia860DataConfig`	An immutable pydantic model to validate EIA 860 data configuration.
`Eia860mDataConfig`	An immutable pydantic model to validate EIA 860m data configuration.
`Eia757aDataConfig`	An immutable pydantic model to validate EIA 757a data configuration.
`Eia191DataConfig`	An immutable pydantic model to validate EIA 191 data configuration.
`Eia176DataConfig`	An immutable pydantic model to validate EIA 176 data configuration.
`EiaAeoDataConfig`	An immutable pydantic model to validate EIA AEO data configuration.
`VceRareDataConfig`	An immutable pydantic model to validate VCE RARE data configuration.
`CensusPepDataConfig`	An immutable pydantic model to validate Census PEP data configuration.
`GlueDataConfig`	An immutable pydantic model to validate Glue data configuration.
`GridPathRaToolkitTechType`	Enum to constrain GridPath RA Toolkit technology types.
`GridPathRaToolkitProcLevel`	Enum to constraint GridPath RA Toolkit processing levels.
`GridPathRaToolkitDataConfig`	An immutable pydantic model to validate GridPath RA Toolkit data configuration.
`EiaDataConfig`	An immutable pydantic model to validate EIA datasets data configuration.
`Rus7DataConfig`	An immutable pydantic model to validate RUS-7 datasets data configuration.
`Rus12DataConfig`	An immutable pydantic model to validate RUS Form 12 data configuration.
`PudlDataConfig`	An immutable pydantic model to validate PUDL Dataset data configuration.
`FercDbfToSqliteDataConfig`	Base class for all FERC DBF-to-SQLite data config models.
`Ferc1DbfToSqliteDataConfig`	An immutable Pydantic model to validate FERC 1 to SQLite data config.
`FercGenericXbrlToSqliteDataConfig`	An immutable pydantic model to validate Ferc1 to SQLite data config.
`Ferc1XbrlToSqliteDataConfig`	An immutable pydantic model to validate Ferc1 to SQLite data config.
`Ferc2XbrlToSqliteDataConfig`	An immutable pydantic model to validate FERC from 2 XBRL to SQLite data config.
`Ferc2DbfToSqliteDataConfig`	An immutable Pydantic model to validate FERC 2 to SQLite data config.
`Ferc6DbfToSqliteDataConfig`	An immutable Pydantic model to validate FERC 6 to SQLite data config.
`Ferc6XbrlToSqliteDataConfig`	An immutable pydantic model to validate FERC from 6 XBRL to SQLite data config.
`Ferc60DbfToSqliteDataConfig`	An immutable Pydantic model to validate FERC 60 to SQLite data config.
`Ferc60XbrlToSqliteDataConfig`	An immutable pydantic model to validate FERC from 60 XBRL to SQLite data config.
`Ferc714XbrlToSqliteDataConfig`	An immutable pydantic model to validate FERC from 714 XBRL to SQLite data config.
`FercToSqliteDataConfig`	An immutable pydantic model to validate FERC XBRL to SQLite data config.
`GlobalDataConfig`	Main settings validation class.

Functions#

_zenodo_doi_to_url(doi) → pydantic.AnyHttpUrl

Create a DOI URL out o a Zenodo DOI.

Module Contents#

pudl.settings.logger[source]#

class pudl.settings.FercForm[source]#

Bases: enum.Enum

Contains full list of supported FERC forms.

FORM1 = 1[source]#

FORM2 = 2[source]#

FORM6 = 6[source]#

FORM60 = 60[source]#

FORM714 = 714[source]#

__str__()[source]#: Format this as fercX, the way we use it everywhere else.

class pudl.settings.FrozenBaseModel(/, **data: Any)[source]#

Bases: pydantic.BaseModel

BaseModel with global configuration.

model_config[source]#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class pudl.settings.GenericDataConfig(/, **data: Any)[source]#

Bases: FrozenBaseModel

An abstract pydantic model for generic datasets.

Each dataset must specify working partitions. A dataset can have an arbitrary number of partitions.

data_source: ClassVar[pudl.metadata.classes.DataSource][source]#: The DataSource metadata object for this dataset.

validate_partitions()[source]#

Ensure that partitions and their values are valid.

Checks that:

all partitions specified by the data source exist,
partitions are not None
only known to be working partition values are specified
no duplicate partition values are specified

property partitions: list[dict[str, str]][source]#

Return list of dictionaries representing individual partitions.

Convert a list of partitions into a list of dictionaries of partitions. This is intended to be used to store partitions in a format that is easy to use with pandas.json_normalize().

class pudl.settings.Ferc1DataConfig(/, **data: Any)[source]#

Bases: GenericDataConfig

An immutable pydantic model to validate Ferc1DataConfig.

data_source: ClassVar[pudl.metadata.classes.DataSource][source]#: The DataSource metadata object for this dataset.

years: list[int][source]#: The list of years to validate.

property dbf_years[source]#: Return validated years for which DBF data is available.

property xbrl_years[source]#: Return validated years for which XBRL data is available.

class pudl.settings.Ferc714DataConfig(/, **data: Any)[source]#

Bases: GenericDataConfig

An immutable pydantic model to validate Ferc714DataConfig.

data_source: ClassVar[pudl.metadata.classes.DataSource][source]#: The DataSource metadata object for this dataset.

years: list[int][source]#: The list of years to validate.

property csv_years[source]#: Return validated years for which CSV data is available.

property xbrl_years[source]#: Return validated years for which XBRL data is available.

class pudl.settings.EpaCemsDataConfig(/, **data: Any)[source]#

Bases: GenericDataConfig

An immutable pydantic model to validate EPA CEMS data configuration.

data_source: ClassVar[pudl.metadata.classes.DataSource][source]#: The DataSource metadata object for this dataset.

year_quarters: list[str][source]#: The list of years-quarters to validate.

classmethod allow_all_keyword_year_quarters(year_quarters)[source]#: Allow users to specify [‘all’] to get all quarters.

class pudl.settings.EpaMatsDataConfig(/, **data: Any)[source]#

Bases: GenericDataConfig

An immutable pydantic model to validate EPA MATS data configuration.

data_source: ClassVar[pudl.metadata.classes.DataSource][source]#: The DataSource metadata object for this dataset.

year_quarters: list[str][source]#: The list of years-quarters to validate.

classmethod allow_all_keyword_year_quarters(year_quarters)[source]#: Allow users to specify [‘all’] to get all quarters.

class pudl.settings.PhmsaGasDataConfig(/, **data: Any)[source]#

Bases: GenericDataConfig

An immutable pydantic model to validate PHMSA data configuration.

data_source: ClassVar[pudl.metadata.classes.DataSource][source]#: The DataSource metadata object for this dataset.

years: list[int][source]#: The list of years to validate.

property extraction_years: list[int][source]#

The list of years to extract.

These are different from the standard years because the oldest years (1970 - 1989) are published with multiple years in each tab. Instead of running the extraction step on each year and filtering on the year from each tab, we extract the whole tab all at once using the first year in the tab as the partition.

class pudl.settings.Sec10kDataConfig(/, **data: Any)[source]#

Bases: GenericDataConfig

An immutable pydantic model to validate SEC 10-K data configuration.

data_source: ClassVar[pudl.metadata.classes.DataSource][source]#: The DataSource metadata object for this dataset.

years: list[int][source]#: The list of valid years for which SEC 10-K data is available.

tables: list[str][source]#

class pudl.settings.NrelAtbDataConfig(/, **data: Any)[source]#

Bases: GenericDataConfig

An immutable pydantic model to validate NREL ATB data configuration.

data_source: ClassVar[pudl.metadata.classes.DataSource][source]#: The DataSource metadata object for this dataset.

years: list[int][source]#: The list of years to validate.

class pudl.settings.Eia923DataConfig(/, **data: Any)[source]#

Bases: GenericDataConfig

An immutable pydantic model to validate EIA 923 data configuration.

data_source: ClassVar[pudl.metadata.classes.DataSource][source]#: The DataSource metadata object for this dataset.

years: list[int][source]#: The list of years to validate.

class pudl.settings.Eia930DataConfig(/, **data: Any)[source]#

Bases: GenericDataConfig

An immutable pydantic model to validate EIA 930 data configuration.

data_source: ClassVar[pudl.metadata.classes.DataSource][source]#: The DataSource metadata object for this dataset.

half_years: list[str][source]#: The list of half years to validate.

classmethod allow_all_keyword_half_years(half_years)[source]#: Allow users to specify [‘all’] to get all half-years.

class pudl.settings.Eia861DataConfig(/, **data: Any)[source]#

Bases: GenericDataConfig

An immutable pydantic model to validate EIA 861 data configuration.

data_source: ClassVar[pudl.metadata.classes.DataSource][source]#: The DataSource metadata object for this dataset.

years: list[int][source]#: The list of years to validate.

class pudl.settings.Eia860DataConfig(/, **data: Any)[source]#

Bases: GenericDataConfig

An immutable pydantic model to validate EIA 860 data configuration.

This model also checks 860m data configuration.

data_source: ClassVar[pudl.metadata.classes.DataSource][source]#: The DataSource metadata object for this dataset.

years: list[int][source]#: The list of years to validate.

eia860m: bool = True[source]#: Whether or not to incorporate an EIA-860m month.

all_eia860m_year_months: list[str][source]#: The list of all EIA-860m year-months.

eia860m_year_months: list[str][source]#: The 860m year-months to incorporate.

classmethod add_other_860m_years(v, info: pydantic.ValidationInfo) → list[str][source]#

Find extra years from EIA860m if applicable.

There’s a gap in reporting (after the new year but before the EIA 860 early release data in June) when we rely on two years worth of EIA860m data instead of just one. This function adds the last available month_year values for each year of 860m data that is not yet available in 860.

classmethod no_repeat_years(v, info: pydantic.ValidationInfo) → list[str][source]#: Make sure there are no duplicate 860m year values.

classmethod validate_eia860m_params(v, info: pydantic.ValidationInfo) → list[str][source]#: Check that the year_month values for eia860m_year_months are valid.

classmethod only_years_not_in_eia860(v, info: pydantic.ValidationInfo) → list[str][source]#: Ensure no EIA860m values are from years already in EIA860.

class pudl.settings.Eia860mDataConfig(/, **data: Any)[source]#

Bases: GenericDataConfig

An immutable pydantic model to validate EIA 860m data configuration.

data_source: ClassVar[pudl.metadata.classes.DataSource][source]#: The DataSource metadata object for this dataset.

year_months: list[str][source]#: The 860m year to date.

classmethod allow_all_keyword_year_months(year_months)[source]#: Allow users to specify [‘all’] to get all quarters.

class pudl.settings.Eia757aDataConfig(/, **data: Any)[source]#

Bases: GenericDataConfig

An immutable pydantic model to validate EIA 757a data configuration.

data_source: ClassVar[pudl.metadata.classes.DataSource][source]#: The DataSource metadata object for this dataset.

years: list[int][source]#: The list of years to validate.

class pudl.settings.Eia191DataConfig(/, **data: Any)[source]#

Bases: GenericDataConfig

An immutable pydantic model to validate EIA 191 data configuration.

data_source: ClassVar[pudl.metadata.classes.DataSource][source]#: The DataSource metadata object for this dataset.

years: list[int][source]#: The list of years to validate.

class pudl.settings.Eia176DataConfig(/, **data: Any)[source]#

Bases: GenericDataConfig

An immutable pydantic model to validate EIA 176 data configuration.

data_source: ClassVar[pudl.metadata.classes.DataSource][source]#: The DataSource metadata object for this dataset.

years: list[int][source]#: The list of years to validate.

class pudl.settings.EiaAeoDataConfig(/, **data: Any)[source]#

Bases: GenericDataConfig

An immutable pydantic model to validate EIA AEO data configuration.

data_source: ClassVar[pudl.metadata.classes.DataSource][source]#: The DataSource metadata object for this dataset.

years: list[int][source]#: The list of years to validate.

class pudl.settings.VceRareDataConfig(/, **data: Any)[source]#

Bases: GenericDataConfig

An immutable pydantic model to validate VCE RARE data configuration.

data_source: ClassVar[pudl.metadata.classes.DataSource][source]#: The DataSource metadata object for this dataset.

years: list[int][source]#: The list of years to validate.

fips: bool = True[source]#: Include FIPS codes in VCE RARE Power Dataset.

class pudl.settings.CensusPepDataConfig(/, **data: Any)[source]#

Bases: GenericDataConfig

An immutable pydantic model to validate Census PEP data configuration.

data_source: ClassVar[pudl.metadata.classes.DataSource][source]#: The DataSource metadata object for this dataset.

years: list[int][source]#: The list of years to validate.

class pudl.settings.GlueDataConfig(/, **data: Any)[source]#

Bases: FrozenBaseModel

An immutable pydantic model to validate Glue data configuration.

eia: bool = True[source]#: Include eia in glue data configuration.

ferc1: bool = True[source]#: Include ferc1 in glue data configuration.

class pudl.settings.GridPathRaToolkitTechType[source]#

Bases: enum.StrEnum

Enum to constrain GridPath RA Toolkit technology types.

WIND[source]#

SOLAR[source]#

class pudl.settings.GridPathRaToolkitProcLevel[source]#

Bases: enum.StrEnum

Enum to constraint GridPath RA Toolkit processing levels.

EXTENDED[source]#

class pudl.settings.GridPathRaToolkitDataConfig(/, **data: Any)[source]#

Bases: GenericDataConfig

An immutable pydantic model to validate GridPath RA Toolkit data configuration.

Note that the default values for technology_types, processing_levels, and daily_weather are such that by default, all working partitions will be included.

data_source: ClassVar[pudl.metadata.classes.DataSource][source]#: The DataSource metadata object for this dataset.

technology_types: list[str] = ['wind', 'solar'][source]#

processing_levels: list[str] = ['extended'][source]#

daily_weather: bool = True[source]#

classmethod deduplicate_list(v)[source]#: Deduplicate technology type and processing level values.

classmethod allowed_technology_types(v: list[str]) → list[str][source]#: Ensure that technology types are valid.

classmethod allowed_processing_levels(v: list[str]) → list[str][source]#: Ensure that processing levels are valid.

property parts: list[str][source]#: Construct parts from selected technologies, processing levels, and daily weather.

class pudl.settings.EiaDataConfig(/, **data: Any)[source]#

Bases: FrozenBaseModel

An immutable pydantic model to validate EIA datasets data configuration.

eia176: Eia176DataConfig | None = None[source]#

eia191: Eia191DataConfig | None = None[source]#

eia757a: Eia757aDataConfig | None = None[source]#

eia860: Eia860DataConfig | None = None[source]#

eia860m: Eia860mDataConfig | None = None[source]#

eia861: Eia861DataConfig | None = None[source]#

eia923: Eia923DataConfig | None = None[source]#

eia930: Eia930DataConfig | None = None[source]#

eiaaeo: EiaAeoDataConfig | None = None[source]#

classmethod default_load_all(data: dict[str, Any]) → dict[str, Any][source]#: If no datasets are specified default to all.

classmethod check_eia_dependencies(data: dict[str, Any]) → dict[str, Any][source]#

Make sure the dependencies between the eia datasets are satisfied.

Dependencies: * eia923 requires eia860 for harvesting purposes.

Parameters:: values (Dict[str, BaseModel]) – dataset data configuration.
Returns:: dataset data configuration.
Return type:: values (Dict[str, BaseModel])

validate_eia860m_consistency()[source]#

After-construction validation: ensure eia860/eia860m are consistent.

Note: this only validates and raises; it does not attempt to inject a missing eia860m config (after-mode validators run after models are constructed).

class pudl.settings.Rus7DataConfig(/, **data: Any)[source]#

Bases: GenericDataConfig

An immutable pydantic model to validate RUS-7 datasets data configuration.

data_source: ClassVar[pudl.metadata.classes.DataSource][source]#: The DataSource metadata object for this dataset.

years: list[int][source]#: The list of years to validate.

class pudl.settings.Rus12DataConfig(/, **data: Any)[source]#

Bases: GenericDataConfig

An immutable pydantic model to validate RUS Form 12 data configuration.

data_source: ClassVar[pudl.metadata.classes.DataSource][source]#: The DataSource metadata object for this dataset.

years: list[int][source]#: The list of years to validate.

class pudl.settings.PudlDataConfig(/, **data: Any)[source]#

Bases: FrozenBaseModel

An immutable pydantic model to validate PUDL Dataset data configuration.

eia: EiaDataConfig | None = None[source]#

epacems: EpaCemsDataConfig | None = None[source]#

epamats: EpaMatsDataConfig | None = None[source]#

ferc1: Ferc1DataConfig | None = None[source]#

ferc714: Ferc714DataConfig | None = None[source]#

glue: GlueDataConfig | None = None[source]#

gridpathratoolkit: GridPathRaToolkitDataConfig | None = None[source]#

nrelatb: NrelAtbDataConfig | None = None[source]#

phmsagas: PhmsaGasDataConfig | None = None[source]#

sec10k: Sec10kDataConfig | None = None[source]#

vcerare: VceRareDataConfig | None = None[source]#

censuspep: CensusPepDataConfig | None = None[source]#

rus7: Rus7DataConfig | None = None[source]#

rus12: Rus12DataConfig | None = None[source]#

classmethod default_load_all(data: dict[str, Any]) → dict[str, Any][source]#

If no datasets are specified default to all.

Parameters:: data – PUDL data configuration.
Returns:: Validated PUDL data configuration.

classmethod add_glue_data_config(data: dict[str, Any]) → dict[str, Any][source]#

Add glue data configuration if ferc1 and eia data are both requested.

Parameters:: values (Dict[str, BaseModel]) – PUDL data configuration.
Returns:: PUDL data configuration.
Return type:: values (Dict[str, BaseModel])

get_datasets()[source]#: Gets dictionary of PUDL data configuration.

make_datasources_table(data_store: pudl.workspace.datastore.Datastore) → pandas.DataFrame[source]#

Compile a table of dataset information.

There are three places we can look for information about a dataset: * the datastore (for DOIs, working partitions, etc) * the PUDL data configuration (for partitions that are used in the ETL) * the DataSource info (which is stored within the PUDL data configuration)

The PUDL data configuration and the datastore have different levels of nesting - and therefore names for datasets. The nesting happens particularly with the EIA data.

Parameters:: data_store – An initialized PUDL Datastore from which the DOI’s for each raw input dataset can be obtained.
Returns:: a dataframe describing the partitions and DOI’s of each of the datasets in this PUDL data config object.

class pudl.settings.FercDbfToSqliteDataConfig(/, **data: Any)[source]#

Bases: GenericDataConfig

Base class for all FERC DBF-to-SQLite data config models.

Declares the years and refyear attributes shared by every FERC DBF form so that FercDbfExtractor can be typed against this base rather than the looser GenericDataConfig.

years: list[int] = [][source]#: Years of DBF data to extract.

refyear: ClassVar[int][source]#: Reference year used to build the destination schema; provided by each subclass.

class pudl.settings.Ferc1DbfToSqliteDataConfig(/, **data: Any)[source]#

Bases: FercDbfToSqliteDataConfig

An immutable Pydantic model to validate FERC 1 to SQLite data config.

data_source: ClassVar[pudl.metadata.classes.DataSource][source]#: The DataSource metadata object for this dataset.

years: list[int][source]#: The list of years to validate.

refyear: ClassVar[int][source]#: The reference year for the dataset.

class pudl.settings.FercGenericXbrlToSqliteDataConfig(_case_sensitive: bool | None = None, _nested_model_default_partial_update: bool | None = None, _env_prefix: str | None = None, _env_prefix_target: pydantic_settings.sources.EnvPrefixTarget | None = None, _env_file: pydantic_settings.sources.DotenvType | None = ENV_FILE_SENTINEL, _env_file_encoding: str | None = None, _env_file_depth: int | None = None, _env_ignore_empty: bool | None = None, _env_nested_delimiter: str | None = None, _env_nested_max_split: int | None = None, _env_parse_none_str: str | None = None, _env_parse_enums: bool | None = None, _cli_prog_name: str | None = None, _cli_parse_args: bool | list[str] | tuple[str, ...] | None = None, _cli_settings_source: pydantic_settings.sources.CliSettingsSource[Any] | None = None, _cli_parse_none_str: str | None = None, _cli_hide_none_type: bool | None = None, _cli_avoid_json: bool | None = None, _cli_enforce_required: bool | None = None, _cli_use_class_docs_for_groups: bool | None = None, _cli_show_env_vars: bool | None = None, _cli_exit_on_error: bool | None = None, _cli_prefix: str | None = None, _cli_flag_prefix_char: str | None = None, _cli_implicit_flags: bool | Literal['dual', 'toggle'] | None = None, _cli_ignore_unknown_args: bool | None = None, _cli_kebab_case: bool | Literal['all', 'no_enums'] | None = None, _cli_shortcuts: collections.abc.Mapping[str, str | list[str]] | None = None, _secrets_dir: pydantic_settings.sources.PathType | None = None, _build_sources: tuple[tuple[pydantic_settings.sources.PydanticBaseSettingsSource, ...], dict[str, Any]] | None = None, **values: Any)[source]#

Bases: pydantic_settings.BaseSettings

An immutable pydantic model to validate Ferc1 to SQLite data config.

years: list[int][source]#: The list of years to validate.

class pudl.settings.Ferc1XbrlToSqliteDataConfig(_case_sensitive: bool | None = None, _nested_model_default_partial_update: bool | None = None, _env_prefix: str | None = None, _env_prefix_target: pydantic_settings.sources.EnvPrefixTarget | None = None, _env_file: pydantic_settings.sources.DotenvType | None = ENV_FILE_SENTINEL, _env_file_encoding: str | None = None, _env_file_depth: int | None = None, _env_ignore_empty: bool | None = None, _env_nested_delimiter: str | None = None, _env_nested_max_split: int | None = None, _env_parse_none_str: str | None = None, _env_parse_enums: bool | None = None, _cli_prog_name: str | None = None, _cli_parse_args: bool | list[str] | tuple[str, ...] | None = None, _cli_settings_source: pydantic_settings.sources.CliSettingsSource[Any] | None = None, _cli_parse_none_str: str | None = None, _cli_hide_none_type: bool | None = None, _cli_avoid_json: bool | None = None, _cli_enforce_required: bool | None = None, _cli_use_class_docs_for_groups: bool | None = None, _cli_show_env_vars: bool | None = None, _cli_exit_on_error: bool | None = None, _cli_prefix: str | None = None, _cli_flag_prefix_char: str | None = None, _cli_implicit_flags: bool | Literal['dual', 'toggle'] | None = None, _cli_ignore_unknown_args: bool | None = None, _cli_kebab_case: bool | Literal['all', 'no_enums'] | None = None, _cli_shortcuts: collections.abc.Mapping[str, str | list[str]] | None = None, _secrets_dir: pydantic_settings.sources.PathType | None = None, _build_sources: tuple[tuple[pydantic_settings.sources.PydanticBaseSettingsSource, ...], dict[str, Any]] | None = None, **values: Any)[source]#

Bases: FercGenericXbrlToSqliteDataConfig

An immutable pydantic model to validate Ferc1 to SQLite data config.

data_source: ClassVar[pudl.metadata.classes.DataSource][source]#

years: list[int][source]#: The list of years to validate.

class pudl.settings.Ferc2XbrlToSqliteDataConfig(_case_sensitive: bool | None = None, _nested_model_default_partial_update: bool | None = None, _env_prefix: str | None = None, _env_prefix_target: pydantic_settings.sources.EnvPrefixTarget | None = None, _env_file: pydantic_settings.sources.DotenvType | None = ENV_FILE_SENTINEL, _env_file_encoding: str | None = None, _env_file_depth: int | None = None, _env_ignore_empty: bool | None = None, _env_nested_delimiter: str | None = None, _env_nested_max_split: int | None = None, _env_parse_none_str: str | None = None, _env_parse_enums: bool | None = None, _cli_prog_name: str | None = None, _cli_parse_args: bool | list[str] | tuple[str, ...] | None = None, _cli_settings_source: pydantic_settings.sources.CliSettingsSource[Any] | None = None, _cli_parse_none_str: str | None = None, _cli_hide_none_type: bool | None = None, _cli_avoid_json: bool | None = None, _cli_enforce_required: bool | None = None, _cli_use_class_docs_for_groups: bool | None = None, _cli_show_env_vars: bool | None = None, _cli_exit_on_error: bool | None = None, _cli_prefix: str | None = None, _cli_flag_prefix_char: str | None = None, _cli_implicit_flags: bool | Literal['dual', 'toggle'] | None = None, _cli_ignore_unknown_args: bool | None = None, _cli_kebab_case: bool | Literal['all', 'no_enums'] | None = None, _cli_shortcuts: collections.abc.Mapping[str, str | list[str]] | None = None, _secrets_dir: pydantic_settings.sources.PathType | None = None, _build_sources: tuple[tuple[pydantic_settings.sources.PydanticBaseSettingsSource, ...], dict[str, Any]] | None = None, **values: Any)[source]#

Bases: FercGenericXbrlToSqliteDataConfig

An immutable pydantic model to validate FERC from 2 XBRL to SQLite data config.

data_source: ClassVar[pudl.metadata.classes.DataSource][source]#

years: list[int][source]#: The list of years to validate.

class pudl.settings.Ferc2DbfToSqliteDataConfig(/, **data: Any)[source]#

Bases: FercDbfToSqliteDataConfig

An immutable Pydantic model to validate FERC 2 to SQLite data config.

data_source: ClassVar[pudl.metadata.classes.DataSource][source]#: The DataSource metadata object for this dataset.

years: list[int][source]#: The list of years to validate.

refyear: ClassVar[int][source]#: The reference year for the dataset.

class pudl.settings.Ferc6DbfToSqliteDataConfig(/, **data: Any)[source]#

Bases: FercDbfToSqliteDataConfig

An immutable Pydantic model to validate FERC 6 to SQLite data config.

data_source: ClassVar[pudl.metadata.classes.DataSource][source]#: The DataSource metadata object for this dataset.

years: list[int][source]#: The list of years to validate.

refyear: ClassVar[int][source]#: The reference year for the dataset.

class pudl.settings.Ferc6XbrlToSqliteDataConfig(_case_sensitive: bool | None = None, _nested_model_default_partial_update: bool | None = None, _env_prefix: str | None = None, _env_prefix_target: pydantic_settings.sources.EnvPrefixTarget | None = None, _env_file: pydantic_settings.sources.DotenvType | None = ENV_FILE_SENTINEL, _env_file_encoding: str | None = None, _env_file_depth: int | None = None, _env_ignore_empty: bool | None = None, _env_nested_delimiter: str | None = None, _env_nested_max_split: int | None = None, _env_parse_none_str: str | None = None, _env_parse_enums: bool | None = None, _cli_prog_name: str | None = None, _cli_parse_args: bool | list[str] | tuple[str, ...] | None = None, _cli_settings_source: pydantic_settings.sources.CliSettingsSource[Any] | None = None, _cli_parse_none_str: str | None = None, _cli_hide_none_type: bool | None = None, _cli_avoid_json: bool | None = None, _cli_enforce_required: bool | None = None, _cli_use_class_docs_for_groups: bool | None = None, _cli_show_env_vars: bool | None = None, _cli_exit_on_error: bool | None = None, _cli_prefix: str | None = None, _cli_flag_prefix_char: str | None = None, _cli_implicit_flags: bool | Literal['dual', 'toggle'] | None = None, _cli_ignore_unknown_args: bool | None = None, _cli_kebab_case: bool | Literal['all', 'no_enums'] | None = None, _cli_shortcuts: collections.abc.Mapping[str, str | list[str]] | None = None, _secrets_dir: pydantic_settings.sources.PathType | None = None, _build_sources: tuple[tuple[pydantic_settings.sources.PydanticBaseSettingsSource, ...], dict[str, Any]] | None = None, **values: Any)[source]#

Bases: FercGenericXbrlToSqliteDataConfig

An immutable pydantic model to validate FERC from 6 XBRL to SQLite data config.

data_source: ClassVar[pudl.metadata.classes.DataSource][source]#

years: list[int][source]#: The list of years to validate.

class pudl.settings.Ferc60DbfToSqliteDataConfig(/, **data: Any)[source]#

Bases: FercDbfToSqliteDataConfig

An immutable Pydantic model to validate FERC 60 to SQLite data config.

data_source: ClassVar[pudl.metadata.classes.DataSource][source]#: The DataSource metadata object for this dataset.

years: list[int][source]#: The list of years to validate.

refyear: ClassVar[int][source]#: The reference year for the dataset.

class pudl.settings.Ferc60XbrlToSqliteDataConfig(_case_sensitive: bool | None = None, _nested_model_default_partial_update: bool | None = None, _env_prefix: str | None = None, _env_prefix_target: pydantic_settings.sources.EnvPrefixTarget | None = None, _env_file: pydantic_settings.sources.DotenvType | None = ENV_FILE_SENTINEL, _env_file_encoding: str | None = None, _env_file_depth: int | None = None, _env_ignore_empty: bool | None = None, _env_nested_delimiter: str | None = None, _env_nested_max_split: int | None = None, _env_parse_none_str: str | None = None, _env_parse_enums: bool | None = None, _cli_prog_name: str | None = None, _cli_parse_args: bool | list[str] | tuple[str, ...] | None = None, _cli_settings_source: pydantic_settings.sources.CliSettingsSource[Any] | None = None, _cli_parse_none_str: str | None = None, _cli_hide_none_type: bool | None = None, _cli_avoid_json: bool | None = None, _cli_enforce_required: bool | None = None, _cli_use_class_docs_for_groups: bool | None = None, _cli_show_env_vars: bool | None = None, _cli_exit_on_error: bool | None = None, _cli_prefix: str | None = None, _cli_flag_prefix_char: str | None = None, _cli_implicit_flags: bool | Literal['dual', 'toggle'] | None = None, _cli_ignore_unknown_args: bool | None = None, _cli_kebab_case: bool | Literal['all', 'no_enums'] | None = None, _cli_shortcuts: collections.abc.Mapping[str, str | list[str]] | None = None, _secrets_dir: pydantic_settings.sources.PathType | None = None, _build_sources: tuple[tuple[pydantic_settings.sources.PydanticBaseSettingsSource, ...], dict[str, Any]] | None = None, **values: Any)[source]#

Bases: FercGenericXbrlToSqliteDataConfig

An immutable pydantic model to validate FERC from 60 XBRL to SQLite data config.

data_source: ClassVar[pudl.metadata.classes.DataSource][source]#

years: list[int][source]#: The list of years to validate.

class pudl.settings.Ferc714XbrlToSqliteDataConfig(_case_sensitive: bool | None = None, _nested_model_default_partial_update: bool | None = None, _env_prefix: str | None = None, _env_prefix_target: pydantic_settings.sources.EnvPrefixTarget | None = None, _env_file: pydantic_settings.sources.DotenvType | None = ENV_FILE_SENTINEL, _env_file_encoding: str | None = None, _env_file_depth: int | None = None, _env_ignore_empty: bool | None = None, _env_nested_delimiter: str | None = None, _env_nested_max_split: int | None = None, _env_parse_none_str: str | None = None, _env_parse_enums: bool | None = None, _cli_prog_name: str | None = None, _cli_parse_args: bool | list[str] | tuple[str, ...] | None = None, _cli_settings_source: pydantic_settings.sources.CliSettingsSource[Any] | None = None, _cli_parse_none_str: str | None = None, _cli_hide_none_type: bool | None = None, _cli_avoid_json: bool | None = None, _cli_enforce_required: bool | None = None, _cli_use_class_docs_for_groups: bool | None = None, _cli_show_env_vars: bool | None = None, _cli_exit_on_error: bool | None = None, _cli_prefix: str | None = None, _cli_flag_prefix_char: str | None = None, _cli_implicit_flags: bool | Literal['dual', 'toggle'] | None = None, _cli_ignore_unknown_args: bool | None = None, _cli_kebab_case: bool | Literal['all', 'no_enums'] | None = None, _cli_shortcuts: collections.abc.Mapping[str, str | list[str]] | None = None, _secrets_dir: pydantic_settings.sources.PathType | None = None, _build_sources: tuple[tuple[pydantic_settings.sources.PydanticBaseSettingsSource, ...], dict[str, Any]] | None = None, **values: Any)[source]#

Bases: FercGenericXbrlToSqliteDataConfig

An immutable pydantic model to validate FERC from 714 XBRL to SQLite data config.

data_source: ClassVar[pudl.metadata.classes.DataSource][source]#

years: list[int][source]#: The list of years to validate.

class pudl.settings.FercToSqliteDataConfig(_case_sensitive: bool | None = None, _nested_model_default_partial_update: bool | None = None, _env_prefix: str | None = None, _env_prefix_target: pydantic_settings.sources.EnvPrefixTarget | None = None, _env_file: pydantic_settings.sources.DotenvType | None = ENV_FILE_SENTINEL, _env_file_encoding: str | None = None, _env_file_depth: int | None = None, _env_ignore_empty: bool | None = None, _env_nested_delimiter: str | None = None, _env_nested_max_split: int | None = None, _env_parse_none_str: str | None = None, _env_parse_enums: bool | None = None, _cli_prog_name: str | None = None, _cli_parse_args: bool | list[str] | tuple[str, ...] | None = None, _cli_settings_source: pydantic_settings.sources.CliSettingsSource[Any] | None = None, _cli_parse_none_str: str | None = None, _cli_hide_none_type: bool | None = None, _cli_avoid_json: bool | None = None, _cli_enforce_required: bool | None = None, _cli_use_class_docs_for_groups: bool | None = None, _cli_show_env_vars: bool | None = None, _cli_exit_on_error: bool | None = None, _cli_prefix: str | None = None, _cli_flag_prefix_char: str | None = None, _cli_implicit_flags: bool | Literal['dual', 'toggle'] | None = None, _cli_ignore_unknown_args: bool | None = None, _cli_kebab_case: bool | Literal['all', 'no_enums'] | None = None, _cli_shortcuts: collections.abc.Mapping[str, str | list[str]] | None = None, _secrets_dir: pydantic_settings.sources.PathType | None = None, _build_sources: tuple[tuple[pydantic_settings.sources.PydanticBaseSettingsSource, ...], dict[str, Any]] | None = None, **values: Any)[source]#

Bases: pydantic_settings.BaseSettings

An immutable pydantic model to validate FERC XBRL to SQLite data config.

ferc1_dbf: Ferc1DbfToSqliteDataConfig | None = None[source]#

ferc1_xbrl: Ferc1XbrlToSqliteDataConfig | None = None[source]#

ferc2_dbf: Ferc2DbfToSqliteDataConfig | None = None[source]#

ferc2_xbrl: Ferc2XbrlToSqliteDataConfig | None = None[source]#

ferc6_dbf: Ferc6DbfToSqliteDataConfig | None = None[source]#

ferc6_xbrl: Ferc6XbrlToSqliteDataConfig | None = None[source]#

ferc60_dbf: Ferc60DbfToSqliteDataConfig | None = None[source]#

ferc60_xbrl: Ferc60XbrlToSqliteDataConfig | None = None[source]#

ferc714_xbrl: Ferc714XbrlToSqliteDataConfig | None = None[source]#

classmethod default_load_all(data: dict[str, Any]) → dict[str, Any][source]#: If no datasets are specified default to all.

get_data_config(dataset: str | FercForm, data_format: Literal['dbf', 'xbrl']) → FercGenericXbrlToSqliteDataConfig | FercDbfToSqliteDataConfig | None[source]#

Look up extraction settings by dataset (fercX) and data format (dbf or xbrl).

Returns None if the dataset/format combination is not configured.

get_dataset_years(dataset: str | FercForm, data_format: Literal['dbf', 'xbrl']) → list[int][source]#

Look up extraction years by dataset (fercX) and data format (dbf or xbrl).

Raises:: ValueError – if the dataset/format combination is not configured.

class pudl.settings.GlobalDataConfig(_case_sensitive: bool | None = None, _nested_model_default_partial_update: bool | None = None, _env_prefix: str | None = None, _env_prefix_target: pydantic_settings.sources.EnvPrefixTarget | None = None, _env_file: pydantic_settings.sources.DotenvType | None = ENV_FILE_SENTINEL, _env_file_encoding: str | None = None, _env_file_depth: int | None = None, _env_ignore_empty: bool | None = None, _env_nested_delimiter: str | None = None, _env_nested_max_split: int | None = None, _env_parse_none_str: str | None = None, _env_parse_enums: bool | None = None, _cli_prog_name: str | None = None, _cli_parse_args: bool | list[str] | tuple[str, ...] | None = None, _cli_settings_source: pydantic_settings.sources.CliSettingsSource[Any] | None = None, _cli_parse_none_str: str | None = None, _cli_hide_none_type: bool | None = None, _cli_avoid_json: bool | None = None, _cli_enforce_required: bool | None = None, _cli_use_class_docs_for_groups: bool | None = None, _cli_show_env_vars: bool | None = None, _cli_exit_on_error: bool | None = None, _cli_prefix: str | None = None, _cli_flag_prefix_char: str | None = None, _cli_implicit_flags: bool | Literal['dual', 'toggle'] | None = None, _cli_ignore_unknown_args: bool | None = None, _cli_kebab_case: bool | Literal['all', 'no_enums'] | None = None, _cli_shortcuts: collections.abc.Mapping[str, str | list[str]] | None = None, _secrets_dir: pydantic_settings.sources.PathType | None = None, _build_sources: tuple[tuple[pydantic_settings.sources.PydanticBaseSettingsSource, ...], dict[str, Any]] | None = None, **values: Any)[source]#

Bases: pydantic_settings.BaseSettings

Main settings validation class.

ferc_to_sqlite: FercToSqliteDataConfig | None = None[source]#

pudl: PudlDataConfig | None = None[source]#

name: str | None = None[source]#

title: str | None = None[source]#

description: str | None = None[source]#

version: str | None = None[source]#

classmethod from_yaml(path: str | pathlib.Path) → GlobalDataConfig[source]#

Create validated GlobalDataConfig from a local YAML file path.

Parameters:: path – Path to a YAML file. Relative paths are resolved against the current working directory and ~ is expanded.
Returns:: A GlobalDataConfig object.

validate_xbrl_years()[source]#

Ensure the XBRL years in PudlDataConfig align with FercToSqliteDataConfig.

For each of the FERC forms that we are processing in PUDL, check to ensure that the years we are trying to process in the PUDL ETL are included in the XBRL to SQLite data config.

property pudl_data_config: PudlDataConfig[source]#: Return validated PUDL data config or raise if it is unavailable.

pudl.settings._zenodo_doi_to_url(doi: pudl.workspace.datastore.ZenodoDoi) → pydantic.AnyHttpUrl[source]#: Create a DOI URL out o a Zenodo DOI.