pudl.transform.eia176#

Module to perform data cleaning functions on EIA176 data tables.

Attributes#

`logger`
`name_cleaner`
`DROP_OPERATING_STATES`
`CONTINUATION_LINES_ALLOWED_NON_SUBDIVISION_CODES`
`UNKNOWN_TYPES`
`SUPPLEMENTAL_GASEOUS_FUEL_TYPE_MAP`
`OTHER_DISPOSITION_TYPE_MAP`
`NATIONAL_ADJUSTMENT_STATE_CODES`
`MAX_NATIONAL_ADJUSTMENT_RECORDS`

Functions#

`_core_eia176__numeric_data`(raw_eia176__numeric_data) → tuple[dagster.Output, dagster.Output]	Process EIA 176 custom report data into company and aggregate outputs.
`get_wide_table`(long_table, primary_key) → pandas.DataFrame	Take a 'long' or entity-attribute-value table and return a wide table with one column per attribute/variable.
`_subdivision_code_map`(core_pudl__codes_subdivisions) → pandas.Series	Map subdivision names and codes to canonical two-letter codes.
`normalize_continuation_line_location_codes`(df, ...) → pandas.DataFrame	Validate and normalize EIA-176 continuation line location codes.
`_get_continuation_code_type`(continuation_code, ...) → pandas.Series	Classify EIA-176 continuation codes as subnational or national/other codes.
`_find_continuation_line_total_mismatches`(...) → pandas.DataFrame	Compare detailed continuation line totals with reported company-level totals.
`validate_totals`(_core_eia176__yearly_company_data, ...) → dagster.AssetCheckResult	Compare reported and calculated totals for different geographical aggregates.
`core_eia176__yearly_gas_disposition_by_consumer`(...) → pandas.DataFrame	Produce annual company-level gas disposition by consumer class (EIA-176).
`core_eia176__yearly_gas_imports`(...) → pandas.DataFrame	Produce company-level detailed annual gas imports (EIA-176, Line 3.0).
`core_eia176__yearly_supplemental_gaseous_fuel_supplies`(...) → pandas.DataFrame	Produce detailed annual supplemental gaseous fuel supplies (EIA-176, Line 6.0).
`core_eia176__yearly_gas_exports`(...) → pandas.DataFrame	Produce detailed annual out-of-state gas deliveries (EIA-176, Line 14.0).
`core_eia176__yearly_gas_disposition_other`(...) → pandas.DataFrame	Produce detailed annual gas disposition to other uses (EIA-176, Line 18.4).
`core_eia176__yearly_gas_supply`(...) → pandas.DataFrame	Produce company-level natural and supplemental gas supply (EIA176, Lines 1.0-7.0).
`_compare_eia176_continuation_line_total`(df, ...) → pandas.DataFrame	Compare a wide EIA-176 value column against summed continuation-line values.
`core_eia176__yearly_gas_disposition`(...) → pandas.DataFrame	Produce company-level gas disposition (EIA176, Lines 9.0 and 12.0-20.0).
`core_eia176__yearly_liquefied_natural_gas_inventory`(...)	Operator's LNG storage volume and capacity (EIA176, lines 8.0-8.2).
`core_eia176__yearly_company_characteristics`(...) → pandas.DataFrame	Produce annual company-level operational and ownership characteristics (EIA-176).
`_normalize_operating_states`(...[, column])	Map full state names to their postal abbreviations.

Module Contents#

pudl.transform.eia176.logger[source]#

pudl.transform.eia176.name_cleaner[source]#

pudl.transform.eia176.DROP_OPERATING_STATES = ('fed. gulf of mexico', 'mexico', 'other')[source]#

pudl.transform.eia176.CONTINUATION_LINES_ALLOWED_NON_SUBDIVISION_CODES[source]#

pudl.transform.eia176.UNKNOWN_TYPES = ('', '.', '0', 'na', 'nan', 'n/a', 'n.a.', 'none', 'not applicable', 'not available')[source]#

pudl.transform.eia176.SUPPLEMENTAL_GASEOUS_FUEL_TYPE_MAP[source]#

pudl.transform.eia176.OTHER_DISPOSITION_TYPE_MAP[source]#

pudl.transform.eia176.NATIONAL_ADJUSTMENT_STATE_CODES[source]#

pudl.transform.eia176.MAX_NATIONAL_ADJUSTMENT_RECORDS = 28[source]#

pudl.transform.eia176._core_eia176__numeric_data(raw_eia176__numeric_data: pandas.DataFrame) → tuple[dagster.Output, dagster.Output][source]#

Process EIA 176 custom report data into company and aggregate outputs.

Take raw dataframe produced by querying all forms from the EIA 176 custom report and return two wide tables with primary keys and one column per variable.

One table with data for each year and company, one with state- and US-level aggregates per year.

pudl.transform.eia176.get_wide_table(long_table: pandas.DataFrame, primary_key: list[str]) → pandas.DataFrame[source]#: Take a ‘long’ or entity-attribute-value table and return a wide table with one column per attribute/variable.

pudl.transform.eia176._subdivision_code_map(core_pudl__codes_subdivisions: pandas.DataFrame) → pandas.Series[source]#: Map subdivision names and codes to canonical two-letter codes.

pudl.transform.eia176.normalize_continuation_line_location_codes(df: pandas.DataFrame, core_pudl__codes_subdivisions: pandas.DataFrame, column: str) → pandas.DataFrame[source]#

Validate and normalize EIA-176 continuation line location codes.

Values matching core_pudl__codes_subdivisions by name or code are converted to canonical two-letter state, province, or territory codes. Known EIA continuation line codes that are not state, province, or territory codes are allowed through unchanged so unexpected values still fail loudly. These allowed values are raw EIA REF_CODE values, not standardized ISO country codes.

pudl.transform.eia176._get_continuation_code_type(continuation_code: pandas.Series, core_pudl__codes_subdivisions: pandas.DataFrame) → pandas.Series[source]#: Classify EIA-176 continuation codes as subnational or national/other codes.

pudl.transform.eia176._find_continuation_line_total_mismatches(detail_records: pandas.DataFrame, _core_eia176__yearly_company_data: pandas.DataFrame, company_total_column: str) → pandas.DataFrame[source]#: Compare detailed continuation line totals with reported company-level totals.

pudl.transform.eia176.validate_totals(_core_eia176__yearly_company_data: pandas.DataFrame, _core_eia176__yearly_aggregate_data: pandas.DataFrame) → dagster.AssetCheckResult[source]#

Compare reported and calculated totals for different geographical aggregates.

EIA reports an adjustment company at the area-level, so these values are expected to be identical. Once we validate this, we can preserve the detailed data and discard the aggregate data to remove duplicate information.

pudl.transform.eia176.core_eia176__yearly_gas_disposition_by_consumer(_core_eia176__yearly_company_data: pandas.DataFrame, core_pudl__codes_subdivisions: pandas.DataFrame) → pandas.DataFrame[source]#

Produce annual company-level gas disposition by consumer class (EIA-176).

Transforms company-level EIA-176 data into a normalized table with one row per (“report_year”, “operator_id_eia”, “operating_state”, “customer_class”, “revenue_class”) and three value columns: “consumers”, “revenue”, and “volume_mcf”.

Processing:

Select sales and transport metrics for residential, commercial, industrial, electric_power, vehicle_fuel, and other.
Validate that sales_volume + transport_volume == total volume per customer class.
Normalize operating_state to two-letter subdivision codes via core_pudl__codes_subdivisions; drop rows with unknown states (these rows must contain zeros across value columns).
Drop rows where all of consumers/revenue/volume_mcf are NULL.

Parameters:

_core_eia176__yearly_company_data – Wide company-level EIA-176 data with per-metric columns.
core_pudl__codes_subdivisions – Mapping from subdivision_name to subdivision_code used to normalize operating_state.

Raises:

AssertionError – If component volumes don’t sum to totals, or if rows with unknown operating_state contain non-zero values.

Notes

volume_mcf is thousand cubic feet (MCF).
consumers is a count; revenue is nominal USD as reported.
customer_class and revenue_class are returned as categoricals.

pudl.transform.eia176.core_eia176__yearly_gas_imports(raw_eia176__continuation_text_lines: pandas.DataFrame, core_pudl__codes_subdivisions: pandas.DataFrame, _core_eia176__yearly_company_data: pandas.DataFrame) → pandas.DataFrame[source]#: Produce company-level detailed annual gas imports (EIA-176, Line 3.0).

pudl.transform.eia176.core_eia176__yearly_supplemental_gaseous_fuel_supplies(raw_eia176__continuation_text_lines: pandas.DataFrame, core_pudl__codes_subdivisions: pandas.DataFrame, _core_eia176__yearly_company_data: pandas.DataFrame) → pandas.DataFrame[source]#: Produce detailed annual supplemental gaseous fuel supplies (EIA-176, Line 6.0).

pudl.transform.eia176.core_eia176__yearly_gas_exports(raw_eia176__continuation_text_lines: pandas.DataFrame, core_pudl__codes_subdivisions: pandas.DataFrame, _core_eia176__yearly_company_data: pandas.DataFrame) → pandas.DataFrame[source]#: Produce detailed annual out-of-state gas deliveries (EIA-176, Line 14.0).

pudl.transform.eia176.core_eia176__yearly_gas_disposition_other(raw_eia176__continuation_text_lines: pandas.DataFrame, core_pudl__codes_subdivisions: pandas.DataFrame, _core_eia176__yearly_company_data: pandas.DataFrame) → pandas.DataFrame[source]#: Produce detailed annual gas disposition to other uses (EIA-176, Line 18.4).

pudl.transform.eia176.core_eia176__yearly_gas_supply(_core_eia176__yearly_company_data: pandas.DataFrame, core_pudl__codes_subdivisions: pandas.DataFrame, raw_eia176__continuation_text_lines: pandas.DataFrame) → pandas.DataFrame[source]#: Produce company-level natural and supplemental gas supply (EIA176, Lines 1.0-7.0).

pudl.transform.eia176._compare_eia176_continuation_line_total(df: pandas.DataFrame, raw_eia176__continuation_text_lines: pandas.DataFrame, *, line: int, value_col: str, continuation_col: str) → pandas.DataFrame[source]#: Compare a wide EIA-176 value column against summed continuation-line values.

pudl.transform.eia176.core_eia176__yearly_gas_disposition(_core_eia176__yearly_company_data: pandas.DataFrame, core_pudl__codes_subdivisions: pandas.DataFrame, raw_eia176__continuation_text_lines: pandas.DataFrame) → pandas.DataFrame[source]#: Produce company-level gas disposition (EIA176, Lines 9.0 and 12.0-20.0).

pudl.transform.eia176.core_eia176__yearly_liquefied_natural_gas_inventory(_core_eia176__yearly_company_data: pandas.DataFrame, core_pudl__codes_subdivisions: pandas.DataFrame)[source]#: Operator’s LNG storage volume and capacity (EIA176, lines 8.0-8.2).

pudl.transform.eia176.core_eia176__yearly_company_characteristics(raw_eia176__operation_types_and_sector_items: pandas.DataFrame, _core_eia176__yearly_company_data: pandas.DataFrame) → pandas.DataFrame[source]#

Produce annual company-level operational and ownership characteristics (EIA-176).

Combines boolean operation-type and ownership-type columns from raw_eia176__operation_types_and_sector_items with numeric and boolean fields from _core_eia176__yearly_company_data.

One row per (report_year, operator_id_eia).

Processing:

Select all is_* boolean columns and other_ownership_description from the raw operation types table. The operating_state column in that table already contains two-letter postal codes.
Left-join company-level numeric and boolean fields from the company data table.
Drop national-level adjustment records (operating_state in NATIONAL_ADJUSTMENT_STATE_CODES): these represent cross-border totals, not state-level operator data.
Convert "X"-encoded columns to boolean (True where marked, False otherwise).
Replace the 1.0 float artifact in other_ownership_description (present in 2012-2015 LNG terminal records) with pd.NA, then apply simplify_strings().

Parameters:

raw_eia176__operation_types_and_sector_items – Raw EIA-176 operation types and sector items table; primary source for all is_* columns and operating_state.
_core_eia176__yearly_company_data – Wide company-level EIA-176 data; provides numeric and boolean company fields.

pudl.transform.eia176._normalize_operating_states(core_pudl__codes_subdivisions, df, column: str = 'operating_state')[source]#

Map full state names to their postal abbreviations.

This uses the latest year of Census PEP data as the reference. If a full state name is not included in this data, it is set to NA.