pudl.metadata.dtypes#

Canonical PUDL dtype mappings and dtype-application helpers.

This module serves two related purposes:

  1. Define the canonical mapping from PUDL’s simplified field types

    (string, integer, geometry, etc.) to the concrete dtype objects used by supported tabular backends like pandas, Polars, SQLite, DuckDB, and PyArrow.

  2. Expose helper functions that resolve those backend dtypes for either the global

    field metadata or a concrete PUDL resource schema, and apply them to pandas or Polars dataframes.

When a concrete resource is provided to get_pudl_dtypes(), the resource schema is authoritative. That means resource-specific field typing and enum/category information already encoded in PUDL_PACKAGE will be used directly where possible.

Not every backend supports every canonical PUDL field type. In particular, some backends do not yet support PUDL’s geometry fields. In those cases the dtype helpers intentionally omit unsupported fields rather than returning an incompatible dtype mapping.

This module intentionally keeps the import of PUDL_PACKAGE local to the helper functions that need it, so the metadata class graph does not introduce a module import cycle.

Attributes#

FIELD_DTYPES_POLARS

Polars data type by simplified PUDL field type.

FIELD_DTYPES_DUCKDB

DuckDB data type by simplified PUDL field type.

FIELD_DTYPES_PANDAS

Pandas data type by simplified PUDL field type.

FIELD_DTYPES_PYARROW

FIELD_DTYPES_SQLITE

SQLAlchemy column types by simplified PUDL field type.

CONSTRAINT_DTYPES

Python types for field constraints by simplified PUDL field type.

PERIODS

Functions converting datetimes to period start times, by time period.

PudlDtypeBackend

_DTYPE_MAPS_BY_BACKEND

Functions#

_get_applicable_dtypes(→ dict[str, Any])

Return the subset of resolved dtypes needed to cast the given columns.

get_pudl_dtypes(→ dict[str, Any])

Compile a dictionary of field dtypes.

_get_pudl_namespace_dtypes(→ dict[str, Any])

Compile a dictionary of field dtypes based on the namespace-level field metadata.

_get_pudl_resource_dtypes(→ dict[str, Any])

Compile a dictionary of field dtypes for a specific PUDL resource.

apply_pudl_dtypes(...)

Apply dtypes to those columns in a dataframe that have PUDL types defined.

apply_pudl_dtypes_polars(→ polars.LazyFrame)

Apply dtypes to those columns in a dataframe that have PUDL types defined.

Module Contents#

pudl.metadata.dtypes.FIELD_DTYPES_POLARS: dict[str, type[polars.DataType] | polars.DataType][source]#

Polars data type by simplified PUDL field type.

pudl.metadata.dtypes.FIELD_DTYPES_DUCKDB: dict[str, duckdb.sqltypes.DuckDBPyType][source]#

DuckDB data type by simplified PUDL field type.

pudl.metadata.dtypes.FIELD_DTYPES_PANDAS: dict[str, str][source]#

Pandas data type by simplified PUDL field type.

pudl.metadata.dtypes.FIELD_DTYPES_PYARROW: dict[str, pyarrow.DataType][source]#
pudl.metadata.dtypes.FIELD_DTYPES_SQLITE: dict[str, type[sqlalchemy.types.TypeEngine] | sqlalchemy.types.TypeEngine][source]#

SQLAlchemy column types by simplified PUDL field type.

pudl.metadata.dtypes.CONSTRAINT_DTYPES: dict[str, type][source]#

Python types for field constraints by simplified PUDL field type.

pudl.metadata.dtypes.PERIODS: dict[str, collections.abc.Callable[[pandas.Series], pandas.Series | pandas.DataFrame]][source]#

Functions converting datetimes to period start times, by time period.

pudl.metadata.dtypes.PudlDtypeBackend[source]#
pudl.metadata.dtypes._DTYPE_MAPS_BY_BACKEND: dict[PudlDtypeBackend, dict[str, Any]][source]#
pudl.metadata.dtypes._get_applicable_dtypes(columns: list[str], field_namespace: str | None, resource: str | None, dtype_backend: PudlDtypeBackend, strict: bool) → dict[str, Any][source]#

Return the subset of resolved dtypes needed to cast the given columns.

pudl.metadata.dtypes.get_pudl_dtypes(field_namespace: str | None = None, resource: str | None = None, dtype_backend: PudlDtypeBackend = 'pandas') → dict[str, Any][source]#

Compile a dictionary of field dtypes.

Parameters:

  • field_namespace – The field namespace (e.g. ferc1, eia) to use for overriding the default field types. If None, no namespace overrides are applied.

  • resource – The resource (table) name whose schema should define the field types. If provided, resource field types are authoritative.

  • dtype_backend – Named dtype backend to compile. Supported values are "pandas", "polars", "sqlite", "duckdb", and "pyarrow".

Returns:

A mapping of PUDL field names to their associated data types.

pudl.metadata.dtypes._get_pudl_namespace_dtypes(field_namespace: str | None = None, dtype_backend: PudlDtypeBackend = 'pandas') → dict[str, Any][source]#

Compile a dictionary of field dtypes based on the namespace-level field metadata.

If no field_namespace is provided, the global PUDL field metadata is used.

Parameters:

  • field_namespace – The field namespace (e.g. ferc1, eia) whose schema should define the field types. If None, no namespace overrides are applied.

  • dtype_backend – Named dtype backend to compile. Supported values are "pandas", "polars", "sqlite", "duckdb", and "pyarrow".

Returns:

A mapping of PUDL field names to their associated data types.

pudl.metadata.dtypes._get_pudl_resource_dtypes(resource: str, dtype_backend: PudlDtypeBackend = 'pandas') → dict[str, Any][source]#

Compile a dictionary of field dtypes for a specific PUDL resource.

Parameters:

  • resource – The resource (table) name whose schema should define the field types. If provided, resource field types are authoritative.

  • dtype_backend – Named dtype backend to compile. Supported values are "pandas", "polars", "sqlite", "duckdb", and "pyarrow".

Returns:

A mapping of PUDL field names to their associated data types.

pudl.metadata.dtypes.apply_pudl_dtypes(df: pandas.DataFrame | geopandas.GeoDataFrame, field_namespace: str | None = None, resource: str | None = None, strict: bool = False) → pandas.DataFrame | geopandas.GeoDataFrame[source]#

Apply dtypes to those columns in a dataframe that have PUDL types defined.

Parameters:

  • df – The dataframe to apply types to. Not all columns need to have types defined in the PUDL metadata unless you pass strict=True.

  • field_namespace – The field namespace to use for overrides, if any.

  • resource – The resource (table) name whose schema should define the field types. If provided, resource field types are authoritative.

  • strict – whether or not all columns need a corresponding field.

Returns:

The input dataframe, but with standard PUDL types applied.

pudl.metadata.dtypes.apply_pudl_dtypes_polars(lf: polars.LazyFrame, field_namespace: str | None = None, resource: str | None = None, strict: bool = False) → polars.LazyFrame[source]#

Apply dtypes to those columns in a dataframe that have PUDL types defined.

Parameters:

  • lf – The LazyFrame to apply types to. Not all columns need to have types defined in the PUDL metadata unless you pass strict=True.

  • field_namespace – The field namespace to use for overrides, if any.

  • resource – The resource (table) name whose schema should define the field types. If provided, resource field types are authoritative.

  • strict – whether or not all columns need a corresponding field.

Returns:

The input LazyFrame, but with standard PUDL types applied.