pudl.extract.xbrl#

Generic extractor for all FERC XBRL data.

Attributes#

logger

Classes#

_FilteringStream

Pass-through text stream that drops matching noisy lines.

FercXbrlDatastore

Simple datastore wrapper for accessing ferc1 xbrl resources.

Functions#

_suppress_arelle_message_spam()

Filter known Arelle console spam without suppressing normal logs.

convert_form(→ None)

Clone a single FERC XBRL form to SQLite.

Module Contents#

pudl.extract.xbrl.logger[source]#
class pudl.extract.xbrl._FilteringStream(wrapped, drop_patterns: list[re.Pattern[str]])[source]#

Pass-through text stream that drops matching noisy lines.

_wrapped[source]#
_drop_patterns[source]#
_dropped_previous_line = False[source]#
write(text: str) → int[source]#
flush() → None[source]#
pudl.extract.xbrl._suppress_arelle_message_spam()[source]#

Filter known Arelle console spam without suppressing normal logs.

class pudl.extract.xbrl.FercXbrlDatastore(datastore: pudl.workspace.datastore.Datastore)[source]#

Simple datastore wrapper for accessing ferc1 xbrl resources.

datastore[source]#
get_taxonomy(form: pudl.settings.FercForm) → io.BytesIO[source]#

Returns the path to the taxonomy entry point within the an archive.

get_filings(year: int, form: pudl.settings.FercForm) → io.BytesIO[source]#

Return the corresponding archive full of XBRL filings.

pudl.extract.xbrl.convert_form(ferc_to_sqlite: pudl.settings.FercToSqliteDataConfig, form: pudl.settings.FercForm, datastore: FercXbrlDatastore, pudl_paths: pudl.workspace.setup.PudlPaths, batch_size: int | None = None, workers: int | None = None, loglevel: str = 'INFO') → None[source]#

Clone a single FERC XBRL form to SQLite.

Parameters:

  • ferc_to_sqlite – Validated data configuration for converting FERC data to SQLite.

  • form – FERC form number.

  • datastore – Instance of a FERC XBRL datastore for retrieving data.

  • pudl_paths – PudlPaths resource.

  • batch_size – Number of XBRL filings to process in a single CPU process.

  • workers – Number of CPU processes to create for processing XBRL filings.

  • loglevel – Log level to pass to ferc_xbrl_extractor.