pudl.ferc_to_sqlite
===================

.. py:module:: pudl.ferc_to_sqlite

.. autoapi-nested-parse::

   Dagster definitions for the FERC to SQLite process.



Submodules
----------

.. toctree::
   :maxdepth: 1

   /autoapi/pudl/ferc_to_sqlite/cli/index


Attributes
----------

.. autoapisummary::

   pudl.ferc_to_sqlite.ALL_DBF_EXTRACTORS
   pudl.ferc_to_sqlite.logger
   pudl.ferc_to_sqlite.default_resources_defs
   pudl.ferc_to_sqlite.ferc_to_sqlite_full_settings
   pudl.ferc_to_sqlite.ferc_to_sqlite_full
   pudl.ferc_to_sqlite.ferc_to_sqlite_fast_settings
   pudl.ferc_to_sqlite.ferc_to_sqlite_fast
   pudl.ferc_to_sqlite.defs


Classes
-------

.. autoapisummary::

   pudl.ferc_to_sqlite.RuntimeSettings
   pudl.ferc_to_sqlite.EtlSettings
   pudl.ferc_to_sqlite.XbrlFormNumber


Functions
---------

.. autoapisummary::

   pudl.ferc_to_sqlite.xbrl2sqlite_op_factory
   pudl.ferc_to_sqlite.datastore
   pudl.ferc_to_sqlite.ferc_to_sqlite_settings
   pudl.ferc_to_sqlite.ferc_to_sqlite


Package Contents
----------------

.. py:data:: ALL_DBF_EXTRACTORS

.. py:function:: xbrl2sqlite_op_factory(form: pudl.settings.XbrlFormNumber) -> collections.abc.Callable

   Generates xbrl2sqlite op for a given FERC form.


.. py:class:: RuntimeSettings

   Bases: :py:obj:`dagster.ConfigurableResource`


   Encodes runtime settings for the ferc_to_sqlite graphs.


   .. py:attribute:: xbrl_num_workers
      :type:  None | int
      :value: None



   .. py:attribute:: xbrl_batch_size
      :type:  int
      :value: 50



.. py:function:: datastore(init_context) -> pudl.workspace.datastore.Datastore

   Dagster resource to interact with Zenodo archives.


.. py:function:: ferc_to_sqlite_settings(init_context) -> pudl.settings.FercToSqliteSettings

   Dagster resource for parameterizing the ``ferc_to_sqlite`` graph.

   This resource allows us to specify the years we want to process for each datasource
   in the Dagit UI.


.. py:class:: EtlSettings(_case_sensitive: bool | None = None, _nested_model_default_partial_update: bool | None = None, _env_prefix: str | None = None, _env_prefix_target: pydantic_settings.sources.EnvPrefixTarget | None = None, _env_file: pydantic_settings.sources.DotenvType | None = ENV_FILE_SENTINEL, _env_file_encoding: str | None = None, _env_ignore_empty: bool | None = None, _env_nested_delimiter: str | None = None, _env_nested_max_split: int | None = None, _env_parse_none_str: str | None = None, _env_parse_enums: bool | None = None, _cli_prog_name: str | None = None, _cli_parse_args: bool | list[str] | tuple[str, Ellipsis] | None = None, _cli_settings_source: pydantic_settings.sources.CliSettingsSource[Any] | None = None, _cli_parse_none_str: str | None = None, _cli_hide_none_type: bool | None = None, _cli_avoid_json: bool | None = None, _cli_enforce_required: bool | None = None, _cli_use_class_docs_for_groups: bool | None = None, _cli_exit_on_error: bool | None = None, _cli_prefix: str | None = None, _cli_flag_prefix_char: str | None = None, _cli_implicit_flags: bool | Literal['dual', 'toggle'] | None = None, _cli_ignore_unknown_args: bool | None = None, _cli_kebab_case: bool | Literal['all', 'no_enums'] | None = None, _cli_shortcuts: collections.abc.Mapping[str, str | list[str]] | None = None, _secrets_dir: pydantic_settings.sources.PathType | None = None, _build_sources: tuple[tuple[pydantic_settings.sources.PydanticBaseSettingsSource, Ellipsis], dict[str, Any]] | None = None, **values: Any)

   Bases: :py:obj:`pydantic_settings.BaseSettings`


   Main settings validation class.


   .. py:attribute:: ferc_to_sqlite_settings
      :type:  FercToSqliteSettings | None
      :value: None



   .. py:attribute:: datasets
      :type:  DatasetsSettings | None
      :value: None



   .. py:attribute:: name
      :type:  str | None
      :value: None



   .. py:attribute:: title
      :type:  str | None
      :value: None



   .. py:attribute:: description
      :type:  str | None
      :value: None



   .. py:attribute:: version
      :type:  str | None
      :value: None



   .. py:attribute:: publish_destinations
      :type:  list[str]
      :value: []


      This is list of fsspec compatible paths to publish the output datasets to.


   .. py:method:: from_yaml(path: str) -> EtlSettings
      :classmethod:


      Create an EtlSettings instance from a yaml_file path.

      :param path: path to a yaml file; this could be remote.

      :returns: An ETL settings object.



   .. py:method:: validate_xbrl_years()

      Ensure the XBRL years in DatasetsSettings align with FercToSqliteSettings.

      For each of the FERC forms that we are processing in PUDL, check to ensure
      that the years we are trying to process in the PUDL ETL are included in the
      XBRL to SQLite settings.



.. py:class:: XbrlFormNumber(*args, **kwds)

   Bases: :py:obj:`enum.Enum`


   Contains full list of supported FERC XBRL forms.


   .. py:attribute:: FORM1
      :value: 1



   .. py:attribute:: FORM2
      :value: 2



   .. py:attribute:: FORM6
      :value: 6



   .. py:attribute:: FORM60
      :value: 60



   .. py:attribute:: FORM714
      :value: 714



.. py:data:: logger

.. py:function:: ferc_to_sqlite()

   Clone the FERC FoxPro databases and XBRL filings into SQLite.


.. py:data:: default_resources_defs

.. py:data:: ferc_to_sqlite_full_settings

.. py:data:: ferc_to_sqlite_full

.. py:data:: ferc_to_sqlite_fast_settings

.. py:data:: ferc_to_sqlite_fast

.. py:data:: defs
   :type:  dagster.Definitions

   A collection of dagster assets, resources, IO managers, and jobs for the FERC to
   SQLite ETL.