pudl.dagster.resources#

Dagster resources for PUDL.

This module defines the configurable resources that PUDL assets depend on at runtime, such as data configuration, datastore access, and other run-scoped helpers, along with the default resource mapping used by the assembled code location. Add dagster.ConfigurableResource classes and configured singleton instances here when they provide external services or shared runtime context to assets and jobs. Keep asset logic out of this module; it should focus on dependency injection and default resource wiring.

For the underlying Dagster concept, see https://docs.dagster.io/guides/build/external-resources

Attributes#

global_data_config_resource

pudl_paths_resource

zenodo_doi_settings_resource

datastore_resource

ferc_xbrl_runtime_settings

ferceqr_data_config

zulip_notification_resource

default_resources

Classes#

PudlPathsResource

Load the input/output paths used by Dagster-managed PUDL runs.

FercXbrlRuntimeSettings

Encodes runtime settings for the ferc_to_sqlite graphs.

GlobalDataConfigResource

Load validated PUDL data configuration from a shared ETL YAML file.

ZenodoDoiSettingsResource

Load the canonical Zenodo DOI settings for Dagster-managed runs.

DatastoreResource

Dagster resource to interact with Zenodo archives.

FercEqrDataConfig

Configure which archived FERC EQR filings are available for extraction.

ZulipNotificationResource

Send notifications to Zulip streams via the Zulip API.

Module Contents#

class pudl.dagster.resources.PudlPathsResource[source]#

Bases: dagster.ConfigurableResource

Load the input/output paths used by Dagster-managed PUDL runs.

Explicit Dagster resource config takes precedence. Any unset field falls back to the current process environment so dg runs, local .env files, test fixtures, and container-provided environment variables all share a single typed entry point.

pudl_input: str[source]#
pudl_output: str[source]#
create_resource(context) → pudl.workspace.setup.PudlPaths[source]#

Create validated runtime path settings for the current Dagster run.

class pudl.dagster.resources.FercXbrlRuntimeSettings[source]#

Bases: dagster.ConfigurableResource

Encodes runtime settings for the ferc_to_sqlite graphs.

xbrl_num_workers: None | int = None[source]#
xbrl_batch_size: int = 50[source]#
xbrl_loglevel: str = 'INFO'[source]#
class pudl.dagster.resources.GlobalDataConfigResource[source]#

Bases: dagster.ConfigurableResource

Load validated PUDL data configuration from a shared ETL YAML file.

global_data_config_path: str = ''[source]#
create_resource(context) → pudl.settings.GlobalDataConfig[source]#

Create runtime data configuration from the configured YAML file.

class pudl.dagster.resources.ZenodoDoiSettingsResource[source]#

Bases: dagster.ConfigurableResource

Load the canonical Zenodo DOI settings for Dagster-managed runs.

Two configuration paths are supported:

  • Inline defaults (zenodo_dois_path=None): uses the canonical Zenodo DOIs that are hardcoded as defaults in ZenodoDoiSettings. This is the normal production path — no extra config file is needed.

  • Path override (zenodo_dois_path="..."): loads DOIs from an external YAML file, allowing deployments or tests to substitute different DOIs without modifying the source code.

zenodo_dois_path: str | None = None[source]#
create_resource(context) → pudl.workspace.datastore.ZenodoDoiSettings[source]#

Create runtime DOI settings, optionally from an override YAML file.

class pudl.dagster.resources.DatastoreResource[source]#

Bases: dagster.ConfigurableResource

Dagster resource to interact with Zenodo archives.

zenodo_dois: dagster.ResourceDependency[ZenodoDoiSettingsResource][source]#
pudl_paths: dagster.ResourceDependency[PudlPathsResource][source]#
cloud_cache_path: str = 's3://pudl.catalyst.coop/zenodo'[source]#
use_local_cache: bool = True[source]#
create_resource(context) → pudl.workspace.datastore.Datastore[source]#

Create a configured datastore runtime object.

class pudl.dagster.resources.FercEqrDataConfig[source]#

Bases: dagster.ConfigurableResource

Configure which archived FERC EQR filings are available for extraction.

The default value of ferceqr_archive_uri points to the published archive of FERC EQR filings on GCS which is what we use in production. For testing or development, this can be overridden to point to a local path with a subset of the archive.

ferceqr_archive_uri: str[source]#
property ferceqr_archive_path: upath.UPath[source]#

Return UPath pointing to archive base path.

class pudl.dagster.resources.ZulipNotificationResource[source]#

Bases: dagster.ConfigurableResource

Send notifications to Zulip streams via the Zulip API.

base_url: str = 'https://catalyst-cooperative.zulipchat.com'[source]#
bot_email: str = 'build-status-bot@catalyst-cooperative.zulipchat.com'[source]#
api_key: str[source]#
timeout_seconds: int = 30[source]#
send_stream_message(*, stream: str, topic: str, content: str, file_path: str | pathlib.Path | None = None) → dict[source]#

Send a message to a Zulip stream topic and return the API response.

Optionally upload a file and attach a download link to the message content.

Sends are best-effort: all failures are logged as warnings and returned in the result dict so callers can inspect them, but no exception is raised. This ensures a notification hiccup never crashes an asset.

pudl.dagster.resources.global_data_config_resource[source]#
pudl.dagster.resources.pudl_paths_resource[source]#
pudl.dagster.resources.zenodo_doi_settings_resource[source]#
pudl.dagster.resources.datastore_resource[source]#
pudl.dagster.resources.ferc_xbrl_runtime_settings[source]#
pudl.dagster.resources.ferceqr_data_config[source]#
pudl.dagster.resources.zulip_notification_resource[source]#
pudl.dagster.resources.default_resources: dict[str, Any][source]#