pudl.dagster.resources#

Dagster resources for PUDL.

This module defines the configurable resources that PUDL assets depend on at runtime, such as data configuration, datastore access, and other run-scoped helpers, along with the default resource mapping used by the assembled code location. Add dagster.ConfigurableResource classes and configured singleton instances here when they provide external services or shared runtime context to assets and jobs. Keep asset logic out of this module; it should focus on dependency injection and default resource wiring.

For the underlying Dagster concept, see https://docs.dagster.io/guides/build/external-resources

Attributes#

`global_data_config_resource`
`pudl_paths_resource`
`zenodo_doi_settings_resource`
`datastore_resource`
`ferc_xbrl_runtime_settings`
`ferceqr_archive`
`ferceqr_deployment_targets`
`zulip_notification_resource`
`default_resources`

Classes#

`PudlPathsResource`	Load the input/output paths used by Dagster-managed PUDL runs.
`FercXbrlRuntimeSettings`	Encodes runtime settings for the ferc_to_sqlite graphs.
`GlobalDataConfigResource`	Load validated PUDL data configuration from a shared ETL YAML file.
`ZenodoDoiSettingsResource`	Load the canonical Zenodo DOI settings for Dagster-managed runs.
`DatastoreResource`	Dagster resource to interact with Zenodo archives.
`FercEqrArchiveResource`	Configure which archived FERC EQR filings are available for extraction.
`FercEqrDeploymentTargetConfig`	A single deployment destination for FERC EQR outputs.
`FercEqrDeploymentResource`	One or more deployment destinations for FERC EQR outputs.
`ZulipNotificationResource`	Send notifications to Zulip streams via the Zulip API.

Module Contents#

class pudl.dagster.resources.PudlPathsResource[source]#

Bases: dagster.ConfigurableResource

Load the input/output paths used by Dagster-managed PUDL runs.

Explicit Dagster resource config takes precedence. Any unset field falls back to the current process environment so dg runs, local .env files, test fixtures, and container-provided environment variables all share a single typed entry point.

pudl_input: str[source]#

pudl_output: str[source]#

create_resource(context) → pudl.workspace.setup.PudlPaths[source]#: Create validated runtime path settings for the current Dagster run.

class pudl.dagster.resources.FercXbrlRuntimeSettings[source]#

Bases: dagster.ConfigurableResource

Encodes runtime settings for the ferc_to_sqlite graphs.

xbrl_num_workers: None | int = None[source]#

xbrl_batch_size: int = 50[source]#

xbrl_loglevel: str = 'INFO'[source]#

class pudl.dagster.resources.GlobalDataConfigResource[source]#

Bases: dagster.ConfigurableResource

Load validated PUDL data configuration from a shared ETL YAML file.

global_data_config_path: str = ''[source]#

create_resource(context) → pudl.settings.GlobalDataConfig[source]#: Create runtime data configuration from the configured YAML file.

class pudl.dagster.resources.ZenodoDoiSettingsResource[source]#

Bases: dagster.ConfigurableResource

Load the canonical Zenodo DOI settings for Dagster-managed runs.

Two configuration paths are supported:

Inline defaults (zenodo_dois_path=None): uses the canonical Zenodo DOIs that are hardcoded as defaults in ZenodoDoiSettings. This is the normal production path — no extra config file is needed.
Path override (zenodo_dois_path="..."): loads DOIs from an external YAML file, allowing deployments or tests to substitute different DOIs without modifying the source code.

zenodo_dois_path: str | None = None[source]#

create_resource(context) → pudl.workspace.datastore.ZenodoDoiSettings[source]#: Create runtime DOI settings, optionally from an override YAML file.

class pudl.dagster.resources.DatastoreResource[source]#

Bases: dagster.ConfigurableResource

Dagster resource to interact with Zenodo archives.

zenodo_dois: dagster.ResourceDependency[ZenodoDoiSettingsResource][source]#

pudl_paths: dagster.ResourceDependency[PudlPathsResource][source]#

cloud_cache_path: str = 's3://pudl.catalyst.coop/zenodo'[source]#

use_local_cache: bool = True[source]#

create_resource(context) → pudl.workspace.datastore.Datastore[source]#: Create a configured datastore runtime object.

class pudl.dagster.resources.FercEqrArchiveResource[source]#

Bases: dagster.ConfigurableResource

Configure which archived FERC EQR filings are available for extraction.

The default value of path points to the published archive of FERC EQR filings on GCS which is what we use in production. For testing or development, this can be overridden to point to a local path with a subset of the archive.

path: str[source]#

property upath: upath.UPath[source]#: Return UPath pointing to archive base path.

class pudl.dagster.resources.FercEqrDeploymentTargetConfig(**config_dict)[source]#

Bases: dagster.Config

A single deployment destination for FERC EQR outputs.

path is a UPath-compatible string: an absolute local directory path, file:// URI, gs:// URI, or s3:// URI. storage_options is unpacked as **kwargs when constructing the UPath, allowing per-target fsspec settings such as requester_pays=True for requester-pays GCS buckets.

path: str[source]#

storage_options: dict[str, Any][source]#

append_build_id: bool = False[source]#

classmethod validate_path(value: str) → str[source]#: Validate deployment targets as remote URLs or local directories.

class pudl.dagster.resources.FercEqrDeploymentResource[source]#

Bases: dagster.ConfigurableResource

One or more deployment destinations for FERC EQR outputs.

Deployment targets can be provided directly as structured config or loaded from a YAML file. Direct deployment_targets take precedence. When neither explicit targets nor a deployment config path are provided, deployment is skipped.

deployment_targets: list[FercEqrDeploymentTargetConfig] = [][source]#

deployment_config_path: str | None = None[source]#

classmethod from_yaml(deployment_config_path: str | pathlib.Path) → FercEqrDeploymentResource[source]#: Create a FERC EQR deployment resource from a YAML config file.

configured_targets() → list[FercEqrDeploymentTargetConfig][source]#: Return deployment-target config with explicit overrides taking precedence.

resolved_targets() → list[upath.UPath][source]#

Return the list of UPath deployment destinations.

Each configured target is converted to a UPath using its provided storage_options.

class pudl.dagster.resources.ZulipNotificationResource[source]#

Bases: dagster.ConfigurableResource

Send notifications to Zulip streams via the Zulip API.

base_url: str = 'https://catalyst-cooperative.zulipchat.com'[source]#

bot_email: str = 'build-status-bot@catalyst-cooperative.zulipchat.com'[source]#

api_key: str[source]#

timeout_seconds: int = 30[source]#

send_stream_message(*, stream: str, topic: str, content: str, file_path: str | pathlib.Path | None = None) → dict[source]#

Send a message to a Zulip stream topic and return the API response.

Optionally upload a file and attach a download link to the message content.

Sends are best-effort: all failures are logged as warnings and returned in the result dict so callers can inspect them, but no exception is raised. This ensures a notification hiccup never crashes an asset.

pudl.dagster.resources.global_data_config_resource[source]#

pudl.dagster.resources.pudl_paths_resource[source]#

pudl.dagster.resources.zenodo_doi_settings_resource[source]#

pudl.dagster.resources.datastore_resource[source]#

pudl.dagster.resources.ferc_xbrl_runtime_settings[source]#

pudl.dagster.resources.ferceqr_archive[source]#

pudl.dagster.resources.ferceqr_deployment_targets[source]#

pudl.dagster.resources.zulip_notification_resource[source]#

pudl.dagster.resources.default_resources: dict[str, Any][source]#