pudl.scripts.batch_config#

Generate a Google Batch Job configuration file.

This runs on bare GitHub Actions runners, without the full pixi/pudl environment – the inline script metadata above lets uv run install just the handful of dependencies this script actually needs (stdlib plus click) into an ephemeral environment, rather than requiring a full pudl install first.

The --container-* flags are named after their equivalents in gcloud compute instances update-container.

Attributes#

`logger`
`DEFAULT_MACHINE_TYPE`
`DEFAULT_DISK_GB`
`DEFAULT_DISK_TYPE`

Functions#

`_parse_container_env`(→ dict[str, str])	Parse --container-env KEY=VALUE pairs into a dict.
`_lookup_machine_spec`(→ tuple[int, int])	Return `(cpuMilli, memoryMib)` for a real GCE machine type, via `gcloud`.
`to_config`(→ dict[str, Any])	Munge arguments into a configuration dictionary.
`main`(→ None)	Generate a Batch configuration file.

Module Contents#

pudl.scripts.batch_config.logger[source]#

pudl.scripts.batch_config.DEFAULT_MACHINE_TYPE = 'c4d-standard-8'[source]#

pudl.scripts.batch_config.DEFAULT_DISK_GB = 250[source]#

pudl.scripts.batch_config.DEFAULT_DISK_TYPE = 'hyperdisk-balanced'[source]#

pudl.scripts.batch_config._parse_container_env(container_env: tuple[str, ...]) → dict[str, str][source]#

Parse –container-env KEY=VALUE pairs into a dict.

Raises if the same key is given more than once.

pudl.scripts.batch_config._lookup_machine_spec(machine_type: str) → tuple[int, int][source]#

Return (cpuMilli, memoryMib) for a real GCE machine type, via gcloud.

Batch’s computeResource.cpuMilli/memoryMib default to 2000/2000 (2 vCPU, 2 GB) if left unset – regardless of the machine type pinned in allocationPolicy. This looks up the real values and fills them in so the job doesn’t lie about the resources it has available.

pudl.scripts.batch_config.to_config(*, container_image: str, container_env: tuple[str, ...], container_command: str, container_arg: tuple[str, ...], machine_type: str, cpu_milli: int, memory_mib: int, disk_gb: int, disk_type: str, batch_job_id: str, pipeline: str) → dict[str, Any][source]#: Munge arguments into a configuration dictionary.

pudl.scripts.batch_config.main(container_image: str, container_command: str, container_env: tuple[str, ...], container_arg: tuple[str, ...], machine_type: str, disk_gb: int, disk_type: str, batch_job_id: str, pipeline: str, output: pathlib.Path) → None[source]#: Generate a Batch configuration file.