# Developer Guide

* [Development Setup](dev_setup.md)
  * [Fork and Clone the PUDL Repository](dev_setup.md#fork-and-clone-the-pudl-repository)
  * [Create the PUDL Dev Environment](dev_setup.md#create-the-pudl-dev-environment)
  * [Updating the PUDL Development Environment](dev_setup.md#updating-the-pudl-development-environment)
  * [Set Up Code Linting](dev_setup.md#set-up-code-linting)
  * [Creating a Workspace](dev_setup.md#creating-a-workspace)
* [Running the ETL Pipeline](run_the_etl.md)
  * [Alembic](run_the_etl.md#alembic)
  * [Dagster](run_the_etl.md#dagster)
  * [Running the ETL via the Dagster UI](run_the_etl.md#running-the-etl-via-the-dagster-ui)
  * [Running the ETL via CLI](run_the_etl.md#running-the-etl-via-cli)
  * [Additional Notes](run_the_etl.md#additional-notes)
* [Troubleshooting Dagster](troubleshooting_dagster.md)
  * [Reloading code locations](troubleshooting_dagster.md#reloading-code-locations)
  * [Viewing Logs](troubleshooting_dagster.md#viewing-logs)
  * [Keeping local Dagster instance up-to-date](troubleshooting_dagster.md#keeping-local-dagster-instance-up-to-date)
  * [Assets getting out of sync](troubleshooting_dagster.md#assets-getting-out-of-sync)
  * [Configuring resources](troubleshooting_dagster.md#configuring-resources)
* [Project Management](project_management.md)
  * [Issues and Project Tracking](project_management.md#issues-and-project-tracking)
  * [Our GitHub Workflow](project_management.md#our-github-workflow)
  * [Pull Requests](project_management.md#pull-requests)
  * [Releases](project_management.md#releases)
  * [User Support](project_management.md#user-support)
* [Testing PUDL](testing.md)
  * [Software Tests](testing.md#software-tests)
  * [Running the tests and other tasks with pixi](testing.md#running-the-tests-and-other-tasks-with-pixi)
  * [Selecting Input Data for Integration Tests](testing.md#selecting-input-data-for-integration-tests)
  * [Running pytest Directly](testing.md#running-pytest-directly)
* [Data validation quickstart](data_validation_quickstart.md)
  * [Setup](data_validation_quickstart.md#setup)
  * [Running the data validation tests](data_validation_quickstart.md#running-the-data-validation-tests)
  * [Updating table schemas](data_validation_quickstart.md#updating-table-schemas)
  * [Updating row counts](data_validation_quickstart.md#updating-row-counts)
* [Data validation reference](data_validation_reference.md)
  * [Overview](data_validation_reference.md#overview)
  * [Common workflows](data_validation_reference.md#common-workflows)
  * [PUDL Specific Design Choices](data_validation_reference.md#pudl-specific-design-choices)
  * [Branch builds](data_validation_reference.md#branch-builds)
  * [Running dbt directly](data_validation_reference.md#running-dbt-directly)
  * [Data validation in our integration tests](data_validation_reference.md#data-validation-in-our-integration-tests)
  * [Debugging data validation failures](data_validation_reference.md#debugging-data-validation-failures)
  * [Applying pre-defined validations to existing data](data_validation_reference.md#applying-pre-defined-validations-to-existing-data)
  * [Adding new tables](data_validation_reference.md#adding-new-tables)
  * [Defining new data validation tests](data_validation_reference.md#defining-new-data-validation-tests)
* [Metadata editing guide](metadata.md)
  * [Dataset-level metadata](metadata.md#dataset-level-metadata)
  * [Table-level metadata](metadata.md#table-level-metadata)
  * [Field metadata](metadata.md#field-metadata)
  * [Machine-readable units with pint](metadata.md#machine-readable-units-with-pint)
* [Building the Documentation](build_docs.md)
* [Working with the Datastore](datastore.md)
  * [Adding a new Dataset to the Datastore](datastore.md#adding-a-new-dataset-to-the-datastore)
* [Converting raw FERC data to SQLite](clone_ferc1.md)
  * [FERC SQLite provenance compatibility](clone_ferc1.md#ferc-sqlite-provenance-compatibility)
* [Existing Data Updates](existing_data_updates.md)
  * [1. Obtain Fresh Data](existing_data_updates.md#obtain-fresh-data)
  * [2. Map the Structure of the New Data](existing_data_updates.md#map-the-structure-of-the-new-data)
  * [3. Test Data Extraction](existing_data_updates.md#test-data-extraction)
  * [4. Update Table & Column Transformations](existing_data_updates.md#update-table-column-transformations)
  * [5. Update the PUDL DB Schema](existing_data_updates.md#update-the-pudl-db-schema)
  * [6. Connect Datasets](existing_data_updates.md#connect-datasets)
  * [7. Update the Output Routines](existing_data_updates.md#update-the-output-routines)
  * [8. Run the ETL](existing_data_updates.md#run-the-etl)
  * [9. Run and Update Data Validations](existing_data_updates.md#run-and-update-data-validations)
  * [10. Update the Documentation](existing_data_updates.md#update-the-documentation)
* [Run a Versioned Release](run_a_release.md)
  * [Here’s how to do it!](run_a_release.md#here-s-how-to-do-it)
* [PUDL ID Mapping](pudl_id_mapping.md)
  * [The Status of FERC 1 and EIA IDs](pudl_id_mapping.md#the-status-of-ferc-1-and-eia-ids)
  * [PUDL IDs](pudl_id_mapping.md#pudl-ids)
  * [FERC1 Utility IDs](pudl_id_mapping.md#ferc1-utility-ids)
  * [Checking for Unmapped Records](pudl_id_mapping.md#checking-for-unmapped-records)
  * [Assigning PUDL IDs to Unmapped Records](pudl_id_mapping.md#assigning-pudl-ids-to-unmapped-records)
  * [Testing Newly Mapped Records](pudl_id_mapping.md#testing-newly-mapped-records)
  * [Integrating Newly Mapped Records into PUDL](pudl_id_mapping.md#integrating-newly-mapped-records-into-pudl)
* [Naming Conventions](naming_conventions.md)
  * [Asset Naming Conventions](naming_conventions.md#asset-naming-conventions)
  * [Columns and Field Names](naming_conventions.md#columns-and-field-names)
  * [Naming Conventions in Code](naming_conventions.md#naming-conventions-in-code)
* [Best practices for working with LLMs](llm_best_practices.md)
* [Data and ETL Design Guidelines](data_guidelines.md)
  * [Input vs. Output Data](data_guidelines.md#input-vs-output-data)
  * [Minimize Data Alteration](data_guidelines.md#minimize-data-alteration)
  * [Make Tidy Data](data_guidelines.md#make-tidy-data)
  * [Use Consistent Units](data_guidelines.md#use-consistent-units)
  * [Partition Big Data](data_guidelines.md#partition-big-data)
  * [Complete, Continuous Time Series](data_guidelines.md#complete-continuous-time-series)
* [Nightly Data Builds](nightly_data_builds.md)
  * [Breaking the Builds](nightly_data_builds.md#breaking-the-builds)
  * [Build Action](nightly_data_builds.md#build-action)
  * [Deployment Action](nightly_data_builds.md#deployment-action)
  * [Google Compute Engine](nightly_data_builds.md#google-compute-engine)
  * [Docker](nightly_data_builds.md#docker)
  * [How to access the nightly build outputs from AWS](nightly_data_builds.md#how-to-access-the-nightly-build-outputs-from-aws)
  * [How to access the nightly build outputs and logs (for the Catalyst team only)](nightly_data_builds.md#how-to-access-the-nightly-build-outputs-and-logs-for-the-catalyst-team-only)
* [FERC EQR Data Builds](ferceqr_data_builds.md)
  * [Notable differences from nightly builds](ferceqr_data_builds.md#notable-differences-from-nightly-builds)
* [Infrastructure as Code](infrastructure_as_code.md)
  * [Overview](infrastructure_as_code.md#overview)
  * [Setup](infrastructure_as_code.md#setup)
  * [Development](infrastructure_as_code.md#development)
* [Managing External Contributions](managing_external_contributions.md)
  * [Overview](managing_external_contributions.md#overview)
  * [Creating ‘good first issues’](managing_external_contributions.md#creating-good-first-issues)
  * [Guidelines for contributor review](managing_external_contributions.md#guidelines-for-contributor-review)
  * [Common contributor management decisions](managing_external_contributions.md#common-contributor-management-decisions)
  * [Github incantations](managing_external_contributions.md#github-incantations)