Skip to content

pudl_diff.outputs

Writing the rows that differ between two tables to Parquet files.

ParquetOutput dataclass

Metadata about a single Parquet side-output file written to disk.

Source code in src/pudl_diff/outputs.py 
19
20
21
22
23
24
25
26
27
28
29
30
31
32
@dataclass(frozen=True)
class ParquetOutput:
    """Metadata about a single Parquet side-output file written to disk."""

    path: Path
    bytes: int
    hash: str
    """`"sha256:<hexdigest>"`, matching the convention PUDL's enriched
    `datapackage.json` uses for its own resource files."""
    total_row_count: int
    """Total number of differing rows on this side, before any capping by
    `max_rows_per_output_parquet`."""
    rows_written: int
    """Number of rows actually written to `path`."""

hash instance-attribute

"sha256:<hexdigest>", matching the convention PUDL's enriched datapackage.json uses for its own resource files.

rows_written instance-attribute

Number of rows actually written to path.

total_row_count instance-attribute

Total number of differing rows on this side, before any capping by max_rows_per_output_parquet.

RowDiffParquetOutputs dataclass

The left-only and right-only Parquet files written for one table's row diff.

Source code in src/pudl_diff/outputs.py 
35
36
37
38
39
40
@dataclass(frozen=True)
class RowDiffParquetOutputs:
    """The left-only and right-only Parquet files written for one table's row diff."""

    left: ParquetOutput
    right: ParquetOutput

write_row_diff_parquet(row_diff, output_path, table_name, *, right_table_name=None, max_rows_per_output_parquet=None)

Write a table's differing rows to left-only/right-only Parquet files.

Writes <table_name>_left_only.parquet and <right_table_name>_right_only.parquet under output_path (created if it doesn't exist), each matching the schema of the corresponding source table. For a table with a primary key, each file holds that side's rows from the symmetric difference of primary keys, plus that side's values for rows whose non-PK data differs. For a table without one, each file holds that side's rows from the symmetric difference of whole rows.

Parameters:

Name Type Description Default
row_diff RowSetDiff | KeyedRowDiff | None

The row-level comparison result to write out, e.g. from row_diff. If None - row-level comparison was skipped for this table - nothing is written and this function returns None.

 required
output_path str | PathLike[str]

Directory to write the two Parquet files into.

 required
table_name str

Used as the filename prefix for the left output file, and for the right one too unless right_table_name is given.

 required
right_table_name str | None

Used as the filename prefix for the right output file, if it differs from table_name - e.g. writing out a diff between a core_ table and the out_ table built from it, as passed to compare_table(). Defaults to table_name.

 None
max_rows_per_output_parquet int | None

If given, caps the number of rows written to each file. ParquetOutput.total_row_count still reflects the true (uncapped) row count.

 None

Returns:

Type Description
RowDiffParquetOutputs | None

The two files' paths and metadata, or None if row_diff is
RowDiffParquetOutputs | None

None.
Source code in src/pudl_diff/outputs.py 
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
def write_row_diff_parquet(
    row_diff: RowSetDiff | KeyedRowDiff | None,
    output_path: str | os.PathLike[str],
    table_name: str,
    *,
    right_table_name: str | None = None,
    max_rows_per_output_parquet: int | None = None,
) -> RowDiffParquetOutputs | None:
    """Write a table's differing rows to left-only/right-only Parquet files.

    Writes `<table_name>_left_only.parquet` and
    `<right_table_name>_right_only.parquet` under `output_path` (created
    if it doesn't exist), each matching the schema of the corresponding
    source table. For a table with a primary key, each file holds that
    side's rows from the symmetric difference of primary keys, plus that
    side's values for rows whose non-PK data differs. For a table without
    one, each file holds that side's rows from the symmetric difference of
    whole rows.

    Args:
        row_diff: The row-level comparison result to write out, e.g. from
            `row_diff`. If `None` - row-level
            comparison was skipped for this table - nothing is written and
            this function returns `None`.
        output_path: Directory to write the two Parquet files into.
        table_name: Used as the filename prefix for the left output file,
            and for the right one too unless `right_table_name` is given.
        right_table_name: Used as the filename prefix for the right output
            file, if it differs from `table_name` - e.g. writing out a
            diff between a `core_` table and the `out_` table built from it,
            as passed to `compare_table()`. Defaults to `table_name`.
        max_rows_per_output_parquet: If given, caps the number of rows
            written to each file. `ParquetOutput.total_row_count`
            still reflects the true (uncapped) row count.

    Returns:
        The two files' paths and metadata, or `None` if `row_diff` is
        `None`.
    """
    if row_diff is None:
        return None
    right_table_name = right_table_name or table_name
    output_path = Path(output_path)
    output_path.mkdir(parents=True, exist_ok=True)
    (left_lf, left_count), (right_lf, right_count) = row_diff_left_right_frames(
        row_diff
    )
    return RowDiffParquetOutputs(
        left=_write_parquet_output(
            left_lf,
            left_count,
            output_path / f"{table_name}_left_only.parquet",
            max_rows_per_output_parquet,
        ),
        right=_write_parquet_output(
            right_lf,
            right_count,
            output_path / f"{right_table_name}_right_only.parquet",
            max_rows_per_output_parquet,
        ),
    )