Skip to content

pudl_diff.schema

Comparing the schemas of two tables.

SchemaDiff dataclass

The difference between two table schemas.

Column order is not considered significant, per PUDL Diff's definition of "functionally identical" tables.

Source code in src/pudl_diff/schema.py 
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
@dataclass(frozen=True)
class SchemaDiff:
    """The difference between two table schemas.

    Column order is not considered significant, per PUDL Diff's definition of
    "functionally identical" tables.
    """

    columns_only_in_left: list[str]
    columns_only_in_right: list[str]
    dtype_changes: dict[str, tuple[pl.DataType, pl.DataType]]
    """Maps column name to a `(left_dtype, right_dtype)` pair, for columns
    present in both schemas but with differing dtypes."""
    left_column_count: int
    """Total number of columns in the left schema."""
    right_column_count: int
    """Total number of columns in the right schema."""

    @property
    def is_identical(self) -> bool:
        """Whether the two schemas have the same columns and dtypes."""
        return not (
            self.columns_only_in_left
            or self.columns_only_in_right
            or self.dtype_changes
        )

dtype_changes instance-attribute

Maps column name to a (left_dtype, right_dtype) pair, for columns present in both schemas but with differing dtypes.

is_identical property

Whether the two schemas have the same columns and dtypes.

left_column_count instance-attribute

Total number of columns in the left schema.

right_column_count instance-attribute

Total number of columns in the right schema.

compare_schemas(left, right)

Compare two table schemas.

Parameters:

Name Type Description Default
left Schema

Schema of the "left" table, e.g. from PudlDiffDataset.scan_table(table_name).collect_schema().

 required
right Schema

Schema of the "right" table, to compare against left.

 required
Source code in src/pudl_diff/schema.py 
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
def compare_schemas(left: pl.Schema, right: pl.Schema) -> SchemaDiff:
    """Compare two table schemas.

    Args:
        left: Schema of the "left" table, e.g. from
            `PudlDiffDataset.scan_table(table_name).collect_schema()`.
        right: Schema of the "right" table, to compare against `left`.
    """
    left_columns = set(left)
    right_columns = set(right)
    shared_columns = left_columns & right_columns
    dtype_changes = {
        name: (left[name], right[name])
        for name in shared_columns
        if left[name] != right[name]
    }
    return SchemaDiff(
        columns_only_in_left=sorted(left_columns - right_columns),
        columns_only_in_right=sorted(right_columns - left_columns),
        dtype_changes=dtype_changes,
        left_column_count=len(left),
        right_column_count=len(right),
    )