# ferc_xbrl_extractor.datapackage

Define structures for creating a datapackage descriptor.

## Attributes

| [`logger`](#ferc_xbrl_extractor.datapackage.logger)                                 |                                                                                        |
|-------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------|
| [`ENTITY_ID`](#ferc_xbrl_extractor.datapackage.ENTITY_ID)                           | Field representing an entity ID (Present in all tables).                               |
| [`FILING_NAME`](#ferc_xbrl_extractor.datapackage.FILING_NAME)                       | Field representing the filing name (Present in all tables).                            |
| [`PUBLICATION_TIME`](#ferc_xbrl_extractor.datapackage.PUBLICATION_TIME)             | Field representing the publication time (injected into all tables).                    |
| [`START_DATE`](#ferc_xbrl_extractor.datapackage.START_DATE)                         | Field representing start date (Present in all duration tables).                        |
| [`END_DATE`](#ferc_xbrl_extractor.datapackage.END_DATE)                             | Field representing end date (Present in all duration tables).                          |
| [`INSTANT_DATE`](#ferc_xbrl_extractor.datapackage.INSTANT_DATE)                     | Field representing an instant date (Present in all instant tables).                    |
| [`DURATION_COLUMNS`](#ferc_xbrl_extractor.datapackage.DURATION_COLUMNS)             | Fields common to all duration tables.                                                  |
| [`INSTANT_COLUMNS`](#ferc_xbrl_extractor.datapackage.INSTANT_COLUMNS)               | Fields common to all instant tables.                                                   |
| [`FIELD_TO_PANDAS`](#ferc_xbrl_extractor.datapackage.FIELD_TO_PANDAS)               | Pandas data type by schema field type (Data Package field.type).                       |
| [`CONVERT_DTYPES`](#ferc_xbrl_extractor.datapackage.CONVERT_DTYPES)                 | Map callables to schema field type to convert parsed values (Data Package field.type). |
| [`TABLE_NAME_PATTERN`](#ferc_xbrl_extractor.datapackage.TABLE_NAME_PATTERN)         | Simple regex pattern used to clean up table names.                                     |
| [`UPPERCASE_WORD_PATTERN`](#ferc_xbrl_extractor.datapackage.UPPERCASE_WORD_PATTERN) | Regex pattern to find fully uppercase words.                                           |

## Classes

| [`Field`](#ferc_xbrl_extractor.datapackage.Field)             | A generic field descriptor, as per Frictionless Data specs.       |
|---------------------------------------------------------------|-------------------------------------------------------------------|
| [`Schema`](#ferc_xbrl_extractor.datapackage.Schema)           | A generic table schema, as per Frictionless Data specs.           |
| [`Dialect`](#ferc_xbrl_extractor.datapackage.Dialect)         | Dialect used for frictionless SQL resources.                      |
| [`Resource`](#ferc_xbrl_extractor.datapackage.Resource)       | A generic tabular data resource, as per Frictionless Data specs.  |
| [`FactTable`](#ferc_xbrl_extractor.datapackage.FactTable)     | Class to handle constructing a dataframe from an XBRL fact table. |
| [`Datapackage`](#ferc_xbrl_extractor.datapackage.Datapackage) | A generic Data Package, as per Frictionless Data specs.           |

## Functions

| [`_get_fields_from_concepts`](#ferc_xbrl_extractor.datapackage._get_fields_from_concepts)(→ tuple[list[Field], ...)   | Traverse concept tree to get columns and axes that will be used in output table.   |
|-----------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------|
| [`_lowercase_words`](#ferc_xbrl_extractor.datapackage._lowercase_words)(→ str)                                        | Convert fully uppercase words so only first letter is uppercase.                   |
| [`clean_table_names`](#ferc_xbrl_extractor.datapackage.clean_table_names)(→ str | None)                               | Function to clean table names.                                                     |
| [`fuzzy_dedup`](#ferc_xbrl_extractor.datapackage.fuzzy_dedup)(→ pandas.DataFrame)                                     | Deduplicate a 1-column dataframe with numbers that are close in value.             |

## Module Contents

### ferc_xbrl_extractor.datapackage.logger *: logging.Logger*

### *class* ferc_xbrl_extractor.datapackage.Field(/, \*\*data: Any)

Bases: `pydantic.BaseModel`

A generic field descriptor, as per Frictionless Data specs.

See [https://specs.frictionlessdata.io/table-schema/#field-descriptors](https://specs.frictionlessdata.io/table-schema/#field-descriptors).

#### name *: str*

#### title *: str*

#### type_ *: str* *= None*

#### format_ *: str* *= None*

#### description *: str*

#### *classmethod* from_concept(concept: [ferc_xbrl_extractor.taxonomy.Concept](../taxonomy/index.md#ferc_xbrl_extractor.taxonomy.Concept)) → [Field](#ferc_xbrl_extractor.datapackage.Field)

Construct a Field from an XBRL Concept.

* **Parameters:**
  **concept** – XBRL Concept used to create a Field.

#### \_\_hash_\_()

Implement hash method to allow creating sets of Fields.

### ferc_xbrl_extractor.datapackage.ENTITY_ID *: [Field](#ferc_xbrl_extractor.datapackage.Field)*

Field representing an entity ID (Present in all tables).

### ferc_xbrl_extractor.datapackage.FILING_NAME *: [Field](#ferc_xbrl_extractor.datapackage.Field)*

Field representing the filing name (Present in all tables).

### ferc_xbrl_extractor.datapackage.PUBLICATION_TIME *: [Field](#ferc_xbrl_extractor.datapackage.Field)*

Field representing the publication time (injected into all tables).

### ferc_xbrl_extractor.datapackage.START_DATE *: [Field](#ferc_xbrl_extractor.datapackage.Field)*

Field representing start date (Present in all duration tables).

### ferc_xbrl_extractor.datapackage.END_DATE *: [Field](#ferc_xbrl_extractor.datapackage.Field)*

Field representing end date (Present in all duration tables).

### ferc_xbrl_extractor.datapackage.INSTANT_DATE *: [Field](#ferc_xbrl_extractor.datapackage.Field)*

Field representing an instant date (Present in all instant tables).

### ferc_xbrl_extractor.datapackage.DURATION_COLUMNS

Fields common to all duration tables.

### ferc_xbrl_extractor.datapackage.INSTANT_COLUMNS

Fields common to all instant tables.

### ferc_xbrl_extractor.datapackage.FIELD_TO_PANDAS *: dict[str, str]*

Pandas data type by schema field type (Data Package field.type).

### ferc_xbrl_extractor.datapackage.CONVERT_DTYPES *: dict[str, collections.abc.Callable]*

Map callables to schema field type to convert parsed values (Data Package field.type).

### ferc_xbrl_extractor.datapackage.TABLE_NAME_PATTERN *: re.Pattern[str]*

Simple regex pattern used to clean up table names.

### ferc_xbrl_extractor.datapackage.UPPERCASE_WORD_PATTERN *: re.Pattern[str]*

Regex pattern to find fully uppercase words.

There are several tables in the FERC taxonomy that contain completely uppercase words,
which make converting to snakecase difficult.

### ferc_xbrl_extractor.datapackage.\_get_fields_from_concepts(concept: [ferc_xbrl_extractor.taxonomy.Concept](../taxonomy/index.md#ferc_xbrl_extractor.taxonomy.Concept), period_type: str) → tuple[list[[Field](#ferc_xbrl_extractor.datapackage.Field)], list[[Field](#ferc_xbrl_extractor.datapackage.Field)]]

Traverse concept tree to get columns and axes that will be used in output table.

A ‘fact table’ in XBRL arranges Concepts into a a tree where the leaf nodes are
individual facts that will become columns in the output tables. Axes are used to
identify context of each fact, and will become a part of the primary key in the
output table.

* **Parameters:**
  * **concept** – The root concept of the tree.
  * **period_type** – Period type of current table (only return columns with corresponding
    period type).
* **Returns:**
  Axes in table (become part of primary key).
  columns: List of fields in table.
* **Return type:**
  [axes](#ferc_xbrl_extractor.datapackage.FactTable.axes)

### ferc_xbrl_extractor.datapackage.\_lowercase_words(name: str) → str

Convert fully uppercase words so only first letter is uppercase.

Pattern finds uppercase characters that are immediately preceded by
an uppercase character. Later when the name is converted to snakecase,
an underscore would be inserted between each of these characters if this
conversion is not performed.

### ferc_xbrl_extractor.datapackage.clean_table_names(name: str) → str | None

Function to clean table names.

* **Parameters:**
  **name** – Unprocessed table name.
* **Returns:**
  Cleaned table name or None if table name doesn’t match expected
  : pattern.
* **Return type:**
  table_name

### *class* ferc_xbrl_extractor.datapackage.Schema(/, \*\*data: Any)

Bases: `pydantic.BaseModel`

A generic table schema, as per Frictionless Data specs.

See [https://specs.frictionlessdata.io/table-schema/](https://specs.frictionlessdata.io/table-schema/).

#### fields *: list[[Field](#ferc_xbrl_extractor.datapackage.Field)]*

#### primary_key *: list[str]*

#### *classmethod* from_concept_tree(concept: [ferc_xbrl_extractor.taxonomy.Concept](../taxonomy/index.md#ferc_xbrl_extractor.taxonomy.Concept), period_type: str) → [Schema](#ferc_xbrl_extractor.datapackage.Schema)

Deduce schema from concept tree.

Traverse Concept tree to get columns that should comprise output table.
Concepts with names ending in ‘Axis’ will become a part of the composite
primary key for each table. Tables with a duration period type will also
have the columns ‘entity_id’, ‘filing_name’, ‘start_date’, and ‘end_date’ in
their primary key, while tables with ‘instant’ period type will include
‘entity_id’, ‘filing_name’, and ‘date’. The remaining columns will come from
leaf nodes in the concept graph.

* **Parameters:**
  * **concept** – Root concept of concept tree.
  * **period_type** – Period type of table.

### *class* ferc_xbrl_extractor.datapackage.Dialect(/, \*\*data: Any)

Bases: `pydantic.BaseModel`

Dialect used for frictionless SQL resources.

#### table *: str*

### *class* ferc_xbrl_extractor.datapackage.Resource(/, \*\*data: Any)

Bases: `pydantic.BaseModel`

A generic tabular data resource, as per Frictionless Data specs.

See [https://specs.frictionlessdata.io/data-resource](https://specs.frictionlessdata.io/data-resource).

#### path *: str*

#### profile *: str* *= 'tabular-data-resource'*

#### name *: str*

#### dialect *: [Dialect](#ferc_xbrl_extractor.datapackage.Dialect)*

#### title *: str*

#### description *: str*

#### format_ *: str* *= None*

#### mediatype *: str* *= 'application/vnd.sqlite3'*

#### schema_ *: [Schema](#ferc_xbrl_extractor.datapackage.Schema)* *= None*

#### *classmethod* from_link_role(fact_table: [ferc_xbrl_extractor.taxonomy.LinkRole](../taxonomy/index.md#ferc_xbrl_extractor.taxonomy.LinkRole), period_type: str, db_uri: str) → [Resource](#ferc_xbrl_extractor.datapackage.Resource) | None

Generate a Resource from a fact table (defined by a LinkRole).

If the fact table is empty, i.e. there are no data columns, return None.

* **Parameters:**
  * **fact_table** – Link role which defines a fact table.
  * **period_type** – Period type of table.
  * **db_uri** – Path to database required for a Frictionless resource.

#### get_period_type() → Literal['instant', 'duration']

Helper function to get period type from schema.

#### merge_resources(other: [Resource](#ferc_xbrl_extractor.datapackage.Resource), other_version: str) → [Resource](#ferc_xbrl_extractor.datapackage.Resource)

Merge same resource from multiple taxonomies.

This method attempts to merge resource definitions from multiple taxonomies
creating a unified schema for the table in question. It does this by first
comparing the primary keys of the two tables. If the primary keys aren’t
exactly the same it will raise an error. For the remaining columns, this
method will check if there are any that are new or missing in `other`.
New columns will be added to the tables schema, and missing columns will
be logged, but remain in the schema.

### *class* ferc_xbrl_extractor.datapackage.FactTable(schema: [Schema](#ferc_xbrl_extractor.datapackage.Schema), period_type: str)

Class to handle constructing a dataframe from an XBRL fact table.

Structure of the dataframe is defined by the XBRL taxonomy. Facts and contexts
parsed from an individual XBRL filing are then used to populate the dataframe
with relevant data.

#### schema *: [Schema](#ferc_xbrl_extractor.datapackage.Schema)*

#### columns *: dict[str, str]*

#### axes *: list[str]*

#### data_columns *: list[str]*

#### instant *: bool*

#### construct_dataframe(instance: [ferc_xbrl_extractor.instance.Instance](../instance/index.md#ferc_xbrl_extractor.instance.Instance)) → pandas.DataFrame

Construct dataframe from a parsed XBRL instance.

* **Parameters:**
  **instance** – Parsed XBRL instance used to construct dataframe.

### *class* ferc_xbrl_extractor.datapackage.Datapackage(/, \*\*data: Any)

Bases: `pydantic.BaseModel`

A generic Data Package, as per Frictionless Data specs.

See [https://specs.frictionlessdata.io/data-package](https://specs.frictionlessdata.io/data-package).

#### profile *: str* *= 'tabular-data-package'*

#### name *: str*

#### title *: str* *= 'Ferc1 data extracted from XBRL filings'*

#### resources *: list[[Resource](#ferc_xbrl_extractor.datapackage.Resource)]*

#### *classmethod* from_taxonomies(taxonomies: dict[str, [ferc_xbrl_extractor.taxonomy.Taxonomy](../taxonomy/index.md#ferc_xbrl_extractor.taxonomy.Taxonomy)], db_uri: str, form_number: int = 1) → [Datapackage](#ferc_xbrl_extractor.datapackage.Datapackage)

Construct a Datapackage from parsed XBRL taxonomies.

FERC regularly releases new versions of their XBRL taxonomies, meaning
data from different years conforms to slightly different structures. This
method will attempt to merge these taxonomy versions into a single unified
schema defined in a Datapackage descriptor.

The exact logic for merging taxonomies is as follows. First, the oldest
available taxonomy is used to construct a baseline datapackage descriptor.
Next, it will parse subsequent versions and compare the set of tables
found with the baseline. New tables will be added to the schema, removed
tables will simply be logged but remain in the schema, and tables in both
versions will do a deeper column level comparison. For more info on the table
comparison, see `Resource.merge_resources`.

* **Parameters:**
  * **taxonomies** – List of taxonomies to merge into a Datapackage.
  * **db_uri** – Path to database required for a Frictionless resource.
  * **form_number** – FERC form number used for datapackage name.

#### get_fact_tables(filter_tables: set[str] | None = None) → dict[str, [FactTable](#ferc_xbrl_extractor.datapackage.FactTable)]

Use schema’s defined in datapackage resources to construct FactTables.

* **Parameters:**
  **filter_tables** – Optionally specify the set of tables to extract.
  If None, all possible tables will be extracted.

### ferc_xbrl_extractor.datapackage.fuzzy_dedup(df: pandas.DataFrame) → pandas.DataFrame

Deduplicate a 1-column dataframe with numbers that are close in value.

We pick the number with the highest precision, up to a max precision of 6
digits after the decimal point.

If we get passed duplicated str values, or non-numeric values at all, we
raise a ValueError - though we can add more code here to handle specific
cases if we need to.

* **Parameters:**
  **df** – the dataframe to be deduplicated.