clean

Step one of the pipeline: turn a messy edge or trade table into a flow frame, and get a report of every change. Available as nv.clean, nv.CleaningReport, nv.CleaningStep and nv.MirrorStats. The bundled FAOSTAT sample in its raw bulk-file layout, nv.datasets.faostat.raw_sample(), is a ready-made messy input to try it on.

clean

clean(
    raw: DataFrame,
    *,
    source: str | None = None,
    target: str | None = None,
    weight: str | None = None,
    time: str | None = None,
    category: str | None = None,
    unit: str | None = None,
    reporter: str | None = None,
    partner: str | None = None,
    direction: str | None = None,
    reported_by: str | None = None,
    fill: Mapping[str, object] | None = None,
    direction_values: Mapping[str, Direction] | None = None,
    prefer: Prefer = "target",
    aliases: Mapping[str, str] | None = None,
    unit_conversions: Mapping[str, tuple[str, float]]
    | None = None,
    drop_nodes: Iterable[str] = (),
    zeros: Literal["drop", "keep"] = "drop",
    self_loops: bool = False,
    duplicates: Literal["sum", "keep"] = "sum",
    labels: Mapping[str, str] | None = None,
) -> tuple[FlowFrame, CleaningReport]

Turn a messy edge or trade table into a flow frame, and report every change.

Parameters:

Name	Type	Description	Default
raw	DataFrame	The table as it arrived. It is not modified.	required
source	str | None	Names of the raw columns that hold each role. Any left as None are guessed from the header names (case, spaces and underscores are ignored; columns whose names contain "code", "(m49)" or "flag" are skipped). For example origin, exporter or from is taken as the source, and value, quantity or count as the weight.	None
target	str | None	Names of the raw columns that hold each role. Any left as None are guessed from the header names (case, spaces and underscores are ignored; columns whose names contain "code", "(m49)" or "flag" are skipped). For example origin, exporter or from is taken as the source, and value, quantity or count as the weight.	None
weight	str | None	Names of the raw columns that hold each role. Any left as None are guessed from the header names (case, spaces and underscores are ignored; columns whose names contain "code", "(m49)" or "flag" are skipped). For example origin, exporter or from is taken as the source, and value, quantity or count as the weight.	None
time	str | None	Names of the raw columns that hold each role. Any left as None are guessed from the header names (case, spaces and underscores are ignored; columns whose names contain "code", "(m49)" or "flag" are skipped). For example origin, exporter or from is taken as the source, and value, quantity or count as the weight.	None
category	str | None	Names of the raw columns that hold each role. Any left as None are guessed from the header names (case, spaces and underscores are ignored; columns whose names contain "code", "(m49)" or "flag" are skipped). For example origin, exporter or from is taken as the source, and value, quantity or count as the weight.	None
unit	str | None	Names of the raw columns that hold each role. Any left as None are guessed from the header names (case, spaces and underscores are ignored; columns whose names contain "code", "(m49)" or "flag" are skipped). For example origin, exporter or from is taken as the source, and value, quantity or count as the weight.	None
reporter	str | None	For trade tables where each row is one country's report: the reporting country, the partner country, and the column that says whether the row is an export or an import. Use these instead of source and target.	None
partner	str | None	For trade tables where each row is one country's report: the reporting country, the partner country, and the column that says whether the row is an export or an import. Use these instead of source and target.	None
direction	str | None	For trade tables where each row is one country's report: the reporting country, the partner country, and the column that says whether the row is an export or an import. Use these instead of source and target.	None
reported_by	str | None	For source/target tables that already hold both reports of a flow: the column saying who reported it (values containing "export" or "import").	None
fill	Mapping[str, object] | None	Constants for roles the table has no column for, keyed by time, category, weight or unit, for example fill={"time": 2020, "category": "migrants", "unit": "persons"}.	None
direction_values	Mapping[str, Direction] | None	Raw direction value to "export" or "import". By default a value containing "export" (case-insensitive) is an export and one containing "import" is an import. Rows with other values are dropped.	None
prefer	Prefer	For a flow reported by both sides: "target" keeps the target's (importer's) report, the same rule as faostat.load(..., reporter="combined"); "source" keeps the source's report; "mean" and "max" combine the two (reported_by becomes "both").	'target'
aliases	Mapping[str, str] | None	Name to replacement, applied to source, target and category names after whitespace normalization, for example {"Turkey": "Türkiye"}.	None
unit_conversions	Mapping[str, tuple[str, float]] | None	Unit to (new_unit, factor); matching weights are multiplied by factor, for example {"1000 An": ("head", 1000.0)}.	None
drop_nodes	Iterable[str]	Names of rows to drop wherever they appear as source or target, such as the aggregate "World".	()
zeros	Literal['drop', 'keep']	"drop" (default) removes zero weights; "keep" keeps them.	'drop'
self_loops	bool	Keep flows from a node to itself.	False
duplicates	Literal['sum', 'keep']	Rows that share (source, target, time, category, unit, reported_by) are summed ("sum", default) or kept as separate rows ("keep"). Rows identical in every raw column are always dropped first.	'sum'
labels	Mapping[str, str] | None	Display labels for charts, merged over the ones worked out from the raw table (see Returns).	None

Returns:

Name	Type	Description
flows	FlowFrame	The flow-schema columns, then reported_by when the table says who reported each flow ("exporter", "importer" or "both"). Other raw columns are dropped; discarded rows keep them in report.dropped. Names are strings, the time column is int64 and the weight column float64. Rows are sorted by category, time, source, target and unit, with a fresh index. Several units may remain; see report.units. flows.attrs["labels"] holds display labels for charts: the raw column names (so a column called "Persons" labels the weight axis "Persons"), or for reporter tables with export/import directions "Exporter", "Importer", "Exports" and "Imports".
report	CleaningReport	The columns used, per-step row counts, renames, look-alike names, mirror statistics and the dropped rows with reasons.

Raises:

Type	Description
SchemaError	If the columns cannot be worked out (none match, two match the same role, or a required role is missing), a named column does not exist, or a role is given both as a column and in fill.
ValueError	If an option or a fill value is not valid.

Notes

Mirror flows are resolved after duplicates. With prefer="mean" or "max" each side's rows are summed before combining, even with duplicates="keep". prefer has no effect when the table does not say who reported each flow.

Spellings are merged automatically only when they are identical after casefolding, removing accents and removing punctuation and spaces ("Cote d'Ivoire" and "Côte d'Ivoire"); the most frequent spelling wins, ties going to the first alphabetically. Near matches such as typos are only listed in report.possible_aliases.

Examples:

>>> import pandas as pd
>>> raw = pd.DataFrame({
...     "origin": ["Türkiye", "Türkiye", "Turkiye ", "Egypt"],
...     "destination": ["Egypt", "Egypt", "Iraq", "World"],
...     "year": ["2021", 2021, 2021.0, 2021],
...     "value": ["1,000", 250, "40", 5],
... })
>>> flows, report = clean(raw, fill={"category": "Wheat", "unit": "t"},
...                       drop_nodes=["World"])
>>> flows.iloc[:, [0, 1, 4]].values.tolist()  # source, target, weight
[['Türkiye', 'Egypt', 1250.0], ['Türkiye', 'Iraq', 40.0]]
>>> report.renamed
{'Turkiye ': 'Türkiye'}

CleaningReport dataclass

CleaningReport(
    columns: dict[str, str],
    guessed: tuple[str, ...],
    filled: dict[str, str | int | float],
    rows_in: int,
    rows_out: int,
    steps: tuple[CleaningStep, ...],
    renamed: dict[str, str],
    possible_aliases: tuple[tuple[str, str, float], ...],
    mirror: MirrorStats | None,
    units: dict[str, dict[str, int]],
    dropped: DataFrame,
)

Everything clean did to a table.

Parameters:

Name	Type	Description	Default
columns	dict[str, str]	Raw column name to role, for every column that was used.	required
guessed	tuple[str, ...]	Roles whose column was guessed from the header names. The other roles in columns were given by the caller.	required
filled	dict[str, str | int | float]	Roles filled with a constant (the fill argument).	required
rows_in	int	Rows in the raw table.	required
rows_out	int	Rows in the returned flow frame.	required
steps	tuple[CleaningStep, ...]	Each step with its row counts, in the order they ran.	required
renamed	dict[str, str]	Original spelling to final name, for source, target and category names that changed (whitespace, aliases, or merged variants).	required
possible_aliases	tuple[tuple[str, str, float], ...]	Pairs of distinct final names that look alike (difflib ratio of at least 0.9) as (name_a, name_b, ratio). They are not merged; pass them in aliases if they are the same thing.	required
mirror	MirrorStats | None	Agreement between source and target reports, or None when the table does not say who reported each flow.	required
units	dict[str, dict[str, int]]	For each category in the output, the number of rows per unit.	required
dropped	DataFrame	The discarded raw rows, with their original index and columns plus a reason column. Rows that were summed or combined into another row are not listed here, because their weight is kept.	required

to_frame

to_frame() -> DataFrame

Return the steps as a table.

Returns:

Type	Description
DataFrame	Columns step, rows_in, rows_out, removed and detail, one row per step.

summary

summary() -> str

Describe the cleaning in plain English, one line per fact.

Returns:

Type	Description
str	A multi-line summary: the row counts, the columns used and the constants filled in, one line per step, then mirror statistics, renames, possible aliases and categories with more than one unit.

CleaningStep dataclass

CleaningStep(
    name: str, rows_in: int, rows_out: int, detail: str = ""
)

One step of clean and its effect on the row count.

Parameters:

Name	Type	Description	Default
name	str	Short name of the step, for example "zeros".	required
rows_in	int	Rows before the step.	required
rows_out	int	Rows after the step. Steps that aggregate (duplicates, mirror flows) reduce the count without discarding weight.	required
detail	str	What the step did, in plain words. May be empty.	''

MirrorStats dataclass

MirrorStats(
    both: int,
    source_only: int,
    target_only: int,
    median_disagreement: float | None,
    prefer: Prefer,
)

How often, and how closely, the two sides of a flow agree.

A flow is identified by (source, target, time, category, unit). In a reporter-based trade table it can be reported by its source (the exporter), by its target (the importer), or by both.

Parameters:

Name	Type	Description	Default
both	int	Flows reported by both sides.	required
source_only	int	Flows reported only by their source.	required
target_only	int	Flows reported only by their target.	required
median_disagreement	float | None	Median over the both flows of |s - t| / max(s, t), where s and t are the two reported weights (0 when both are 0). A value of 0.1 means that for the typical flow the smaller report is 10% below the larger one. None when no flow was reported by both sides.	required
prefer	Prefer	The prefer setting used to resolve the both flows.	required