From raw FAOSTAT rows to charts¶
This notebook follows the whole pipeline on real input: FAOSTAT trade rows as published, cleaned with nv.clean, built into NetworkX graphs with nv.build_graph, and drawn with nv.plot. It uses the sample bundled with netviz-tools (wheat, maize and soya beans, 2010 to 2024), so it runs offline.
The charts show numbers. The notebook states what the numbers are; it does not say why they changed.
import numpy as np
import pandas as pd
import plotly.io as pio
import netviz_tools as nv
from netviz_tools.datasets import faostat
pio.renderers.default = "notebook_connected"
pd.set_option("display.width", 120)
The raw table¶
faostat.raw_sample() returns the rows in the layout of the FAOSTAT bulk file:
raw = faostat.raw_sample()
COLS = ["Reporter Countries", "Partner Countries", "Item", "Element", "Year", "Unit", "Value"]
print(f"{len(raw):,} rows, {raw.shape[1]} columns")
raw[COLS].head()
140,935 rows, 13 columns
| Reporter Countries | Partner Countries | Item | Element | Year | Unit | Value | |
|---|---|---|---|---|---|---|---|
| 0 | Afghanistan | Iran (Islamic Republic of) | Maize (corn) | Export quantity | 2010 | t | 45.0 |
| 1 | Albania | Argentina | Maize (corn) | Import quantity | 2010 | t | 452.0 |
| 2 | Albania | Bosnia and Herzegovina | Maize (corn) | Import quantity | 2010 | t | 107.0 |
| 3 | Albania | Bulgaria | Maize (corn) | Import quantity | 2010 | t | 23.0 |
| 4 | Albania | Chile | Maize (corn) | Import quantity | 2010 | t | 18.0 |
raw["Element"].value_counts()
Element Import quantity 72012 Export quantity 68923 Name: count, dtype: int64
Each row is one country's report. Element says whether the reporter exported to the partner or imported from it, so the same flow can appear twice: once in the exporter's report and once in the importer's. The two reports rarely agree. Here is Ukraine's wheat going to Türkiye in 2022:
same_flow = raw[
(raw["Item"] == "Wheat")
& (raw["Year"] == 2022)
& (
((raw["Reporter Countries"] == "Ukraine") & (raw["Partner Countries"] == "Türkiye"))
| ((raw["Reporter Countries"] == "Türkiye") & (raw["Partner Countries"] == "Ukraine"))
)
]
same_flow[COLS]
| Reporter Countries | Partner Countries | Item | Element | Year | Unit | Value | |
|---|---|---|---|---|---|---|---|
| 134561 | Türkiye | Ukraine | Wheat | Export quantity | 2022 | t | 0.01 |
| 134562 | Türkiye | Ukraine | Wheat | Import quantity | 2022 | t | 2059037.61 |
| 134655 | Ukraine | Türkiye | Wheat | Export quantity | 2022 | t | 2116569.97 |
Ukraine reports 2,116,569.97 tonnes exported to Türkiye; Türkiye reports 2,059,037.61 tonnes imported from Ukraine. (Türkiye also reports exporting 0.01 tonnes to Ukraine, a separate flow in the other direction.) Before this table can become a graph, each row has to become a source-to-target flow, and each flow has to keep one of its two reports.
Adding a few problems¶
The FAOSTAT sample is fairly tidy. To show what else clean handles, the next cell damages a copy of it. These changes were made for this demonstration; they are not in the FAOSTAT data.
- Türkiye's report of its 2023 wheat imports from Ukraine is respelt "Turkiye".
- Egypt's 2023 wheat imports from the Russian Federation are written as text with thousands separators, "5,505,092.83".
- Indonesia's 2023 wheat imports from Australia are added a second time (an exact duplicate row).
- A copy of Morocco's 2023 wheat imports from France is added with the partner missing.
- An aggregate row is added with "World" as the partner of Argentina, holding the sum of Argentina's reported 2023 wheat exports, the way many statistical tables carry totals.
def row(reporter, partner, element, year=2023, item="Wheat"):
hit = raw[
(raw["Reporter Countries"] == reporter)
& (raw["Partner Countries"] == partner)
& (raw["Element"] == element)
& (raw["Year"] == year)
& (raw["Item"] == item)
]
assert len(hit) == 1
return hit.index[0]
messy = raw.copy()
messy["Value"] = messy["Value"].astype(object)
respelt = row("Türkiye", "Ukraine", "Import quantity")
messy.loc[respelt, "Reporter Countries"] = "Turkiye"
as_text = row("Egypt", "Russian Federation", "Import quantity")
messy.loc[as_text, "Value"] = f"{raw.loc[as_text, 'Value']:,}"
duplicate = raw.loc[[row("Indonesia", "Australia", "Import quantity")]]
no_partner = raw.loc[[row("Morocco", "France", "Import quantity")]].assign(
**{"Partner Countries": np.nan}
)
argentina = raw[
(raw["Reporter Countries"] == "Argentina")
& (raw["Element"] == "Export quantity")
& (raw["Year"] == 2023)
& (raw["Item"] == "Wheat")
]
world = argentina.iloc[[0]].assign(
**{
"Partner Countries": "World",
"Partner Country Code": 5000,
"Value": argentina["Value"].sum(),
}
)
added = pd.concat([duplicate, no_partner, world])
added.index = [900001, 900002, 900003]
messy = pd.concat([messy, added])
messy.loc[[respelt, as_text, *added.index], COLS]
| Reporter Countries | Partner Countries | Item | Element | Year | Unit | Value | |
|---|---|---|---|---|---|---|---|
| 137702 | Turkiye | Ukraine | Wheat | Import quantity | 2023 | t | 2496511.11 |
| 135692 | Egypt | Russian Federation | Wheat | Import quantity | 2023 | t | 5,505,092.83 |
| 900001 | Indonesia | Australia | Wheat | Import quantity | 2023 | t | 4344808.15 |
| 900002 | Morocco | NaN | Wheat | Import quantity | 2023 | t | 2291206.18 |
| 900003 | Argentina | World | Wheat | Export quantity | 2023 | t | 2844595.54 |
Cleaning¶
nv.clean works out which column plays which role from the headers, turns each report into a source-to-target flow ("Export" rows keep the reporter as source, "Import" rows reverse it), and keeps one report per flow. drop_nodes=["World"] removes the aggregate. The report lists every step:
flows, report = nv.clean(messy, drop_nodes=["World"])
print(report.summary())
Cleaned 140,938 raw rows into 93,749 flows.
Columns: 'Reporter Countries' -> reporter (guessed), 'Partner Countries' -> partner (guessed), 'Element' -> direction (guessed), 'Year' -> time (guessed), 'Item' -> category (guessed), 'Value' -> weight (guessed), 'Unit' -> unit (guessed)
- exact duplicates: 140,938 -> 140,937 rows
- missing values: 140,937 -> 140,936 rows (1 missing partner)
- direction: 140,936 -> 140,936 rows ('Export quantity' -> export, 'Import quantity' -> import)
- weight: 140,936 -> 140,936 rows
- negative weight: 140,936 -> 140,936 rows
- time: 140,936 -> 140,936 rows
- names: 140,936 -> 140,936 rows (1 spellings changed: 0 by aliases, 1 case, accent or punctuation variants merged)
- drop nodes: 140,936 -> 140,935 rows (dropped rows touching 'World')
- zeros: 140,935 -> 140,935 rows
- self-loops: 140,935 -> 140,870 rows
- duplicates: 140,870 -> 140,870 rows
- mirror flows: 140,870 -> 93,749 rows (dropped 47,121 exporter reports of flows both sides reported)
Mirror flows: 47,121 reported by both sides, 21,790 by the source only, 24,838 by the target only; median disagreement 30.0%; used the target's report where both exist.
Renamed 1 spellings:
'Turkiye' -> 'Türkiye'
Every removed row is kept in report.dropped with its reason. Most of them are the exporter's copy of a flow that the importer also reported:
report.dropped["reason"].value_counts()
reason mirror flow: the target's report was used 47121 self-loop 65 exact duplicate of an earlier row 1 missing partner 1 node listed in drop_nodes 1 Name: count, dtype: int64
report.dropped.loc[added.index, [*COLS, "reason"]]
| Reporter Countries | Partner Countries | Item | Element | Year | Unit | Value | reason | |
|---|---|---|---|---|---|---|---|---|
| 900001 | Indonesia | Australia | Wheat | Import quantity | 2023 | t | 4344808.15 | exact duplicate of an earlier row |
| 900002 | Morocco | NaN | Wheat | Import quantity | 2023 | t | 2291206.18 | missing partner |
| 900003 | Argentina | World | Wheat | Export quantity | 2023 | t | 2844595.54 | node listed in drop_nodes |
report.renamed
{'Turkiye': 'Türkiye'}
The duplicate, the row without a partner and the "World" row were dropped, each with its own reason. "Turkiye" was merged into "Türkiye" because the two spellings are identical once accents are removed; the more frequent spelling won. Near misses that are not identical after that normalization (typos, for example) are only listed in report.possible_aliases, never merged. The text value was parsed as a number.
After cleaning, the damaged table gives exactly the same flows as the untouched sample:
print(flows.equals(nv.clean(raw)[0]))
flows.head()
True
| source | target | time | category | weight | unit | reported_by | |
|---|---|---|---|---|---|---|---|
| 0 | Afghanistan | Iran (Islamic Republic of) | 2010 | Maize (corn) | 45.0 | t | exporter |
| 1 | Afghanistan | Spain | 2010 | Maize (corn) | 17.0 | t | importer |
| 2 | Antigua and Barbuda | Ethiopia | 2010 | Maize (corn) | 53.0 | t | importer |
| 3 | Antigua and Barbuda | Saint Vincent and the Grenadines | 2010 | Maize (corn) | 2.0 | t | importer |
| 4 | Argentina | Albania | 2010 | Maize (corn) | 452.0 | t | importer |
Mirror flows¶
report.mirror summarises the two reports of each flow:
m = report.mirror
print(f"reported by both sides: {m.both:,}")
print(f"reported by exporter only: {m.source_only:,}")
print(f"reported by importer only: {m.target_only:,}")
print(f"median disagreement where both report: {m.median_disagreement:.1%}")
print(f"kept: the {m.prefer}'s (importer's) report")
reported by both sides: 47,121 reported by exporter only: 21,790 reported by importer only: 24,838 median disagreement where both report: 30.0% kept: the target's (importer's) report
Where both sides reported a flow, the two numbers differ by 30.0% in the median case. prefer="target" (the default) keeps the importer's report; prefer="source", "mean" and "max" are the alternatives. The reported_by column records which report each flow came from. For Ukraine to Türkiye in 2022, the kept value is Türkiye's 2,059,037.61 tonnes:
flows[
(flows["source"] == "Ukraine")
& (flows["target"] == "Türkiye")
& (flows["time"] == 2022)
& (flows["category"] == "Wheat")
]
| source | target | time | category | weight | unit | reported_by | |
|---|---|---|---|---|---|---|---|
| 89421 | Ukraine | Türkiye | 2022 | Wheat | 2059037.61 | t | importer |
Building a graph¶
nv.build_graph turns one slice (one year, one item) into a weighted DiGraph. node_attrs=faostat.countries() attaches each country's continent and label point.
g = nv.build_graph(flows, time=2022, category="Wheat", node_attrs=faostat.countries())
print(f"{g.number_of_nodes()} countries, {g.number_of_edges():,} flows")
print({k: v for k, v in g.graph.items() if k != "labels"})
186 countries, 2,143 flows
{'aggregate': None, 'unit': 't', 'time': 2022, 'category': 'Wheat'}
nv.plot.auto picks a chart from the data. Every node here is a country with known coordinates, so it draws a flow map, and says so under the title and in fig.layout.meta:
fig = nv.plot.auto(g)
print(fig.layout.meta["netviz"])
fig.show()
{'kind': 'flow_map', 'reason': 'every node has coordinates'}
The same graph as a network: the 30 countries with the largest exports, coloured by continent, with the 10 largest labelled. Without color_by, nodes would be coloured by trade community, as on the map above.
nv.plot.network(
g,
top_n=30,
color_by="continent",
size_by="out_strength",
show_labels=10,
title="Wheat trade, 2022: 30 largest exporters",
).show()
Which exporters fell the most?¶
nv.plot.ranking with two years and change=True ranks countries by the change in a value between them. size_by="out_strength" is the total a country sends, which for trade data is its exports. The plot takes the flow table directly and builds the graphs itself.
nv.plot.ranking(
flows, category="Wheat", time=[2022, 2023], size_by="out_strength", change=True
).show()
wheat_exports = (
flows[(flows["category"] == "Wheat") & flows["time"].isin([2022, 2023])]
.pivot_table(index="source", columns="time", values="weight", aggfunc="sum", fill_value=0)
.div(1e6)
.assign(change=lambda d: d[2023] - d[2022])
.sort_values("change")
)
pd.concat([wheat_exports.head(3), wheat_exports.tail(3)]).round(1)
| time | 2022 | 2023 | change |
|---|---|---|---|
| source | |||
| Argentina | 14.5 | 3.1 | -11.4 |
| India | 7.3 | 0.2 | -7.1 |
| France | 20.2 | 13.8 | -6.4 |
| Ukraine | 9.8 | 14.5 | 4.7 |
| Canada | 17.2 | 25.8 | 8.6 |
| Russian Federation | 22.4 | 34.0 | 11.5 |
Argentina has the largest drop: its wheat exports went from 14.5 million tonnes in 2022 to 3.1 million tonnes in 2023, 11.4 million tonnes less. India follows with 7.1 million tonnes less and France with 6.4 million tonnes less. The largest increase is the Russian Federation's: 22.4 million tonnes in 2022 and 34.0 million in 2023, 11.5 million more. The chart does not say why any of these changed.
nv.plot.time_series with focus= follows one country's exports and imports year by year:
nv.plot.time_series(flows, focus="Argentina", category="Wheat").show()
What does one country send and receive?¶
nv.plot.compare puts one country's exports and imports of each item side by side:
nv.plot.compare(flows, "Brazil", time=2024).show()
brazil = pd.DataFrame(
{
"exports": flows[flows["source"] == "Brazil"].groupby(["time", "category"])["weight"].sum(),
"imports": flows[flows["target"] == "Brazil"].groupby(["time", "category"])["weight"].sum(),
}
).div(1e6)
brazil.loc[[2023, 2024]].round(2)
| exports | imports | ||
|---|---|---|---|
| time | category | ||
| 2023 | Maize (corn) | 51.14 | 1.36 |
| Soya beans | 98.73 | 0.18 | |
| Wheat | 2.50 | 4.18 | |
| 2024 | Maize (corn) | 43.10 | 1.65 |
| Soya beans | 99.75 | 0.82 | |
| Wheat | 2.90 | 6.65 |
In 2024 Brazil exported 99.75 million tonnes of soya beans and 43.10 million tonnes of maize. For wheat the direction is reversed: 6.65 million tonnes imported against 2.90 million exported. With two years, grey ticks mark 2023, and hover gives the change:
nv.plot.compare(flows, "Brazil", time=[2023, 2024]).show()
Maize exports were 51.14 million tonnes in 2023 and 43.10 million in 2024; wheat imports rose from 4.18 to 6.65 million tonnes.
Flows over time on a map¶
Pass a table with several years to nv.plot.flow_map and the map becomes an animation, one frame per year, with a play button and a slider. Each frame draws that year's 40 largest flows:
nv.plot.flow_map(
flows, category="Soya beans", top_n=40, title="Soya bean trade, 40 largest flows per year"
).show()
brazil_china = (
flows[
(flows["category"] == "Soya beans")
& (flows["source"] == "Brazil")
& (flows["target"] == "China, mainland")
]
.set_index("time")["weight"]
.div(1e6)
.round(1)
)
brazil_china.loc[[2010, 2015, 2020, 2024]]
time 2010 18.6 2015 40.1 2020 64.3 2024 74.6 Name: weight, dtype: float64
soya = flows[flows["category"] == "Soya beans"]
largest = soya.loc[soya.groupby("time")["weight"].idxmax(), ["time", "source", "target", "weight"]]
largest.assign(weight=largest["weight"].div(1e6).round(1)).set_index("time")
| source | target | weight | |
|---|---|---|---|
| time | |||
| 2010 | United States of America | China, mainland | 23.6 |
| 2011 | United States of America | China, mainland | 22.2 |
| 2012 | United States of America | China, mainland | 26.0 |
| 2013 | Brazil | China, mainland | 31.8 |
| 2014 | Brazil | China, mainland | 32.0 |
| 2015 | Brazil | China, mainland | 40.1 |
| 2016 | Brazil | China, mainland | 38.2 |
| 2017 | Brazil | China, mainland | 50.9 |
| 2018 | Brazil | China, mainland | 66.1 |
| 2019 | Brazil | China, mainland | 57.7 |
| 2020 | Brazil | China, mainland | 64.3 |
| 2021 | Brazil | China, mainland | 58.1 |
| 2022 | Brazil | China, mainland | 54.4 |
| 2023 | Brazil | China, mainland | 71.6 |
| 2024 | Brazil | China, mainland | 74.6 |
Brazil's soya bean exports to mainland China grow from 18.6 million tonnes in 2010 to 74.6 million tonnes in 2024. The largest single flow in 2010, 2011 and 2012 runs from the United States to mainland China (23.6 million tonnes in 2010); from 2013 on it is Brazil to mainland China. focus="Brazil" keeps only the flows that touch Brazil:
nv.plot.flow_map(
flows, category="Soya beans", focus="Brazil", top_n=25, title="Brazil's soya bean flows"
).show()
Nothing in these calls is specific to trade. A migration table with origin, destination and year columns, or any other origin-destination table with a time column, goes through the same nv.clean (or nv.to_flowframe if it is already tidy) and the same flow_map call. Pass coords= with longitude and latitude when the node names are not FAOSTAT countries.
Summary¶
nv.cleanturned 140,938 raw rows (140,935 FAOSTAT rows plus 3 added ones) into 93,749 flows, and listed every dropped row with its reason. The added problems were all handled: one duplicate, one missing partner, one "World" aggregate dropped; one spelling merged; one text value parsed.- Where exporter and importer both reported a flow, their numbers differ by 30.0% in the median case; the importer's report was kept.
- Argentina's wheat exports fell by 11.4 million tonnes from 2022 to 2023, the largest drop in the sample.
- Brazil exported 99.75 million tonnes of soya beans in 2024; Brazil to mainland China grew from 18.6 to 74.6 million tonnes between 2010 and 2024.
Data: FAO. 2025. FAOSTAT: Detailed trade matrix. Accessed on 27 September 2026. https://www.fao.org/faostat/en/#data/TM. Licence: CC-BY-4.0.