datasets.faostat¶

The FAOSTAT Detailed Trade Matrix: download, local Parquet store, bundled sample and metadata. See Data and licences for provenance and the ETL rules.

FAOSTAT Detailed Trade Matrix: download, local Parquet store, and queries.

The Detailed Trade Matrix (FAOSTAT domain TM) records bilateral trade in about 560 agricultural products for roughly 220 countries and territories from 1986. FAO publishes it as a single zip of about 420 MB (8.5 GB of CSV).

This module works in two steps:

build_store downloads the zip once with pooch, checks its SHA-256 against the value pinned in this release, and converts it with DuckDB into a Parquet dataset partitioned by item. A manifest.json written next to the data records the source URL, the SHA-256 actually used, the retrieval time, row counts and the library version.
load runs a DuckDB query over only the partitions you ask for and returns a flow frame. Queries take well under a second per item.

load_sample runs the same query over a small Parquet file that ships with the package (wheat, maize and soya beans, 2010 to 2024), so the examples and tests work offline.

Flow frames and labels

The store keeps FAOSTAT's own vocabulary, but load and load_sample return the library's generic flow frame (netviz_tools.FLOW_COLUMNS):

source: the exporting country;
target: the importing country;
time: the year;
category: the FAOSTAT item (commodity);
weight: the quantity, or the trade value with measure="value";
unit: t, head, number or 1000 USD.

Every returned frame also carries a copy of LABELS in flows.attrs["labels"] (with "weight": "Value" for trade values). netviz_tools.build_graph copies it into G.graph["labels"], and the plots use these display labels, so FAOSTAT charts say "Exporter" and "Importer" without you having to type them.

Reporting perspective

Every trade flow can be reported twice: by the exporting country and by the importing country. The two reports often differ, and some countries stop reporting (the Russian Federation reports no exports after 2021). load lets you choose:

"importer" (default): flows as reported by the importing country. More countries report imports than exports (for wheat in 2021, 155 against 99).
"exporter": flows as reported by the exporting country.
"combined": the importer's report where one exists, otherwise the exporter's, decided per (exporter, importer, item, year).

The reporter parameter and the reported_by column of details=True keep the FAOSTAT words "importer" and "exporter".

Data licence

FAOSTAT data are published under CC BY 4.0. Cite them as, for example: "FAO. 2025. FAOSTAT: Detailed trade matrix. Accessed on 27 September 2026. https://www.fao.org/faostat/en/#data/TM. Licence: CC-BY-4.0." Use CITATION for the template.

Reporter `module-attribute` ¶

Reporter: TypeAlias = Literal[
    "importer", "exporter", "combined"
]

Whose report of each flow to use. See the module documentation.

Measure `module-attribute` ¶

Measure: TypeAlias = Literal['quantity', 'value']

"quantity" returns physical amounts (t, head or number); "value" returns trade value in 1000 USD.

ENV_CACHE_DIR `module-attribute` ¶

ENV_CACHE_DIR: Final = 'NETVIZ_TOOLS_CACHE'

Environment variable that overrides the default cache directory.

CITATION `module-attribute` ¶

CITATION: Final = "FAO. {year}. FAOSTAT: Detailed trade matrix. Accessed on {accessed}. https://www.fao.org/faostat/en/#data/TM. Licence: CC-BY-4.0."

Citation template required by the FAO database terms of use.

LABELS `module-attribute` ¶

LABELS: Final[Mapping[str, str]] = {
    "source": "Exporter",
    "target": "Importer",
    "time": "Year",
    "category": "Item",
    "weight": "Quantity",
    "node": "Country",
    "out": "Exports",
    "in": "Imports",
}

Display labels for FAOSTAT trade flows, keyed by generic flow-frame term.

load and load_sample attach a copy to every frame as flows.attrs["labels"] (with "weight": "Value" when measure="value"); graphs and plots pick them up from there.

default_cache_dir ¶

default_cache_dir() -> Path

Return the cache directory used when none is given.

The NETVIZ_TOOLS_CACHE environment variable takes precedence; otherwise the operating system's user cache directory is used (via pooch.os_cache). Nothing is created until data is downloaded.

Returns:

Type	Description
`Path`	The cache directory.

store_path ¶

store_path(
    cache_dir: str | PathLike[str] | None = None,
) -> Path

Return the directory that holds (or will hold) the Parquet store.

Parameters:

Name	Type	Description	Default
`cache_dir`	`str \| PathLike[str] \| None`	Cache directory. Defaults to `default_cache_dir`.	`None`

Returns:

Type	Description
`Path`	`<cache_dir>/faostat_trade`.

items ¶

items(search: str | None = None) -> DataFrame

Return the catalogue of FAOSTAT trade items.

The catalogue ships with the package, so it works before any download.

Parameters:

Name	Type	Description	Default
`search`	`str \| None`	Optional case-insensitive substring to filter item names and slugs.	`None`

Returns:

Type	Description
`DataFrame`	Columns `item_code`, `item` (the FAOSTAT name), `slug` (the snake_case name used by netviz_tools 0.x), `cpc_code`, `quantity_units`, `first_year` and `last_year`.

Examples:

>>> from netviz_tools.datasets import faostat
>>> faostat.items("wheat")["item"].head(2).tolist()
['Wheat', 'Wheat and meslin flour']

countries ¶

countries() -> DataFrame

Return metadata for every FAOSTAT area in the trade matrix.

Regions come from the UN M49 standard; label points (lon, lat) come from Natural Earth. Pass the result to netviz_tools.build_graph as node_attrs to colour nodes by continent.

Returns:

Type	Description
`DataFrame`	Indexed by FAOSTAT area `name`, with columns `fao_code`, `m49`, `iso3`, `region`, `subregion`, `continent`, `lon` and `lat`. `continent` is the M49 region with the Americas split into Northern America, Central America, the Caribbean and South America.

download ¶

download(
    cache_dir: str | PathLike[str] | None = None,
    *,
    known_hash: str | None = FAOSTAT_TRADE_SHA256,
    progressbar: bool = False,
) -> Path

Download the FAOSTAT bulk zip into the cache, or reuse a cached copy.

Parameters:

Name	Type	Description	Default
`cache_dir`	`str \| PathLike[str] \| None`	Cache directory. Defaults to `default_cache_dir`.	`None`
`known_hash`	`str \| None`	Expected SHA-256. Defaults to the value pinned in this release. Pass `None` explicitly to accept whatever file FAO currently serves.	`FAOSTAT_TRADE_SHA256`
`progressbar`	`bool`	Show a download progress bar (requires `tqdm`).	`False`

Returns:

Type	Description
`Path`	Path to the zip.

Raises:

Type	Description
`SourceHashMismatchError`	If the file does not match `known_hash`.

build_store ¶

build_store(
    cache_dir: str | PathLike[str] | None = None,
    *,
    zip_path: str | PathLike[str] | None = None,
    known_hash: str | None = FAOSTAT_TRADE_SHA256,
    force: bool = False,
    progressbar: bool = False,
    threads: int | None = None,
) -> Path

Build the local Parquet store from the FAOSTAT bulk zip.

This is a one-time step. It needs network access (unless zip_path is given), about 9 GB of temporary disk space for the extracted CSV, and a minute or two on a laptop. The finished store is about 400 MB.

Parameters:

Name	Type	Description	Default
`cache_dir`	`str \| PathLike[str] \| None`	Cache directory for the zip and the store. Defaults to `default_cache_dir`.	`None`
`zip_path`	`str \| PathLike[str] \| None`	Use a zip you already have instead of downloading it. It is still checked against `known_hash`.	`None`
`known_hash`	`str \| None`	Expected SHA-256 of the zip. Defaults to the value pinned in this release. FAO replaces the file in place when it publishes an update; to build from a newer file, pass `known_hash=None` explicitly. The SHA-256 actually used is always written to the manifest.	`FAOSTAT_TRADE_SHA256`
`force`	`bool`	Rebuild even if a store already exists.	`False`
`progressbar`	`bool`	Show a download progress bar (requires `tqdm`).	`False`
`threads`	`int \| None`	Number of DuckDB threads. Defaults to all cores.	`None`

Returns:

Type	Description
`Path`	The store directory.

Raises:

Type	Description
`SourceHashMismatchError`	If the zip does not match `known_hash`.
`SchemaError`	If the CSV inside the zip does not have the expected columns.

store_info ¶

store_info(
    cache_dir: str | PathLike[str] | None = None,
) -> dict[str, Any]

Return the manifest of the local store.

Parameters:

Name	Type	Description	Default
`cache_dir`	`str \| PathLike[str] \| None`	Cache directory. Defaults to `default_cache_dir`.	`None`

Returns:

Type	Description
`dict`	The parsed `manifest.json`.

Raises:

Type	Description
`StoreNotFoundError`	If the store has not been built.

load ¶

load(
    items: str | int | Iterable[str | int],
    years: int | Iterable[int] | None = None,
    *,
    reporter: Reporter = "importer",
    measure: Measure = "quantity",
    self_loops: bool = False,
    details: bool = False,
    cache_dir: str | PathLike[str] | None = None,
) -> FlowFrame

Load flows for some items and years from the local store.

Parameters:

Name	Type	Description	Default
`items`	`str \| int \| Iterable[str \| int]`	One or more items, each given as a FAOSTAT item code (`15`), a FAOSTAT name (`"Wheat"`, case-insensitive), or a 0.x slug (`"potatoes_frozen"`). See `items` for the catalogue.	required
`years`	`int \| Iterable[int] \| None`	A year, an iterable of years (such as `range(2015, 2023)`), or `None` for all years.	`None`
`reporter`	`Reporter`	`"importer"` (default), `"exporter"` or `"combined"`. See the module documentation.	`'importer'`
`measure`	`Measure`	`"quantity"` (default) or `"value"` (`1000 USD`).	`'quantity'`
`self_loops`	`bool`	Keep flows reported between a country and itself (re-imports).	`False`
`details`	`bool`	Add `item_code`, `source_code` and `target_code` (FAOSTAT area codes), `reported_by` (`"exporter"` or `"importer"`) and `flag` columns.	`False`
`cache_dir`	`str \| PathLike[str] \| None`	Cache directory that holds the store.	`None`

Returns:

Type	Description
`FlowFrame`	Columns `source` (exporter), `target` (importer), `time` (year), `category` (item), `weight` (quantity or value) and `unit`, sorted by category, time, source and target. `flows.attrs["labels"]` holds a copy of `LABELS`, which plots use for axis and hover labels.

Raises:

Type	Description
`UnknownItemError`	If an item is not in the catalogue. The error suggests close matches.
`StoreNotFoundError`	If `build_store` has not been run.

load_sample ¶

load_sample(
    items: str | int | Iterable[str | int] | None = None,
    years: int | Iterable[int] | None = None,
    *,
    reporter: Reporter = "importer",
    self_loops: bool = False,
    details: bool = False,
) -> FlowFrame

Load flows from the small sample that ships with the package.

The sample holds quantity flows in tonnes (both reporting perspectives) for Wheat, Maize (corn) and Soya beans from 2010 to 2024, taken from the pinned FAOSTAT release (about 140,000 rows, 0.6 MB). It uses the same query as load.

Parameters:

Name	Type	Description	Default
`items`	`str \| int \| Iterable[str \| int] \| None`	Items to keep; defaults to all three sample items.	`None`
`years`	`int \| Iterable[int] \| None`	Years to keep; defaults to all.	`None`
`reporter`	`Reporter`	As for `load`.	`'importer'`
`self_loops`	`Reporter`	As for `load`.	`'importer'`
`details`	`Reporter`	As for `load`.	`'importer'`

Returns:

Type	Description
`FlowFrame`	The same columns as `load`: `source` (exporter), `target` (importer), `time` (year), `category` (item), `weight` (quantity) and `unit`, sorted by category, time, source and target. `flows.attrs["labels"]` holds a copy of `LABELS`, which plots use for axis and hover labels.

Raises:

Type	Description
`UnknownItemError`	If an item is not in the catalogue, or is in the catalogue but not in the sample.

Examples:

>>> from netviz_tools.datasets import faostat
>>> flows = faostat.load_sample(items="Wheat", years=2022)
>>> flows.columns.tolist()
['source', 'target', 'time', 'category', 'weight', 'unit']
>>> flows.attrs["labels"]["source"], flows.attrs["labels"]["target"]
('Exporter', 'Importer')

raw_sample ¶

raw_sample(
    items: str | int | Iterable[str | int] | None = None,
    years: int | Iterable[int] | None = None,
) -> DataFrame

Return the bundled sample in the column layout of the FAOSTAT bulk file.

This is the bundled sample (see load_sample) re-expanded into the layout of the bulk CSV, so that netviz_tools.clean can be demonstrated offline on a realistic reporter-based table. It is not a fresh download. Each row is one country's report: a flow reported by the exporter appears with that country as reporter and the element "Export quantity"; a flow reported by the importer appears with the importer as reporter and "Import quantity". Self-loops are included, as in the bulk file. The M49 and CPC code columns of the bulk file are omitted because the sample does not carry them.

Parameters:

Name	Type	Description	Default
`items`	`str \| int \| Iterable[str \| int] \| None`	Items to keep; defaults to all three sample items.	`None`
`years`	`int \| Iterable[int] \| None`	Years to keep; defaults to all.	`None`

Returns:

Type	Description
`DataFrame`	Columns `Reporter Country Code`, `Reporter Countries`, `Partner Country Code`, `Partner Countries`, `Item Code`, `Item`, `Element Code`, `Element`, `Year Code`, `Year`, `Unit`, `Value` and `Flag`, sorted by item, year, reporter, partner and element.

Raises:

Type	Description
`UnknownItemError`	If an item is not in the catalogue, or is in the catalogue but not in the sample.

Examples:

>>> from netviz_tools.datasets import faostat
>>> raw = faostat.raw_sample(items="Wheat", years=2022)
>>> sorted(raw["Element"].unique())
['Export quantity', 'Import quantity']

datasets.faostat¶

Reporter module-attribute ¶

Measure module-attribute ¶

ENV_CACHE_DIR module-attribute ¶

CITATION module-attribute ¶

LABELS module-attribute ¶

default_cache_dir ¶

store_path ¶

items ¶

countries ¶

download ¶

build_store ¶

store_info ¶

load ¶

load_sample ¶

raw_sample ¶

Reporter `module-attribute` ¶

Measure `module-attribute` ¶

ENV_CACHE_DIR `module-attribute` ¶

CITATION `module-attribute` ¶

LABELS `module-attribute` ¶