Standardized dataset ingestion

VOIAGE separates source parsing from calculation. A provider reads a descriptor and its explicitly declared local CSV resource, then returns an immutable, Arrow-backed NormalizedInputBundle. Calculation code only sees that bundle and an explicit VOI binding; it never depends on Croissant or Frictionless metadata.

The built-in profiles are deliberately conservative: explicit local CSV resources and explicit table schemas. Remote URLs, archive extraction, implicit column inference, and path traversal are rejected. This makes an input portable, auditable, and suitable for an offline calculation workflow.

voiage ingest validate datapackage.json
voiage ingest inspect croissant.json
voiage ingest normalize datapackage.json --output normalized.arrow

Every ingestion command accepts the same local source-policy controls. They never enable network access: use --max-resource-bytes to set a stricter resource cap, --cache-dir to select a verified materialization cache, and --offline to require replay rather than reading a source path. For example:

voiage ingest validate datapackage.json \
  --max-resource-bytes 10485760 \
  --cache-dir .voiage-cache

croissant.json is recognized from its Croissant context and datapackage.json from its Data Package resource list, not from the filename. validate resolves the descriptor and every declared local resource through the source-access policy, then returns a stable JSON confirmation. The inspect response contains the provider, its declared capability profile, safe descriptor provenance, preserved governance metadata, diagnostics, table row counts, schema fingerprint, and content digest; neither command echoes resource contents or credentials.

Local resources are fail-closed: paths cannot escape the configured descriptor root, network access is disabled by default, and a resource larger than the default 512 MiB policy limit is rejected before parsing. Applications can set a smaller explicit SourceAccessPolicy.max_resource_bytes limit for constrained workloads.

Verified local replay

SourceAccessPolicy is local-only by default. For a reviewable offline workflow, materialize an explicitly declared resource with its expected SHA-256 digest into a policy-scoped content-addressed cache. Later, an offline=True policy can replay only that verified digest; it never falls back to a changed or missing source file and never performs a network refresh.

The built-in providers use this path directly. A Croissant distribution may declare sha256; a Frictionless CSV resource may declare a 64-character SHA-256 hash and non-negative bytes. With those declarations, ingesting once with cache_dir creates the verified cache entry, and a later voiage ingest ... --offline --cache-dir .voiage-cache replays it even when the original local CSV is no longer present. Other hash algorithms and unverifiable integrity declarations are rejected.

from pathlib import Path

from voiage.ingestion import SourceAccessPolicy

policy = SourceAccessPolicy(Path("fixtures"), cache_dir=Path(".voiage-cache"))
resource = policy.materialize("samples.csv", sha256="<64-character-sha256>")

replay = SourceAccessPolicy(
    Path("fixtures"),
    cache_dir=Path(".voiage-cache"),
    offline=True,
)
resource = replay.materialize("samples.csv", sha256="<same-sha256>")

Use a stable, explicit cache_namespace when multiple trusted source-policy contexts share a cache directory. Cache entries are digest-checked on write and replay and are kept separate by that namespace.

Transport and performance assurance

Built-in providers do not contain a network transport. Every URI-style resource reference (including HTTP(S), FTP, file:, and SSH forms) is rejected before DNS lookup, redirect handling, download, cache refresh, or error rendering. Archive and transformation declarations are rejected before materialization. Consequently, DNS-rebinding, redirect, decompression, and mutable-remote-data risks are not supported behaviours; a future remote provider must introduce those capabilities behind a separately reviewed policy and receipt contract.

Run the repeatable local benchmark with:

uv run python scripts/benchmark_standardized_ingestion.py

It reports separate parse-to-Arrow, normalization, EVPI, and peak-memory measurements for the canonical Croissant and Frictionless fixtures. Results are release-review evidence rather than hard CI time budgets because hosted runners are intentionally not treated as a stable performance reference. A 64 MiB peak-memory ceiling for each small canonical fixture is enforced to catch materialization regressions without making runner-specific timing claims.

For a VOI calculation, supply a VOIBinding that states the table, fields, strategies, units, and perspective. The library will not guess semantics from field names. This keeps machine-learning, engineering, and business datasets on one preparation path while preserving the existing direct Python and CSV APIs.

Supported-standard matrix

VOIAGE intentionally supports a narrow, auditable profile rather than claiming complete implementation of either upstream standard.

Input family	Supported profile	Consumer contract	Explicitly not supported in the built-in provider
Croissant ML	Croissant 1.1 descriptor, one declared local CSV distribution, one recordSet, explicit fields, local SHA-256 when declared	Produces one Arrow-backed bundle; callers add explicit bindings before preparation.	Remote downloads, archives, transformations, multiple resources, executable metadata, implicit schema inference
Frictionless Data Package	Data Package v1-style descriptor, one or more local CSV resources with explicit field schemas, primary keys, and intra-package foreign keys; optional SHA-256 hash and bytes verification	Produces Arrow-backed tables and preserved key references; callers add explicit bindings before preparation.	Remote URLs, archives, transformations, other integrity algorithms, implicit schema inference
DataFrame interchange	A producer accepted by Arrow’s __dataframe__ interchange conversion, with explicit VOI bindings supplied by the caller	from_dataframe preserves the Arrow schema and honours allow_copy; its bundle enters the same preparation path as providers.	Inferred decision semantics, unsupported nested values, and conversions that require copying when allow_copy=False.

All providers default to local-only access. Inputs that fall outside these profiles fail with a stable IngestionError; they are not partially loaded or silently transformed. Compatibility will expand only with new fixtures, source-policy evidence, and a documented provider capability change.

Worked examples

The same contract supports more than one community-facing example:

• ML: a Croissant recordSet of posterior net-benefit samples for model choice.
• Engineering/operations: a Frictionless package recording alternative design outcomes and costs.
• Business: a Polars or pandas-compatible DataFrame passed through the dataframe-interchange adapter for competing investment strategies.

The adapter deliberately keeps the exchange boundary visible. Use from_dataframe(frame, dataset_id="business", allow_copy=False) when the caller requires Arrow to reject a conversion that would copy the source data; the resulting bundle always follows the same normalized preparation path. The supported SDK contract covers explicit bindings, source-neutral provenance, and Arrow schema preservation; it does not promise identical behaviour for every DataFrame library or unsupported nested dtype.

Each example must declare its binding rather than relying on a domain-specific calculation path, so their resulting net-benefit matrices are directly comparable with the existing VOI APIs.

Third-party providers

Third-party packages may implement IngestionProvider, but they remain outside the calculation runtime. A provider must expose a stable provider_id, a ProviderCapabilities value, can_handle(descriptor), and ingest(descriptor_path, policy=...). can_handle is a conservative, side-effect-free recognizer: it receives an already-read JSON object and must not resolve resources, access the network, or import optional parser stacks. ingest must return a NormalizedInputBundle and must resolve every declared resource through the supplied SourceAccessPolicy.

The frozen v1 SDK additionally requires capabilities.provider_id to equal the provider’s provider_id; registration rejects mismatches. Its versioned compatibility and publication rules live in specs/core-api/ingestion-provider-sdk-v1.md.

Publish an initialized provider instance under the voiage.ingestion.providers entry-point group. The entry-point name is the deployment identifier that applications add to their allow-list; it need not equal provider_id.

[project.entry-points."voiage.ingestion.providers"]
acme-catalog = "acme_voiage:provider"

Applications must opt in explicitly:

from voiage.ingestion import discover_entry_point_providers

providers = discover_entry_point_providers(allowlist={"acme-catalog"})

An empty allow-list does not inspect installed entry points. Entries not named in the allow-list are never loaded, so installing a package cannot change VOIAGE behaviour by itself. Discovery errors use IngestionError; providers should therefore avoid embedding credentials, URLs with secrets, or source contents in their exception messages.

Before publishing, provider authors should test: a valid supported descriptor, unsupported version/media-type failures, traversal and network refusal by SourceAccessPolicy, deterministic provenance and digests, and base-package import isolation. Providers must declare unsupported projection, filtering, streaming, and random-access features as False; no data transformation may be inferred or performed implicitly.