Docs — Sable Foundry — Light
docs / getting-started / quickstart

Quickstart: load a corpus

Every Sable release is a self-describing handover package. The loading test below is our release criterion — if a sample doesn't load into Python without manual editing, the release is blocked.

1 · Verify the manifest

The root manifest.csv lists every file with its path, size, SHA-256 checksum, version and sensitivity level.

shellcopy
$ sha256sum -c manifest.csv --quiet && echo "all files verified"
all files verified

2 · Load annotations and audio

load_corpus.py — included in every releasecopy
import pandas as pd, librosa

utts = pd.read_json("processed/utterances.jsonl", lines=True)
pairs = pd.read_json("processed/translation_pairs.jsonl", lines=True)

# one channel per speaker — overlap preserved
wav, sr = librosa.load(utts.audio_file[0], sr=None)

assert (utts.consent_ref.notna()).all()
assert (utts.qa_status == "pass").all()

3 · Package structure

sable_shona_duplex_v1.0.0/
├── README.md
├── manifest.csv              # path · size · SHA-256 · version · sensitivity
├── metadata/                 # dataset + subset metadata cards (JSON)
├── schema/                   # data_dictionary.csv · schema.json
├── processed/                # release WAV audio + JSONL annotations
├── labels/                   # annotation_guide (versioned) · taxonomy
├── validation/               # quality_report · validation_rules.json · error_log.csv
└── governance/               # consent template (sn/nd/en) · licence · access policy

4 · Utterance schema

One JSONL record per utterance. Full reference in schema/data_dictionary.csv.

FIELDTYPEREQDESCRIPTION / ALLOWED VALUES
{{ f.name }} {{ f.type }} {{ f.req }} {{ f.desc }}

5 · Metadata card

Every subset ships a machine-readable card — title, provenance, licence, coverage, consent status, quality summary, and a named steward.

{
  "title": "Sable Shona Full-Duplex Conversational Corpus v1.0.0",
  "source": "Sable Foundry contributor network (first-party)",
  "language_tags": ["sn", "en"],
  "license": "Commercial / open subset CC BY 4.0",
  "data_sensitivity": "personal, pseudonymised",
  "consent_status": "obtained, per-purpose, revocable",
  "bias_and_coverage_notes": "gaps disclosed in bias register",
  "version": "v1.0.0",
  "contact_or_steward": "Tendai Gatahwa"
}
NEXT Browse the dataset catalog → REFERENCE Licence terms & governance →