Skip to content

ADR-0007: Book Ingestion Pipeline — two-file contract, importer, and provenance

Status

Accepted — implements the Content Pipeline (CSL-22) referenced by ADR-0006. Scoped to the first book ingestion (Mere Christianity); revisit when the ADR-0006 review-state fields and the unified Chapter kind land.

Date

2026-07-09

Context

A book is processed off-repo into passages, each with an AI-written interpretation (deterministic extraction → editorial selection → per-passage AI pass). The result must land in the Wagtail CMS as reviewable content without (a) altering Lewis's words or (b) letting AI-generated text reach the app unreviewed — the constraints ADR-0006 records.

We need a stable, committed contract between that off-repo processing and the CMS, plus an importer, ahead of the review-state schema and the unified readable Chapter model (both proposed in ADR-0006 but not yet built).

Decision Drivers

  • Must: canon passages stored verbatim; nothing AI-written served unreviewed.
  • Must: import is idempotent and never clobbers editorial work on re-run.
  • Should: full provenance preserved (source lines, model, confidence, taxonomy suggestions) so any quote is auditable and re-runnable.
  • Should: human-reviewable input; work metadata not repeated per excerpt.
  • Nice-to-have: no new model surface beyond what's strictly needed now.

Considered Options

Contract format

  • A. One JSONL per work (record = excerpt, work repeated per line). Machine- friendly, but repeats work metadata 264× and reads poorly for editorial QA.
  • B. Two YAML files — book.yaml + excerpts.yaml. Work described once; excerpts reference it by slug; YAML block scalars keep prose readable.

Chosen: B. JSONL remains the intermediate assembly form upstream; the committed contract is the YAML pair.

Where provenance lives

  • A. Drop it on import (lossy — a beautiful interpretation of a misattributed quote is worse than none; loses the audit trail).
  • B. Dedicated columns for each field (large surface; premature before the ADR-0006 review model settles).
  • C. One ingest_meta JSONField on Passage holding the full source record.

Chosen: C. Preserves everything for audit/editor review in one low-risk field; internal (not in api_fields).

Withholding unreviewed AI content

  • A. Wait for the ADR-0006 per-field review flag before importing anything.
  • B. Import as Wagtail draft (live=False) now; the API serves only live content, so drafts are withheld. Adopt the per-field flag when it lands.

Chosen: B as the interim gate; the intended review state is carried in ingest_meta meanwhile.

Decision

  1. Contract: content/ingestion/<work-slug>/{book.yaml, excerpts.yaml}, grouped by trust (canon / interpretation / taxonomy / locator / review). Joins on slug + chapter_id. Documented in the book-processing guide.
  2. import_book management command (interfaces/cli) validates the pair, creates the Book (published) and one draft Passage snippet per passage, matched by a deterministic slug (<chapter_id>-NN). Idempotent: existing excerpts are skipped unless --update-existing. --dry-run validates only. (Snippets, not pages, since ADR-0008.)
  3. Passage.ingest_meta (JSONField, migration 0004) stores the full source record — provenance, AI-generation metadata, and taxonomy suggestions.
  4. Canon verbatim: canon.passage → body wrapped in paragraphs only, no rewording.
  5. Themes are auto-applied from taxonomy.suggested_themes onto the draft revision (an editor confirms before publish) — but link-only against a global controlled vocabulary created once by sync_themes. The importer never creates a Theme; a suggestion outside the vocabulary is skipped and flagged. Tags stay an editorial act. Every suggestion is preserved in ingest_meta regardless.
  6. The off-repo processing working area (raw text, segmentation, per-passage AI output) is archived separately, not committed; the repo owns the stage-④ artifact and the importer.

Consequences

Positive

  • Clean, auditable boundary; any imported quote traces back to exact source lines.
  • Re-import is safe and predictable; editors' work is never overwritten by default.
  • No dependence on the (still-proposed) review-state schema to start ingesting.
  • Minimal new model surface: one JSONField.

Negative

  • interpretation.title/opening/body compose into one flat interpretation RichText (the model isn't structured yet) — lossy for structure, though the originals stay in ingest_meta.
  • Draft/live is coarser than the intended per-field gate: a whole Passage is withheld until published, so canon can't yet publish independently of an unreviewed interpretation.
  • Provenance in a JSON blob isn't queryable like columns.

Risks & Mitigations

  • Risk: re-run clobbers editorial edits. → Skip existing by default; overwrite only behind --update-existing.
  • Risk: import_book diverges from the real data schema. → A test imports the committed Mere Christianity files via --dry-run as a schema-drift guard.
  • Risk: ingest_meta leaks to the API. → Excluded from api_fields; it is internal provenance only.
  • ADR-0006 (content model, review state, serialization) — this implements its Content Pipeline (decision 2).
  • ADR-0004 / ADR-0005.
  • Tickets: CSL-22 (Content Pipeline), CSL-71 (review state), CSL-52 (PR #10).