Book Processing → CMS Ingestion¶

How a Lewis book becomes reviewable Passages in the CMS. The import_book command turns an ingestion artifact into draft Passage snippets. There are two artifact formats: a reviewed JSONL export from the editor tool (one passage per line — the path for new books) and the older two-file YAML contract from the AI pipeline. See ADR-0007 for why it is shaped this way.

Pipeline at a glance¶

raw book (.txt/.md)
   │  ① segment        deterministic, no LLM — faithful boundaries + ranking
   ▼
candidate passages + manifest      (Passage-shaped, line anchors, signals)
   │  ② select         editorial / cheap-model pass — taste stays human
   │  ③ interpret      one AI subagent per passage (voice spec: _SYSTEM_PROMPT.md)
   │  ④ assemble       validate → transform to the two-file contract
   ▼
book.yaml  +  excerpts.yaml         ←── the ingestion contract (committed)
   │  ⑤ import_book    creates the Book + draft Passage snippets
   ▼
Wagtail review queue  →  editor QA  →  publish

Stages ①–④ run in the content-processing working area (the PM's raw drop: source text, segmentation, per-passage AI output). That working area is an audit trail archived separately — it is not committed to this repo. What the repo owns and supports is the stage-④ artifact (book.yaml + excerpts.yaml) and stage ⑤ (import_book).

Two input paths¶

import_book accepts either input for stages ①–④:

1. YAML contract — book.yaml + excerpts.yaml, described below. Produced by the original AI pipeline.
2. Reviewed JSONL export — one passage per line, straight from the editor tool, committed as content/ingestion/<slug>/passages.jsonl. This is the source of truth for books authored in the tool; there is no YAML step. Only passages the tool marks status: approved are imported. See JSONL export.

Both paths feed the same writer, produce the same passage slugs (<work-slug>--<chapter_id>-NN), and share the same rules: the Book must already exist, everything lands as a draft, and themes ride the draft revision.

The ingestion contract (two files)¶

Committed under content/ingestion/<work-slug>/. Fields are grouped by trust level — the whole point, because the importer treats each group differently.

book.yaml — the work, described once:

schema_version: 1
work:
  title: Mere Christianity
  slug: mere-christianity          # lowercase-with-dashes, unique per work
  kind: book
  author: C. S. Lewis
  publication_year: 1952
  edition: public-domain (Project Gutenberg / Distributed Proofreaders)
structure:                          # chapters in order
  - id: book1-ch01-the-law-of-human-nature
    book: 1
    chapter: 1
    title: The Law of Human Nature
    source_lines: [373, 520]

excerpts.yaml — one entry per Passage (the atom), referencing the work by slug:

schema_version: 1
work: mere-christianity             # must equal book.yaml → work.slug
excerpts:
  - locator:                        # provenance — makes faithfulness auditable
      chapter_id: book1-ch01-the-law-of-human-nature   # must exist in book.yaml → structure
      rank: 1
      source_lines: [427, 439]
    canon:                          # TRUSTED — Lewis's verbatim words, never AI-edited
      passage: "This law was called the Law of Nature because people thought…"
      self_contained: true
      word_count: 179
    interpretation:                 # AI-GENERATED — imported as draft, withheld until reviewed
      title: When We Say Someone Is Wrong
      opening: "Lewis reaches for the plainest example he can find…"
      body: "Notice the move he catches us in…"
      reflection_prompt: "The next time you feel someone is genuinely in the wrong…"
      context_note: "Written and first broadcast during the Second World War…"   # optional
      confidence: interpretive      # direct | interpretive | open
      source_anchors: ["Right is a real thing", "the colour of their hair"]
      generated: {model: claude-opus-4-8, generated_at: 2026-07-06T09:09:15Z}
    taxonomy:                       # SUGGESTIONS — preserved for the editor, never auto-applied
      suggested_themes: []
      human_truth: moral blame
      reader_moment: default
      sensitivities: "war; the Nazis invoked as a moral example"
    review:
      canon_status: draft
      interpretation_status: ai_generated

The two files join on slug: every excerpts.yaml → work must equal book.yaml → work.slug, and every excerpt chapter_id must exist in book.yaml → structure. import_book validates both before writing anything.

import_book never creates the Book itself — an editor creates the Book snippet in the CMS first (title, author, publication year, edition are edited-once facts, not something to infer from ingestion data). The only criterion the importer checks is that book.yaml → work.slug matches an existing Book; a missing Book fails validation before anything is written.

Field → CMS mapping¶

import_book writes data.Book (the Work / source) and data.Passage — both Wagtail snippets (ADR-0008); no content tree or prior setup needed.

Review state¶

Import always produces the safe state:

• Passages are created as drafts (live=False) with a draft revision. The Book (a reference object, not AI-written) is published.
• Because the mobile API serves only live content, the AI interpretation is withheld until an editor publishes.
• Editor QA in Wagtail admin: verify the quote against ingest_meta.locator.source_lines, confirm themes/tags, edit or approve the interpretation, then publish.

Draft/live is the interim gate. ADR-0006 (decision 2) plans a dedicated per-field review flag (ai_generated → reviewed) so canon can publish while an unreviewed interpretation stays withheld on the same piece; import_book will set it once it exists. Until then, provenance and the intended review state are kept in ingest_meta.

JSONL export¶

The editor tool exports one JSON object per line. Book-level metadata (title, author, publication year, edition) is not in the export — it lives on the Book snippet in the CMS, so the operator passes only --slug. Chapters are derived from the export. Fields with no CMS home (surrounding context, shelves, review notes) are preserved verbatim under ingest_meta.editor_export for audit.

Commit the .jsonl as it comes from the tool, at content/ingestion/<slug>/passages.jsonl. Re-export the whole book to update it; import is idempotent by slug, so the git diff shows exactly which passages changed. --jsonl also accepts a directory of *.jsonl files for per-batch exports.

Record structure¶

One line is one passage. The importer reads the fields below; the whole line is also preserved verbatim under Passage.ingest_meta.editor_export, so fields with no CMS home (context, shelves_final, source_anchors, score, …) stay available for audit.

{"id": "book1-ch01-…-r1", "work": "The Magician's Nephew",
 "chapter_id": "book1-ch01-one-the-wrong-door", "book": 1, "chapter": 1,
 "chapter_title": "One: The Wrong Door", "selection_rank": 1,
 "status": "approved", "themes_final": ["Temptation"],
 "passage": "…Lewis's verbatim words…",
 "interpretation_object": {"title": "…", "opening": "…", "interpretation": "…",
   "reflection": "…", "confidence": "interpretive", "source_anchors": ["…"]}}

(shown wrapped for readability — each record is a single line).

Importing¶

Where content lives decides where it can be imported:

Sample fixtures use sample- slugs and are excluded from every built image (.dockerignore drops src/tests), so they can never be imported on AWS. Prod is the editors' content-prep environment — only real content ever reaches it.

Flags, both modes: --dry-run validates and reports counts without writing; --update-existing overwrites AI-owned fields on passages that already exist (default: skip, so a re-run never clobbers an editor's work). Import is idempotent by slug — a second run reports every passage as skipped.

Local (developer machine)¶

Run against a checkout, so real content and the bundled samples both work. Create the Book snippet first (the importer links to it, never creates it):

# real JSONL export
python manage.py import_book --jsonl content/ingestion/<slug>/passages.jsonl --slug <slug>

# a bundled sample, to populate a local CMS
python manage.py import_book \
  --jsonl src/tests/fixtures/ingestion/the-magician-s-nephew/passages.jsonl \
  --slug  sample-the-magician-s-nephew

# YAML contract
python manage.py import_book \
  --book content/ingestion/<slug>/book.yaml \
  --excerpts content/ingestion/<slug>/excerpts.yaml

AWS dev and prod (one-off ECS task)¶

Real content is baked into the image at /app/content/ingestion/<slug>/. Run a one-off ECS task with a command override — the same mechanism the deploy pipeline uses for migrate (see .aws/codebuild/buildspec-*.yaml). The deploy's migrate task applies migrations, so no extra setup is needed before import.

Set the target environment, then run (add --dry-run first to validate):

# --- dev ---
CLUSTER=cslewis-app-dev-cluster;  TASKDEF=td-cs-lewis-backend-dev;  CONTAINER=cs-lewis-backend-dev

# --- prod (editors' content-prep environment) ---
# CLUSTER=cslewis-app-prod-cluster; TASKDEF=td-cs-lewis-backend-prod; CONTAINER=cs-lewis-backend-prod

cmd() { printf '{"containerOverrides":[{"name":"%s","command":%s,"cpu":256,"memory":512}]}' "$CONTAINER" "$1"; }

aws ecs run-task --cluster $CLUSTER --task-definition $TASKDEF --launch-type EC2 --count 1 \
  --overrides "$(cmd '["python","manage.py","import_book","--jsonl","/app/content/ingestion/<slug>/passages.jsonl","--slug","<slug>"]')" \
  --profile cslewis --region us-east-2

Wait for the task to stop and check its exit code / CloudWatch logs the same way the migrate step does.

Processing checklist (per book)¶

• The Book snippet exists in the CMS, with the slug book.yaml → work.slug will use.
• Source is the verified, owned edition; recorded in book.yaml → work.edition.
• book.yaml lists every chapter in order; excerpts.yaml → work matches the slug.
• Every canon.passage is verbatim within its source_lines.
• Selection favours human truth / image over transitional argument (quality over quota).
• Each interpretation passes the voice bar (no sermon, no self-help, open only at the leap).
• sensitivities recorded for contested/dated views (gender, war, hell, sexual morality).
• import_book --dry-run passes; then import (drafts).
• Editorial QA completed before publish.