Book Processing → CMS Ingestion¶

How a Lewis book becomes reviewable Passages in the CMS. The import_book command turns an ingestion artifact into draft Passage snippets. There are two artifact formats: a reviewed JSONL export from the editor tool (one passage per line — the path for new books) and the older two-file YAML contract from the AI pipeline. See ADR-0007 for why it is shaped this way.

Pipeline at a glance¶

raw book (.txt/.md)
   │  ① segment        deterministic, no LLM — faithful boundaries + ranking
   ▼
candidate passages + manifest      (Passage-shaped, line anchors, signals)
   │  ② select         editorial / cheap-model pass — taste stays human
   │  ③ interpret      one AI subagent per passage (voice spec: _SYSTEM_PROMPT.md)
   │  ④ assemble       validate → transform to the two-file contract
   ▼
book.yaml  +  excerpts.yaml         ←── the ingestion contract (committed)
   │  ⑤ import_book    creates the Book + draft Passage snippets
   ▼
Wagtail review queue  →  editor QA  →  publish

Stages ①–④ run in the content-processing working area (the PM's raw drop: source text, segmentation, per-passage AI output). That working area is an audit trail archived separately — it is not committed to this repo. What the repo owns and supports is the stage-④ artifact (book.yaml + excerpts.yaml) and stage ⑤ (import_book).

Two input paths¶

import_book accepts either input for stages ①–④:

1. YAML contract — book.yaml + excerpts.yaml, described below. Produced by the original AI pipeline.
2. Reviewed JSONL export — one passage per line, straight from the editor tool, committed as content/ingestion/<slug>/passages.jsonl. This is the source of truth for books authored in the tool; there is no YAML step. Only passages the tool marks status: approved are imported. See JSONL export.

Both paths feed the same writer: everything lands as a draft, and themes ride the draft revision. They differ in two places:

JSONL mode runs the same plan as the CMS Import Passages screen.

The ingestion contract (two files)¶

Committed under content/ingestion/<work-slug>/. Fields are grouped by trust level — the whole point, because the importer treats each group differently.

book.yaml — the work, described once:

schema_version: 1
work:
  title: Mere Christianity
  slug: mere-christianity          # lowercase-with-dashes, unique per work
  kind: book
  author: C. S. Lewis
  publication_year: 1952
  edition: public-domain (Project Gutenberg / Distributed Proofreaders)
structure:                          # chapters in order
  - id: book1-ch01-the-law-of-human-nature
    book: 1
    chapter: 1
    title: The Law of Human Nature
    source_lines: [373, 520]

excerpts.yaml — one entry per Passage (the atom), referencing the work by slug:

schema_version: 1
work: mere-christianity             # must equal book.yaml → work.slug
excerpts:
  - locator:                        # provenance — makes faithfulness auditable
      chapter_id: book1-ch01-the-law-of-human-nature   # must exist in book.yaml → structure
      rank: 1
      source_lines: [427, 439]
    canon:                          # TRUSTED — Lewis's verbatim words, never AI-edited
      passage: "This law was called the Law of Nature because people thought…"
      self_contained: true
      word_count: 179
    interpretation:                 # AI-GENERATED — imported as draft, withheld until reviewed
      title: When We Say Someone Is Wrong
      opening: "Lewis reaches for the plainest example he can find…"
      body: "Notice the move he catches us in…"
      reflection_prompt: "The next time you feel someone is genuinely in the wrong…"
      context_note: "Written and first broadcast during the Second World War…"   # optional
      confidence: interpretive      # direct | interpretive | open
      source_anchors: ["Right is a real thing", "the colour of their hair"]
      generated: {model: claude-opus-4-8, generated_at: 2026-07-06T09:09:15Z}
    taxonomy:                       # SUGGESTIONS — preserved for the editor, never auto-applied
      suggested_themes: []
      human_truth: moral blame
      reader_moment: default
      sensitivities: "war; the Nazis invoked as a moral example"
    review:
      canon_status: draft
      interpretation_status: ai_generated

The two files join on slug: every excerpts.yaml → work must equal book.yaml → work.slug, and every excerpt chapter_id must exist in book.yaml → structure. import_book validates both before writing anything.

In YAML mode import_book never creates the Book itself. An editor creates the Book snippet in the CMS first (title, author, publication year, edition are edited-once facts, not something to infer from ingestion data). The only criterion the importer checks is that book.yaml → work.slug matches an existing Book; a missing Book fails validation before anything is written. JSONL mode creates missing Books instead; see JSONL export.

Field → CMS mapping¶

import_book writes data.Book (the Work / source) and data.Passage — both Wagtail snippets (ADR-0008); no content tree or prior setup needed.

Emphasis in source text¶

Both sides of the contract mark emphasis with Markdown characters: a public-domain book text keeps the underscore italics of its source edition (it happens to pay _me_ personally), and the AI interpretation writes *emphasis*. import_book converts those runs into real marks — <i> and <b>, the tags the Wagtail editor itself writes — through data.richtext.marked_text_to_richtext.

Store them as characters instead and the app shows the markup: the API projects rich text to Markdown, so an unmarked _me_ is escaped to \_me\_ and a reader sees the delimiters (CSL-427).

Only emphasis is read. Headings, lists, links and code are left as typed, because the ingestion text carries none of them and a full Markdown parser would reinterpret Lewis's punctuation. A run has to open and close on a word boundary and start on a non-space, so snake_case, 2*3*4 and a Gutenberg section break (* * * * *) stay literal. A backslash-escaped delimiter (\_, \*) stays literal too.

Passages imported before this conversion existed keep the literal characters: re-import the book with --update-existing, then publish the revision.

Small-caps chapter openings¶

A print edition sets the first words of a chapter in small caps, and the extracted text keeps them as capitals: ONCE THERE WERE FOUR CHILDREN whose names were…. In the editor tool, the reviewer ticks the capitalize_first display transform on such a passage. The export records it as display_transforms, both top-level and under review. import_book then re-cases that leading capitals run and leaves the rest of the passage untouched:

• The run ends at the first word that has a lowercase letter.
• A word in the run keeps a capital when it starts a sentence, is I or an abbreviation such as Mr., or is a name that the book writes capitalised.
• A word counts as a name when the export's own passages capitalise it mid-sentence more often than they write it lowercase. SAID LUCY becomes said Lucy. A run of capitalised words mid-sentence counts as a phrase, so THE WHITE WITCH becomes the White Witch.
• ?” and !” do not start a sentence. The word after them is dialogue attribution (“WAS IT ALL A DREAM?” WONDERED SHASTA becomes “Was it all a dream?” wondered Shasta).

No letters are added or removed, only their case. The passage as exported stays in ingest_meta.editor_export. An unknown display transform fails the import, so a new option in the tool never imports untransformed. The re-casing is a heuristic, so the editor checks the opening when reviewing the draft.

Passages imported before this transform existed keep the capitals. The CMS locks body, so an editor cannot fix them there. A developer saves a re-cased draft revision instead, which changes only body and keeps other edits. Do not use --update-existing for this, because it overwrites an editor's changes to the AI-owned fields. The five affected passages in prod were fixed this way (CSL-456).

Review state¶

Import always produces the safe state:

• Passages are created as drafts (live=False) with a draft revision. The Book (a reference object, not AI-written) is published.
• Because the mobile API serves only live content, the AI interpretation is withheld until an editor publishes.
• Editor QA in Wagtail admin: verify the quote against ingest_meta.locator.source_lines, confirm themes/tags, edit or approve the interpretation, then publish.

Draft/live is the interim gate. ADR-0006 (decision 2) plans a dedicated per-field review flag (ai_generated → reviewed) so canon can publish while an unreviewed interpretation stays withheld on the same piece; import_book will set it once it exists. Until then, provenance and the intended review state are kept in ingest_meta.

JSONL export¶

The editor tool exports one JSON object per line. Each record's work title is matched to a Book (case, spacing and curly or straight apostrophes are ignored). A Book that doesn't exist yet is created first, as a title-only draft: the export has no cover, year or synopsis, so an editor fills those in and publishes it. The API still shows a draft Book as a passage's source; only Featured Books waits for it to be published. --slug sets the new Book's slug (one book per file); without it the slug comes from the title. Fields with no CMS home (surrounding context, shelves, review notes) are preserved verbatim under ingest_meta.editor_export for audit.

Commit the .jsonl as it comes from the tool, at content/ingestion/<slug>/passages.jsonl. Re-export the whole book to update it; a passage already in the CMS is skipped, so the git diff shows exactly which passages changed. "Already in the CMS" means the same slug, or the same quote in the same Book, which also catches passages imported under the older <slug>--<chapter_id>-NN slugs. A skipped passage is never updated; edit it in the CMS. --jsonl also accepts a directory of *.jsonl files for per-batch exports.

Record structure¶

One line is one passage. The importer reads the fields below; the whole line is also preserved verbatim under Passage.ingest_meta.editor_export, so fields with no CMS home (context, shelves_final, source_anchors, score, …) stay available for audit.

{"id": "book1-ch01-…-r1", "work": "The Magician's Nephew",
 "chapter_id": "book1-ch01-one-the-wrong-door", "book": 1, "chapter": 1,
 "chapter_title": "One: The Wrong Door", "selection_rank": 1,
 "status": "approved", "themes_final": ["Temptation"],
 "passage": "…Lewis's verbatim words…",
 "interpretation_object": {"title": "…", "opening": "…", "interpretation": "…",
   "reflection": "…", "confidence": "interpretive", "source_anchors": ["…"]}}

(shown wrapped for readability — each record is a single line).

Importing¶

Where content lives decides where it can be imported:

Sample fixtures use sample- slugs and are excluded from every built image (.dockerignore drops src/tests), so they can never be imported on AWS. Prod is the editors' content-prep environment — only real content ever reaches it.

Flags: --dry-run validates and reports counts without writing anything, Books included. --update-existing (YAML mode only) overwrites AI-owned fields on passages that already exist (default: skip, so a re-run never clobbers an editor's work). A second run reports every passage as skipped.

Local (developer machine)¶

Run against a checkout, so real content and the bundled samples both work. JSONL mode creates a missing Book; YAML mode needs it created in the CMS first:

# real JSONL export
python manage.py import_book --jsonl content/ingestion/<slug>/passages.jsonl --slug <slug>

# a bundled sample, to populate a local CMS
python manage.py import_book \
  --jsonl src/tests/fixtures/ingestion/the-magician-s-nephew/passages.jsonl \
  --slug  sample-the-magician-s-nephew

# YAML contract
python manage.py import_book \
  --book content/ingestion/<slug>/book.yaml \
  --excerpts content/ingestion/<slug>/excerpts.yaml

AWS dev and prod (one-off ECS task)¶

Real content is baked into the image at /app/content/ingestion/<slug>/. Run a one-off ECS task with a command override — the same mechanism the deploy pipeline uses for migrate (see .aws/codebuild/buildspec-*.yaml). The deploy's migrate task applies migrations, so no extra setup is needed before import.

Set the target environment, then run (add --dry-run first to validate):

# --- dev ---
CLUSTER=cslewis-app-dev-cluster;  TASKDEF=td-cs-lewis-backend-dev;  CONTAINER=cs-lewis-backend-dev

# --- prod (editors' content-prep environment) ---
# CLUSTER=cslewis-app-prod-cluster; TASKDEF=td-cs-lewis-backend-prod; CONTAINER=cs-lewis-backend-prod

cmd() { printf '{"containerOverrides":[{"name":"%s","command":%s,"cpu":256,"memory":512}]}' "$CONTAINER" "$1"; }

aws ecs run-task --cluster $CLUSTER --task-definition $TASKDEF --launch-type EC2 --count 1 \
  --overrides "$(cmd '["python","manage.py","import_book","--jsonl","/app/content/ingestion/<slug>/passages.jsonl","--slug","<slug>"]')" \
  --profile cslewis --region us-east-2

Wait for the task to stop and check its exit code / CloudWatch logs the same way the migrate step does.

Processing checklist (per book)¶

• YAML mode: the Book snippet exists in the CMS, with the slug book.yaml → work.slug will use. JSONL mode: an imported Book is a draft; add its cover, year and synopsis, then publish it.
• Source is the verified, owned edition; recorded in book.yaml → work.edition.
• book.yaml lists every chapter in order; excerpts.yaml → work matches the slug.
• Every canon.passage is verbatim within its source_lines.
• Selection favours human truth / image over transitional argument (quality over quota).
• Each interpretation passes the voice bar (no sermon, no self-help, open only at the leap).
• sensitivities recorded for contested/dated views (gender, war, hell, sexual morality).
• import_book --dry-run passes; then import (drafts).
• Editorial QA completed before publish.