Skip to content

ADR-0014: Audio Studio generation runs in a background thread, not the editor's request

Status

Accepted

Date

2026-09-30

Context

The Audio Studio's original design settled on generation running inside the editor's web request: one ElevenLabs call per section, orchestrated by the browser, well inside the ALB's idle timeout. That held while every call was short. It stopped holding once v3 turned out to render at roughly 20ms per character (dev logs, 2026-09-25): a body over about 2,200 characters could not finish before the read timeout, no matter how the timeouts were tuned, because Cloudflare closes any request after 100s and the plan cannot raise that. PR #111 shipped the quick fix — ALB idle timeout 120s, provider read timeout 85s, a 4,000-character single-recording cap (about 80s) — which stopped most calls from failing, at the cost of forbidding the exact long sections (up to ElevenLabs' own 5,000-character limit) the studio exists to produce.

The deeper problem PR #111 did not fix: if the request is cut after ElevenLabs has already started rendering, the take can still succeed server-side, but the editor sees an error and a retry pays for a second call. The fix has to remove the web request as the thing generation's duration is bounded by, not tune the numbers again.

CSL-460 records the shape as settled and leaves the runner open: "the generate POST returns 202 with the take id, a background runner renders, and the browser polls the take... Engineering picks the runner: an in-process thread or a queue. There is no Celery today."

Decision Drivers

  • Must let a section of any length up to ElevenLabs' 5,000-character cap finish, with no timeout tied to the web request.
  • Must survive a closed browser tab or a page reload without losing a take that ElevenLabs already rendered and billed.
  • Must turn a run lost to a deploy or a crash into a failed take with a retry, not a take stuck in generating forever.
  • Must refuse a second generation on a section that already has one in flight.
  • Should cost no new deployed process or infrastructure, consistent with this project's standing decision to drop Celery and provision no persistent worker unless a real bulk workload appears — this is interactive, one-editor-at-a-time work, not a bulk job.
  • Should reuse an existing, reviewed pattern in this codebase rather than invent a new one.

Considered Options

Option A: In-process background thread

A threading.Thread spawned from the view after the take is claimed (row-locked, set to generating, committed), running the ElevenLabs call and the file/state writes independently of the request-response cycle. The view returns 202 immediately with the take's generating state; the browser polls a new status endpoint.

  • Pros: No new deployed process, no new IaC, no new dependency — django-tasks is already a transitive dependency (via Wagtail's search indexing) but is not configured with a durable backend or a worker process anywhere in this repo's IaC today, so reaching for it would mean building that infrastructure now, not reusing something already running. Granian already runs with 512 blocking threads per worker at defaults (measured previously), and the ElevenLabs call is I/O that releases the GIL, so one more thread per generation click is well inside headroom. The claim-lock pattern this needs (select_for_update, reject a second claim while generating and not stuck) is not new: actions/audio_longform.py::generate_segment already does exactly this for long-form chunks, reviewed across four PR #89 review rounds.
  • Cons: A container recycle (an ECS deploy, an OOM, a crash) kills the thread mid-render with no chance to finish or hand off — the take is genuinely lost, not merely delayed, if that happens during the render window. Nothing coordinates across ECS tasks if this service ever runs more than one; a poll can land on a different container than the one running the thread (harmless, since state lives in Postgres, but there is no way to migrate an in-flight thread). Django's per-thread DB connection must be closed explicitly (close_old_connections()) on every exit path, including exceptions, or connections leak over the life of the worker process. Every exception must be caught inside the thread, because there is no HTTP response left to carry it back once 202 has been returned.

Option B: django-tasks database backend + a deployed worker process

Configure django_tasks.backends.database.DatabaseBackend (already installed transitively) and run a continuous manage.py db_worker process to execute queued jobs.

  • Pros: A job survives the web process dying — a separate worker can pick it up. Built-in retry/backoff and a persisted job table give free observability a thread does not.
  • Cons: Needs a new, continuously-running ECS service/task definition, new IaC (compute-stack.ts, both buildspecs, secrets if any), and ongoing operational surface — the exact shape of infrastructure this project has already decided, twice, against provisioning for anything short of a real bulk workload. This is one editor generating one section at a time; there is no queue depth to manage and no throughput problem a worker process would solve that a thread does not already solve.

Decision

We will use Option A, an in-process background thread.

GenerateView/SegmentGenerateView claim the take under select_for_update (rejecting a second claim while a non-stuck generating take already exists for that section — the same pattern generate_segment already uses), commit, then spawn a threading.Thread that runs the ElevenLabs call and the file/state writes, closing its DB connection on every exit path. The view returns 202 immediately with the take's current JSON. The browser polls a new status endpoint until the take reaches ready or failed. take_is_stuck (already built, 5-minute window) is the safety net for a thread lost to a deploy or crash — this ADR leans on it for a case (a genuinely long-running render) it was not previously exercised by, which surfaced a real bug in it: staleness was measured from created_at (row creation), not from when the current claim actually started generating. A take drafted by prepare's auto-draft — which can sit for hours before an editor clicks Generate — read as stuck the instant it was claimed, defeating the in-flight guard the moment a real generation began. Fixed to measure from generation_meta["generating_at"] instead, matching audio_longform's existing correct handling for long-form segments.

This does not give the same guarantee a persisted queue would: a take whose thread is killed by a container recycle mid-render is lost, not resumed, and the editor pays for a retry — the same outcome CSL-460 is trying to shrink, just triggered by a rarer event (a deploy landing in a roughly 100-second window) instead of a near-certain one (any section over ~2,200 characters). Given this project's volume — one CMS, a handful of editors, interactive not bulk — that residual risk is accepted rather than paid for with a new deployed worker.

Consequences

Positive

  • A section up to ElevenLabs' 5,000-character cap can finish; ELEVENLABS_MAX_CHARS reverts to 5000.
  • No new deployed process, IaC, or dependency.
  • Reuses generate_segment's already-reviewed claim-lock pattern rather than inventing a second one.
  • The proxy-timeout tuning from PR #111 (ALB 120s, provider read timeout 85s) stops being load-bearing for the single-recording path; the provider read timeout can be raised to comfortably cover 5,000 characters without needing to fit inside Cloudflare's 100s.

Negative

  • A container recycle mid-render loses that take; the editor sees failed (after the 5-minute stuck window, or sooner once graceful-shutdown handling exists) and retries, paying for a second call.
  • Per-thread DB connection cleanup and defensive exception handling are manual disciplines this pattern requires and a queue would provide for free.
  • No built-in retry/backoff or job-level observability; both are the studio's existing Retry button and the take's error field, sized for interactive use, not bulk monitoring.

Risks & Mitigations

  • Risk: a deploy lands while a thread is mid-render, killing it silently until the 5-minute stuck window notices. → Mitigation: accepted at this volume; revisit if deploy frequency or section length grows enough to make this a frequent editor complaint rather than an occasional one.
  • Risk: an uncaught exception inside the thread leaves a take silently stuck with no message. → Mitigation: the entire thread body is wrapped in try/except that writes failed + an editor-readable message on any exception, belt-and-braces alongside take_is_stuck.
  • Risk: DB connections leak from background threads over the life of a long-running worker process. → Mitigation: close_old_connections() on every exit path (success, caught exception, and via a finally). Not covered by an automated test — the test suite runs generation synchronously (pytest-django blocks DB access from a real thread), so a genuine connection-growth test would need real threads and does not exist yet. The discipline is enforced by code review, not CI.
  • Risk: a real bulk-narration workload appears later (content-arrival day, a catalogue-wide re-narration) and a thread-per-click pattern does not scale to it. → Mitigation: this project has already settled the fallback for that case — a one-off ECS task, not a persistent worker — and this ADR does not foreclose that; it only decides the interactive, one-section-at-a-time path.
  • Risk: more generations in flight at once (more editors, or one editor's chapter-wide batch plus another's) is not bounded by anything this ADR adds — the per-browser-tab cap of 3 (AUDIO_STUDIO_MAX_IN_FLIGHT) is not a server-wide one, so two editors generating concurrently can still exceed ElevenLabs' actual workspace-wide limit (5 concurrent requests) and see a section fail with the provider's own concurrency message. → Mitigation: pre-existing, not introduced here — this gap ("the cap is per-browser... there is no shared throttle, and a retry clears it") was already recorded before this ADR. Thread count itself is not the constraint (granian's 512 threads/worker), ElevenLabs' cap is; a server-side shared semaphore would close this gap if it ever becomes a frequent complaint rather than an occasional retry.
  • Risk: reconcile() (the browser's fallback status check when a generate POST's own response is lost) can, on a multi-worker deployment, be served by a different process than the one still processing that POST, and read "nothing generating yet" a moment before the POST's claim commits. The browser then frees its concurrency slot early and may start a different section, briefly exceeding the per-tab cap of 3. → Mitigation: accepted as-is — the same-section claim lock still makes double-generating that section impossible, so the only consequence is transiently running a request or two above the self-imposed cap of 3, not exceeding ElevenLabs' actual limit of 5 in the common case. Closing this fully needs a client-generated idempotency token threaded through both the POST and every status check, which is disproportionate to a narrow, low-severity timing window; revisit alongside the per-tab-cap gap above if either becomes a frequent complaint.
  • actions/audio_longform.py::generate_segment — the claim-lock pattern this ADR reuses.
  • CSL-420 / PR #111 — the quick fix (timeouts, 4,000-character cap) this ADR supersedes.
  • CSL-367 — long-form chunk generation, which moves to the same runner alongside this change.