Audio Studio operations¶
How the Audio Studio runs in a deployed environment, how to watch its usage, and how to recover when generation goes wrong. This page is for engineers and operators. Editors use the editor guide ยง Narration.
What runs where¶
| Part | Where it runs | Notes |
|---|---|---|
| Studio screens | The CMS, in the web task | One studio per Passage, Writing, Article and Journey |
| Performance cues | Amazon Bedrock, with a Bedrock API key (AWS_BEARER_TOKEN_BEDROCK) |
One call when an editor opens a studio. It never generates paid audio |
| Narration | ElevenLabs, from a background thread in the web task | Only when an editor clicks Generate. There is no worker; the browser polls the take |
| Audio files | S3 cslewis-uploads-<env>, served through CloudFront |
Unsigned URLs |
| Clean-up | prune_audio_takes, daily at 04:00 UTC |
One-off ECS task from EventBridge Scheduler |
The studio needs ELEVENLABS_API_KEY, and cue drafting needs AWS_BEARER_TOKEN_BEDROCK, both in
the cslewis-app-<env> secret. Changing either value needs no task-definition change; the next
task to start picks it up.
Settings¶
| Setting | Dev | Prod | What it controls |
|---|---|---|---|
ELEVENLABS_MAX_CHARS |
5,000 | 5,000 | The longest script one generate request sends |
AUDIO_LONGFORM_ENABLED |
on | off | The backend endpoints that narrate a longer body in chunks. The studio screen does not call them yet |
AUDIO_STUDIO_MAX_IN_FLIGHT |
3 | 3 | Generations one browser runs at once. Set below the account's ElevenLabs concurrency limit (5 when it was chosen) to leave room for a second editor. The limit depends on the ElevenLabs plan: check it there before raising this |
AUDIO_TAKE_RETENTION_DAYS |
7 | 7 | Age at which prune_audio_takes deletes an unapproved take |
Usage and quota¶
- The studio shows the share of the month's ElevenLabs credits left once it drops under 20%. The figure is cached for 10 minutes.
- ElevenLabs bills from the
character-costresponse header, about 0.55 credits per character oneleven_v3. Counting the script's characters overstates the cost about twofold. - The full figures are on the ElevenLabs account's subscription page.
What gets deleted, and what never does¶
- An approved recording lives on
AudioAsset. Nothing on this page deletes it. - Approving a section deletes that section's other takes.
prune_audio_takesdeletes every other take older than 7 days, and any take stuck ingenerating(see below). An editor who leaves a section unapproved for more than 7 days loses its takes and any edited script, and starts that section again.
Recovery¶
A take stays on "Generating"¶
A take is stuck when it has been generating for more than 5 minutes: the thread that owned it
died, usually because the task restarted mid-generation. The editor can click Generate again on that card; the new run takes the claim over.
To clear stuck takes without waiting for 04:00 UTC, run the command as a one-off task:
# --- dev ---
CLUSTER=cslewis-app-dev-cluster; TASKDEF=td-cs-lewis-backend-dev; CONTAINER=cs-lewis-backend-dev
# --- prod ---
# CLUSTER=cslewis-app-prod-cluster; TASKDEF=td-cs-lewis-backend-prod; CONTAINER=cs-lewis-backend-prod
aws ecs run-task --cluster $CLUSTER --task-definition $TASKDEF --launch-type EC2 --count 1 \
--overrides "{\"containerOverrides\":[{\"name\":\"$CONTAINER\",\"command\":[\"python\",\"manage.py\",\"prune_audio_takes\"],\"cpu\":256,\"memory\":512}]}" \
--profile cslewis --region us-east-2
A card shows "Generation failed"¶
The card shows ElevenLabs' own message behind Retry. Nothing retries on its own. A failed card can still have used credits: ElevenLabs may finish the audio and the save then fail, or the response may be lost. Before retrying a failure with no clear cause, check the account's usage page.
| Message | Cause | Fix |
|---|---|---|
| "The ElevenLabs API key is not set up correctly" | The key was rejected | Check the key in cslewis-app-<env> |
| "did not finish within 240 seconds" | A long section ran out of time | Retry. If one section keeps failing, shorten it or upload a recording |
| "Could not connect to ElevenLabs" | Network or ElevenLabs outage | Retry later; check status.elevenlabs.io |
| A quota message from ElevenLabs | The month's credits are spent | Raise the ElevenLabs plan, or wait for the reset |
Generate is disabled on every card¶
The environment has no ELEVENLABS_API_KEY. The page says so in a banner. Add the key to
cslewis-app-<env>, then restart the service or deploy so a new task reads it.
Scripts open without performance cues¶
Bedrock was unavailable or refused the call. The banner says "Performance cues were not drafted",
and each script is the plain text. Editors can still generate. Check AWS_BEARER_TOKEN_BEDROCK in
cslewis-app-<env>, and that its AWS account can use BEDROCK_MODEL_ID in BEDROCK_REGION. After
changing the key, restart the service so a new task reads it.
Fixing the key only helps scripts drafted from then on. The plain-text scripts already drafted are saved and reused when the studio opens again, so they stay without cues. Editors can add cues to them with Insert cue on each card. The studio drafts those sections again, with cues, only once their unapproved drafts are cleaned up after 7 days.
A body is too long to generate¶
Over 5,000 characters, Generate is disabled on that card and the card asks the editor to upload a human recording instead. The backend has long-form endpoints that narrate a longer body in chunks (on in dev, off in prod), but the studio screen does not use them yet.
Approved audio does not play in the app¶
The app only gets audio that narrates the live words. Check these in order:
- The card says "not live yet": the audio was approved against an edit that is not published. Publish the edit, or use "Save audio and publish".
- The text changed after approval: the asset is stale and the API sends
nullfor that track. Generate and approve the section again. - The piece itself is not live.
Logs¶
The studio logs ids and error details only: take and segment ids, and the exception class and message of a failed call. It never logs the narrated text, the script or the key, so routine logs can be shared without exposing licensed text.