Deployment¶

Infrastructure is AWS CDK (TypeScript) under .aws/iac; the application ships as a Docker image built by CodeBuild and run on EC2-backed ECS. A merge to main builds the image, pushes it to ECR, registers a new task definition, and rolls the ECS service. See ADR-0002 for why CDK + EC2 ECS + Aurora Serverless v2.

Environments¶

Environment	URL	Trigger
Dev (AWS)	`https://dev-cslewis.fueled.engineering`	Push to `main` touching `src/`, `.aws/`, or `pyproject.toml`
Production	`https://app.cslewis.com` (pre-prod / staging until launch)	Push of a `v..*` tag

Both environments are live over HTTPS. Dev and prod each run all four stacks with the real Django app; /-/health/ returns {"healthy": true} (Aurora + Redis wired) and HTTP 301-redirects to HTTPS. Prod (app.cslewis.com, Cloudflare zone cslewis.com) went live 2026-08-25 at tag v0.1.0 and is used as a content-prep / pre-prod environment until public launch.

AWS account¶

	Value
Account ID	`987292390939`
Region	`us-east-2`
CLI profile	`cslewis` (all `just` recipes default to it)

The account resolves automatically from the active profile — CDK reads CDK_DEFAULT_ACCOUNT, so nothing hardcodes the ID in the stacks. Set export AWS_PROFILE=cslewis (or pass profile=<name> to just) before running anything below.

Prerequisites (One-Time Setup)¶

Done once per AWS account. Run CDK commands from .aws/iac (npm install first). The just recipes wrap the CDK CLI and default to the cslewis AWS profile.

Bootstrap CDK (once per account/region):
```
just bootstrap
```
Deploy the shared stack (ECR repo + GitHub OIDC provider/role — account-wide, no env):
```
just deploy shared
```
Push the seed image so the first compute deploy comes up healthy before the real image exists (the ECS service points at :seed, an nginx placeholder that returns 200 on /-/alive/):
```
just seed-image
```
Request an ACM public certificate (us-east-2) for the env hostname, DNS-validated in the Cloudflare zone. Add the ACM validation CNAME as DNS-only (grey cloud) — if it's proxied, validation never completes. Once the cert is Issued, put its ARN in .aws/iac/config/<env>/compute.yaml → acm_ssl_cert and redeploy compute to add the :443 listener + an 80→443 redirect. While acm_ssl_cert is empty the ALB is HTTP-only so infra can be validated first.
Cloudflare:
Create a proxied (orange-cloud) DNS record for the app hostname → the ALB DNS name.
Set SSL/TLS mode to Full (strict), but scope it to the hostname with a Configuration Rule (Rules → Configuration Rules: Hostname equals <host> → SSL → Full (strict)), not the zone default. The encryption mode is zone-wide by default, so flipping fueled.engineering itself to strict can break other subdomains whose origins don't serve a valid public cert (Cloudflare 525/526). Conversely, leaving the origin on the zone's Flexible default makes Cloudflare talk HTTP to the ALB, hit the 80→443 redirect, and loop → 503. The per-hostname rule avoids both.
The ALB security group already restricts ingress to Cloudflare's published IP ranges (restrict_alb_to_cloudflare: true; opened on both 80 and 443).
For prod, add Authenticated Origin Pulls (mTLS) for stronger origin lockdown.
Let the RevenueCat webhook bypass bot rules. RevenueCat posts to POST /api/payments/revenuecat/webhook/ from its own servers with a non-browser client. Cloudflare Bot Fight Mode or the Browser Integrity Check can block that client with error 1010 (HTTP 403); the RevenueCat dashboard then shows the delivery as failed. A WAF managed rule can also return a 403, but as a different block, not error 1010. The endpoint authenticates itself with a shared Authorization token, plus an HMAC-SHA256 signature when REVENUECAT_WEBHOOK_HMAC_SECRET is set, so it does not need Cloudflare's bot heuristics. Configure it as follows:
- Add a WAF custom rule (Security → WAF → Custom rules). Always match the host, because sibling subdomains share the zone. Prod: (http.host eq "app.cslewis.com" and http.request.uri.path eq "/api/payments/revenuecat/webhook/"). Dev: (http.host eq "dev-cslewis.fueled.engineering" and http.request.uri.path eq "/api/payments/revenuecat/webhook/"). Adding and http.request.method eq "POST" narrows it further but is not required.
- Set the action to Skip, and skip All Super Bot Fight Mode Rules and Browser Integrity Check. These two produce the error 1010 block on RevenueCat's non-browser client. If a WAF managed rule blocks the path instead (a 403 that is not error 1010, visible in the Security events log), add the managed ruleset to the Skip components too; the endpoint self-authenticates, so skipping it on this one path is safe. Leave rate limiting unticked so the safety-net rule below stays active.
- As code (Cloudflare Rulesets API), the same rule is one entry in the zone's http_request_firewall_custom ruleset: action: "skip" with action_parameters: {"phases": ["http_request_sbfm"], "products": ["bic"]}, where http_request_sbfm is Super Bot Fight Mode and bic is Browser Integrity Check. Create it with POST /zones/{zone_id}/rulesets/{ruleset_id}/rules; the dashboard shows the exact call under the rule's API view.
- The free Bot Fight Mode is account-wide and cannot be scoped by a Skip rule. Super Bot Fight Mode, which a Skip rule can scope, needs a paid plan (Pro or higher). If the zone is on the free plan, either upgrade to a plan with Super Bot Fight Mode or turn Bot Fight Mode off for the zone. Check the current mode under Security → Bots.
- Keep a rate-limiting rule scoped to the same path (for example, over 100 requests per minute per IP) so skipping bot checks does not open a flood path. RevenueCat's real volume is low. Cloudflare L3/L4 DDoS protection stays on regardless.
- Do not rely on an IP allowlist: RevenueCat does not publish stable webhook source IPs. The path plus the endpoint's own token and HMAC checks are the durable control.
- Apply the rule to both zones, dev fueled.engineering and prod cslewis.com, and confirm it on prod before launch.
- Verify after setup: click Send test event on the RevenueCat dashboard webhook page and confirm a 2xx in the delivery log. A backend SubscriptionEvent row carrying the RevenueCat event id confirms the full path reached the app.
- Backstop: the daily reconciliation job (planned, CSL-355 Slice 2) re-syncs subscribers with active or recently expired entitlements from RevenueCat's current state, so a missed webhook self-heals even if a Cloudflare rule regresses. Build it before prod launch.
GitHub → AWS: set the repository variable AWS_ACCOUNT_ID to 987292390939 (used by the workflows to assume github-actions-cs-lewis-backend via OIDC). Create dev and prod GitHub Environments.

Connect CodeBuild to GitHub (CodeConnections) — aws-codebuild-run-build overrides the build source to the GitHub repo, so the account needs a GitHub source credential (this step is separate from the OIDC role above):

# a. create the connection (returns an ARN; status PENDING)
aws codeconnections create-connection --provider-type GitHub --connection-name cslewis-github
# b. complete the handshake in the console: Developer Tools → Settings → Connections →
#    the pending connection → "Update pending connection" → authorize "AWS Connector for
#    GitHub" and install it on the Fueled org WITH access to the cs-lewis-backend repo
# c. register it as CodeBuild's account/region-wide GitHub credential
aws codebuild import-source-credentials --server-type GITHUB \
  --auth-type CODECONNECTIONS --token <connectionArn>

The compute stack grants the CodeBuild role codeconnections:UseConnection. If the AWS Connector GitHub App lacks repo access, the build fails at DOWNLOAD_SOURCE with "repository not found" — fix in GitHub → org Settings → GitHub Apps → AWS Connector for GitHub → Repository access (org owner). A missing credential instead fails with "authentication required".

Populate secrets (see Configuration & Secrets).

Infrastructure Deployment¶

Stacks deploy in dependency order: shared → network → compute → storage. Deploying an env with no stack argument does all three env stacks in order.

just deploy dev network      # VPC
just deploy dev compute      # ECS cluster, ALB, ASG, CodeBuild, IAM, SSM params
just deploy dev storage      # Aurora Serverless v2, Redis, S3 uploads bucket
# or all at once:
just deploy dev

Inspect changes before applying with just diff dev <stack> or render templates with just synth dev <stack>.

After first compute + storage deploy, wire the runtime secret (below) so the real app can boot, then push code to trigger the app deploy.

Application Deployment (CI/CD)¶

Flow: GitHub Actions → OIDC role → CodeBuild → ECR → ECS.

GitHub Actions (.github/workflows/dev-aws-backend.yml) assumes the OIDC role and starts the CodeBuild project cslewis-app-<env> with buildspec-<env>.yaml.
CodeBuild builds .aws/docker/Dockerfile-aws, tags the image with the commit SHA (and latest), and pushes to ECR.
It renders .aws/codebuild/td-cs-lewis-backend.json with envsubst (via render-task-defs.sh), registers a new task-definition revision, and calls ecs update-service --force-new-deployment.
ECS rolls the service (circuit breaker on — a failed rollout auto-rolls-back).

Dev deploys on every qualifying push to main; prod deploys on a v*.*.* tag.

Task-def ownership split (gotcha). CDK creates the initial task definition, but CodeBuild overrides it on every deploy from td-cs-lewis-backend.json. Environment-variable and secret changes must go in that JSON template (and, for new secrets, the CDK secrets map + the render script whitelist) — editing only the CDK stack will not affect running tasks.

Post-deploy data seeding¶

The pipeline runs migrate automatically before each rollout, but content data is not seeded by the pipeline — a freshly deployed environment has an empty CMS. Populate it by running the management commands as one-off ECS tasks (the same run-task mechanism as the migrate step). All are idempotent.

The container runs from WORKDIR /app/src, so pass absolute content paths (/app/content/...); a repo-relative path such as content/ingestion/... resolves against /app/src and fails with "directory not found".

sync_themes — the controlled Theme vocabulary (run first; book imports reference it). Real data; run in every environment.
seed_tags — sample/bootstrap Tag data. Do not run in prod: the Tag vocabulary is editorial and built by editors in the CMS. Treat it like seed (dev-only sample data).
import_book … — a processed book's Book + draft Passages. See the Book Processing guide for the exact run-task invocation.
import_articles --posts /app/content/ingestion/substack/posts — the Substack editorial snapshot as draft Articles (only the 28 client-licensed posts, per content/ingestion/substack/licensed-slugs.txt; the snapshot is committed under content/ingestion/, so it's in the image). Same run-task mechanism.

seed (sample dev users) is a local-dev convenience — its users are DEBUG-gated, so it is effectively a no-op in AWS and isn't run there. Likewise, seed_tags produces sample tags and is not run in prod (see above).

No sample data in prod. Only real content goes into the prod CMS: the controlled Themes, licensed Articles, and processed Books. Placeholder/sample seeders (seed, seed_tags, seed_writings, seed_related) are dev-only.

Configuration & Secrets¶

Non-secret runtime config is set as plain env vars in .aws/codebuild/td-cs-lewis-backend.json (DJANGO_SETTINGS_MODULE, SITE_DOMAIN, DJANGO_DEBUG, DJANGO_LOG_LEVEL, SENTRY_ENVIRONMENT, AWS_S3_UPLOADS_BUCKET, FIREBASE_PROJECT_ID, FIREBASE_WEB_API_KEY, DEV_AUTH_ENABLED). The Firebase values are public identifiers (they ship in the mobile app binary); DEV_AUTH_ENABLED is True on dev, False on prod — it gates the dev-login endpoint that mints bearer tokens without Firebase.
Secrets live in one AWS Secrets Manager secret per env, cslewis-app-<env>, as a JSON blob. The CDK compute stack creates it with empty placeholders on first deploy; populate the real values afterward:
```
aws secretsmanager put-secret-value --secret-id cslewis-app-dev --secret-string '{
  "DATABASE_URL": "postgres://USER:PASS@HOST:5432/cslewisdb",
  "REDIS_URL": "redis://HOST:6379/0",
  "DJANGO_SECRET_KEY": "...",
  "SENTRY_DSN": "...",
  "ELEVENLABS_API_KEY": "...",
  "REVENUECAT_API_KEY": "...",
  "REVENUECAT_PROJECT_ID": "proj7d44db36",
  "REVENUECAT_WEBHOOK_TOKEN": "...",
  "REVENUECAT_WEBHOOK_HMAC_SECRET": ""
}' --profile cslewis
```
put-secret-value replaces the whole JSON, so include every existing key. When you add a new key (as with the REVENUECAT_* keys above), seed it in Secrets Manager before the deploy that references it — the CDK stack only writes empty placeholders on first create, so an existing secret will not gain the key on its own and the ECS rollout fails resolving a missing valueFrom key. The task definition references all four REVENUECAT_* keys unconditionally, so each must exist: REVENUECAT_API_KEY and REVENUECAT_PROJECT_ID drive the v2 current-state sync, and REVENUECAT_WEBHOOK_HMAC_SECRET must exist even when HMAC signing is off; leave it "" to disable verification (the webhook then checks only the Authorization token). DATABASE_URL / REDIS_URL come from the storage stack (Aurora credentials secret rds-credentials-<env> + the Aurora/Redis endpoints; internal hostnames are also published in the private Route53 zone). The task definition injects each key as an env var via valueFrom.
SSM parameters under /cslewis/<env>/ (aws_account_id, secret_arn, task_role_arn, execution_role_arn, cluster_name, …) are written by the compute stack and read by the buildspec — no manual setup.

Rollback¶

Fast path: the ECS deployment circuit breaker auto-rolls-back a rollout whose tasks never become healthy.

Manual: re-point the service at a previous task-definition revision:

aws ecs update-service --cluster cslewis-app-dev-cluster --service cs-lewis-backend \
  --task-definition td-cs-lewis-backend-dev:<PREVIOUS_REVISION> --force-new-deployment --profile cslewis

Or re-run the deploy for a known-good commit SHA. Images are retained in ECR (last 10).

Infrastructure: just diff before any re-deploy; CDK/CloudFormation rolls a failed stack update back automatically. Data stores use RemovalPolicy.RETAIN (prod Aurora also has deletion protection) so a stack rollback never drops data.
Failed compute create (first bring-up): if the ECS rollout circuit breaker trips during the initial create, the stack can land in ROLLBACK_FAILED (the ASG capacity provider is "in use"). To reset: aws cloudformation delete-stack, then — because the EC2 host keeps ECS managed scale-in protection and stalls the ASG teardown — clear it with aws autoscaling set-instance-protection --auto-scaling-group-name cslewis-app-<env>-asg --instance-ids <id> --no-protected-from-scale-in. After delete, remove the two RETAINed, fixed-name leftovers or the redeploy hits "already exists": the log group cslewis-app-<env> and the bucket cslewis-app-alb-logs-<env>-<account>.

Post-Deploy Verification¶

# Liveness (no DB) and readiness (DB + cache) — through Cloudflare:
curl -fsS https://dev-cslewis.fueled.engineering/-/alive/      # -> ok
curl -fsS https://dev-cslewis.fueled.engineering/-/health/     # -> {"healthy": true}

Also confirm: the ECS service shows the new task-def revision and a steady running count; the ALB target group is healthy; CloudWatch logs (cslewis-app-<env>) show granian serving without boot errors; the Wagtail and Django admin login pages load at their configured prefixes (WAGTAIL_ADMIN_URL / DJANGO_ADMIN_URL; defaults studio/ and backstage/, not /cms/ and /admin/).