Operations

This is the on-call runbook. It covers how to deploy, monitor, respond to incidents, and recover from failures. When something breaks, jump to Incident Response and Common Issues & Fixes.

Deployment

Normal Deployment Flow

Work flows in one direction: dev → test → pprd → prod.

Merge PR to dev

CI runs lint + typecheck + unit tests.

Merge dev → test

CI adds integration tests + Playwright + Lighthouse.

Merge test → pprd

CI adds k6 load test + OWASP ZAP.

Merge pprd → prod

Requires manual approval, then auto-deploys.

Emergency Hotfix

For critical production bugs that can’t wait for the full pipeline:

# Create hotfix branch from prod
git checkout prod
git checkout -b hotfix/booking-crash

# Make the fix, commit, push
git push origin hotfix/booking-crash

# Create PR directly to prod (requires manual approval)
# After merge, backport to dev:
git checkout dev
git merge prod

Rollback

AWS web & admin
Render CMS
Database

Customer and admin apps run on AWS Lambda behind CloudFront through sst.aws.Nextjs:

Open GitHub Actions → Deploy AWS → Run workflow
Set git_ref to the last known-good tag or SHA
Run the workflow and verify both /api/health endpoints

A failed SST/Pulumi update can leave resources partially changed. Inspect the deployment logs and stack state, verify both /api/health endpoints, and redeploy the last known-good ref if production is unhealthy. A successful but bad release takes about 3–5 minutes to replace; use a PostHog flag first when it can contain the issue.

Payload CMS (rgss-cms, cms.theroyalglow.in):

Go to Render → Service → Deploys
Click the last known-good deploy → “Rollback”

Monitoring

Daily Checks (2 minutes)

Status page

Check BetterStack status page — all monitors green?

Errors

Check Sentry — any new error issues overnight?

Jobs

Check BetterStack heartbeats — did all scheduled jobs run?

Weekly Checks (15 minutes)

Funnel

Review PostHog funnel — any drop-off increase in booking flow?

Error trends

Review Sentry error trends — any new patterns?

Performance

Check Lighthouse CI scores on recent PRs — any performance regression?

Load

Review k6 load test results — any latency increase?

Alerts

Alert	Source	Action
Site down	BetterStack	Check CloudFront status and Lambda CloudWatch logs; verify Cloudflare DNS
API health failing	BetterStack	Check Lambda CloudWatch logs and Neon DB connection
Scheduled job missed	BetterStack heartbeat	Check QStash dashboard for delivery logs
New error spike	Sentry	Check error details, deploy fix or roll back
Slow response (P95 > 3s)	Sentry performance	Check DB query times, check Neon metrics

Incident Response

Severity Levels

Level	Definition	Response Time	Example
P0	Site completely down	Immediate	CloudFront/Lambda unavailable, Cloudflare DNS outage, or Neon DB unreachable
P1	Core feature broken	< 1 hour	Booking creation failing, auth broken
P2	Feature degraded	< 4 hours	Emails not sending, PDF generation failing
P3	Minor issue	Next business day	Cosmetic bug, non-critical feature broken

P0/P1 Response Steps

Acknowledge

Post in Slack #incidents that you’re investigating.

Diagnose

Check BetterStack and Sentry first, then the affected provider: CloudFront/Lambda in CloudWatch, Cloudflare DNS, Render CMS, or Neon.

Communicate

Update the BetterStack status page with an incident message.

Fix or rollback

Deploy a fix or roll back to the last known-good deployment.

Verify

Confirm Checkly synthetic checks pass.

Post-mortem

Document what happened, why, and how to prevent it.

Common Issues & Fixes

Booking creation failing (500)

Check Sentry for the error — usually a DB connection issue or validation bug
Check Neon DB metrics — is the connection pool exhausted?
Check if a recent migration broke a constraint

Emails not sending

Check Resend dashboard — is the API key valid? Any rate limits?
Check Sentry for UPSTREAM_ERROR from the email service
Emails are non-blocking — bookings still succeed, emails can be retried

PDF generation failing

Check Cloud Run logs for the PDF render service (@rgss/invoicing)
PDF failures are non-blocking — invoice is created in DB, PDF can be regenerated
Manually trigger PDF regeneration via admin panel

Ably realtime not working

Check Ably dashboard — is the app within message limits?
Realtime is non-critical — the app works without it (polling fallback)

QStash job not running

Check QStash dashboard — is the job scheduled? Any delivery failures?
Check BetterStack heartbeat — did the job ping on last run?
Manually trigger the job via the QStash dashboard

Database Operations

Running Migrations

# Generate migration from schema changes
cd packages/db
bunx drizzle-kit generate

# Apply to dev branch
DATABASE_URL=$DATABASE_URL_DEV bunx drizzle-kit migrate

# Apply to test branch
DATABASE_URL=$DATABASE_URL_TEST bunx drizzle-kit migrate

# Apply to prod (requires manual confirmation)
DATABASE_URL=$DATABASE_URL_PROD bunx drizzle-kit migrate

Never run migrations directly on prod without testing on test + pprd first.

Neon Branch Management

# Create a new branch for a feature (via Neon console or API)
# Branches are instant copy-on-write — no data copying needed

# Reset pprd from prod (normally done by GitHub Actions nightly)
curl -X POST https://console.neon.tech/api/v2/projects/$NEON_PROJECT_ID/branches/$PPRD_BRANCH_ID/restore \
  -H "Authorization: Bearer $NEON_API_KEY" \
  -d '{"source_branch_id": "$PROD_BRANCH_ID"}'

Point-in-Time Restore

If a bad migration or data corruption occurs on prod:

Open the branch

Go to Neon console → Project → Branches → prod.

Restore to a point in time

Click “Restore” → select a point in time before the incident.

Verify

This creates a new branch — verify data is correct.

Swap the connection

Point the connection string to the restored branch.

Re-apply migrations

Apply any migrations that happened after the restore point.

Backup & Recovery

Weekly Backups

Every Sunday at 2 AM UTC, GitHub Actions:

Runs pg_dump against Neon prod branch
Uploads compressed dump to Cloudflare R2 (backups/weekly/)
Retains 8 weeks of backups
Pings BetterStack heartbeat on success

Restore from Backup

# Download backup from R2
aws s3 cp s3://theroyalglow-uploads/backups/weekly/backup-2026-06-01.sql.gz . \
  --endpoint-url https://$R2_ACCOUNT_ID.r2.cloudflarestorage.com

# Decompress
gunzip backup-2026-06-01.sql.gz

# Restore to a new Neon branch (never restore directly to prod)
psql $DATABASE_URL_RESTORE < backup-2026-06-01.sql

Scaling

The platform is designed to scale without code changes:

Bottleneck	Solution
Lambda cold starts or duration	Profile CloudWatch duration, collapse sequential DB calls, then cache hot reads in Upstash
CloudFront/static delivery	Inspect cache behaviour and asset headers before changing compute
DB connection pool exhausted	Increase Neon compute size or connection limit
Upstash Redis rate limit	Upgrade Upstash plan
Ably message limit	Upgrade Ably plan (6M/mo free is generous)
Resend email limit	Upgrade Resend plan

Production Verification Checklist

Last modified on September 1, 2026

Getting started

System design

Standards

Application surface

Platform

Quality & security

Delivery workflow

Operations

Deployment

Normal Deployment Flow

Emergency Hotfix

Rollback

Monitoring

Daily Checks (2 minutes)

Weekly Checks (15 minutes)

Alerts

Incident Response

Severity Levels

P0/P1 Response Steps

Common Issues & Fixes

Database Operations

Running Migrations

Neon Branch Management

Point-in-Time Restore

Backup & Recovery

Weekly Backups

Restore from Backup

Scaling

Production Verification Checklist

​Deployment

​Normal Deployment Flow

​Emergency Hotfix

​Rollback

​Monitoring

​Daily Checks (2 minutes)

​Weekly Checks (15 minutes)

​Alerts

​Incident Response

​Severity Levels

​P0/P1 Response Steps

​Common Issues & Fixes

​Database Operations

​Running Migrations

​Neon Branch Management

​Point-in-Time Restore

​Backup & Recovery

​Weekly Backups

​Restore from Backup

​Scaling

​Production Verification Checklist

Deployment

Normal Deployment Flow

Emergency Hotfix

Rollback

Monitoring

Daily Checks (2 minutes)

Weekly Checks (15 minutes)

Alerts

Incident Response

Severity Levels

P0/P1 Response Steps

Common Issues & Fixes

Database Operations

Running Migrations

Neon Branch Management

Point-in-Time Restore

Backup & Recovery

Weekly Backups

Restore from Backup

Scaling

Production Verification Checklist