Runbook: Secrets & Credential Rotation

---

1. Scope of Credentials & Rotation Cadence

Credential Name	Criticality	Deployment Surfaces	Rotation Cadence
PAYSTACK_SECRET_KEY	CRITICAL	Fly.io Backend, Supabase Edge Functions	90 Days / Immediate on Compromise
SUPABASE_SERVICE_ROLE_KEY	CRITICAL	Fly.io Backend, CI/CD Workflows	180 Days / Immediate on Compromise
SUPABASE_JWT_SECRET	CRITICAL	Fly.io Backend API, Supabase Auth Engine	Annual / Immediate on Compromise
DATABASE_URL	CRITICAL	Fly.io Backend API	180 Days / DB Failover
GEMINI_API_KEY	HIGH	Fly.io Backend API (Nduzi AI)	90 Days
CLOUDFLARE_API_TOKEN	HIGH	GitHub Actions CI/CD, DNS Automation	Annual

---

2. Zero-Downtime Rotation Lifecycle (Dual-Key Window)

For live payment keys and database credentials, zero-downtime rotation requires a staged overlapping dual-key transition:

sequenceDiagram
    autonumber
    actor DevOps as Platform Engineer
    participant Provider as Credential Provider (Paystack / Supabase / GCP)
    participant Compute as Fly.io Compute Cluster
    participant Edge as Edge Functions & Storefront
    participant Sentry as Telemetry & Error Tracking

    DevOps->>Provider: 1. Generate Secondary Secret Key (Keep Primary Active)
    DevOps->>Compute: 2. fly secrets set SECRET_KEY="<NEW_KEY>"
    Note over Compute: Fly.io triggers rolling deployment across VM machines
    DevOps->>Edge: 3. Update Supabase Edge Function Secrets
    DevOps->>Compute: 4. Execute Canary Smoke Probes (npm run smoke:payments)
    DevOps->>Sentry: 5. Monitor 401/403 Error Rates for 30 minutes
    alt Zero Authentication Exceptions
        DevOps->>Provider: 6. Deactivate / Revoke Legacy Primary Key
        DevOps->>DevOps: 7. Record Completion in Security Audit Ledger
    else Auth Exceptions Detected
        DevOps->>Compute: 6b. Roll back secrets to Legacy Primary Key
        DevOps->>DevOps: 7b. Declare Incident & Abort Rotation
    end

---

3. Step-by-Step Procedures by Credential

Procedure A: Rotating PAYSTACK_SECRET_KEY

1. Generate Secondary Key: Log into the Paystack Dashboard $\to$ Settings $\to$ API Keys. Generate a new Secret Key without revoking the current active key.
2. Update Backend Machine Secrets:

   fly secrets set PAYSTACK_SECRET_KEY="sk_live_new_..." -a debelu-backend

3. Update Supabase Edge Functions:

   supabase secrets set PAYSTACK_SECRET_KEY="sk_live_new_..."

4. Execute Canary Verification:

   # Run verification probe testing payment intent and webhook verification
   npm run check:payments

5. Revoke Old Key: In Paystack Dashboard, deactivate the legacy key after 30 minutes of clean Sentry telemetry.

---

Procedure B: Rotating SUPABASE_SERVICE_ROLE_KEY

Caution: The service role key bypasses Row-Level Security (RLS). Ensure only trusted administrators execute this procedure.

1. Generate in Supabase Dashboard: Project Settings $\to$ API $\to$ Generate new service role secret.
2. Deploy to Fly.io:

   fly secrets set SUPABASE_SERVICE_ROLE_KEY="eyJhbGciOi..." -a debelu-backend

3. Deploy to GitHub Actions Secrets:

   gh secret set SUPABASE_SERVICE_ROLE_KEY -b"eyJhbGciOi..."

4. Smoke Test Backend RPCs:

   curl -I https://api.debelu.com/health/readiness

5. Revoke Previous Key: Remove legacy service role key in Supabase console.

---

Procedure C: Rotating GEMINI_API_KEY

1. Generate in Google AI Studio: Navigate to API Keys $\to$ Create API Key in project.
2. Update Machine Secret:

   fly secrets set GEMINI_API_KEY="AIzaSy..." -a debelu-backend

3. Test Nduzi AI Assistant:

   curl -X POST https://api.debelu.com/api/gemini/chat \
     -H "Content-Type: application/json" \
     -d '{"message": "Hello Nduzi, test probe"}'

---

4. Emergency Compromise Protocol (Immediate Revocation)

Trigger: A live secret key is inadvertently committed to a public Git repository, leaked in a client-side bundle, or captured in a compromised third-party service.

stateDiagram-v2
    [*] --> Declared : Leak Detected (SEV-1 Incident)
    Declared --> InstantRevocation : Immediate Hard Revocation in Provider Console (T < 5m)
    InstantRevocation --> EmergencyDeploy : Deploy New Keys to Production (T < 15m)
    EmergencyDeploy --> AuditInvestigation : Query Provider Access Logs for Abuse (T < 2h)
    AuditInvestigation --> Disclosure : Complete Blameless Post-Mortem & Security Disclosure
    Disclosure --> [*]

Emergency Action Checklist

1. Do NOT wait for dual-key rolling deployment: Instantly revoke the compromised key in the provider console (Paystack, Supabase, Google Cloud). Brief downtime is accepted over catastrophic financial or data exfiltration.
2. Fast-Deploy Newly Minted Key: Deploy replacement credentials immediately using CLI flags (--stage).
3. Audit Exposure Window:
   • For Paystack: Query transfer logs and checkout events created during the exposure window to identify unauthorized disbursements.
   • For Supabase: Query PostgreSQL query logs for unauthorized service_role table queries or bulk data dumps.
4. Legal & Security Disclosure: File an incident report with the DPO and publish an internal Root Cause Analysis (RCA) within 48 hours.

---

5. Verification Checklist & Compliance Sign-Off

Upon completing any credential rotation, the engineer must verify:

• [ ] Health readiness probe (/health/readiness) returns HTTP 200 OK across all active nodes.
• [ ] Zero 401 Unauthorized spikes in Sentry over a 30-minute observation window.
• [ ] Zero failed background queue jobs in BullMQ.
• [ ] Legacy key tested and confirmed inactive (curl returns 401).
• [ ] Rotation date and operator ID logged in internal security ledger.