Debelu production operations

Prepared 30 September 2026. The Debelu owner is the initial operator; appoint a backup operator before relying on continuous incident coverage. This document records procedures, not evidence that a live financial or restore rehearsal ran.

Service map and access

Service	Production destination	Control panel
Marketing	https://www.debelu.com	Vercel debelu-marketing
Storefront	https://app.debelu.com	Cloudflare Pages debelu-storefront
Admin	https://admin.debelu.com	Cloudflare Pages debelu-admin
API	https://api.debelu.com	Railway discerning-exploration → debelu-backend → production
Database, Auth, notifications	project havugmqmyplqgbrlthhs	Supabase
Product images	https://cdn.debelu.com	R2 debelu-product-images
Outbound email	London eu-west-2	Amazon SES
Inbound mail	Zoho	Zoho Mail

Store credentials in provider secret stores. Enable an authenticator on the staff account in Admin → Settings → Security before requiring it for all staff. Never put customer records, tokens, bank information or webhook bodies in a public issue or release record.

WhatsApp callbacks and support

The Meta app is published and its messages field is subscribed. Signed callbacks are acknowledged only after storage succeeds; failures return 503 for Meta retry. Admin → Support → WhatsApp replies & delivery receipts → Load latest shows the latest 100 entries to a platform admin, protected by staff MFA and no-store responses. Callback text/receipts are retained for 30 days. Do not paste them into public issues. The inbox is read-only; it does not send replies or download media.

STOP, UNSUBSCRIBE, CANCEL, END and QUIT suppress future optional notifications to that number and disable matching account WhatsApp preferences. Suppressions do not expire with callback retention. A profile toggle cannot silently override a STOP: do not remove a suppression without fresh recorded consent and an approved support procedure. Delivery receipts match notification_deliveries through provider_message_id; an API send success is not proof of handset delivery.

Migration 27 and deliver-notification version 12 are deployed. Both WhatsApp retention and notification/log cleanup completed successfully on 1 October 2026.

Daily payment reconciliation

1. In Paystack Live mode, compare successful charges and transfers with Debelu Admin → Finance transactions and payouts. Match provider references, amount, currency, final state and the associated order or wallet transaction.
2. Review pending payments, processing transfers, failed/refunded charges and disputes. A Paystack charge is not proof that the correct order was finalized; confirm the matching Debelu order and ledger result.
3. In Paystack → Developers → Webhooks, inspect failed deliveries. The live URL must be https://api.debelu.com/api/payments/webhook.
4. For a failed delivery, fix the receiving service first, then resend the original provider event. Do not invent a charge or directly edit balances. The event handlers are idempotent; verify the order and ledger change once.
5. Inspect Railway /health/ready and Sentry for payment queue alerts. A webhook returning 200 can mean it was queued, so also investigate worker failures.
6. Record unresolved references, responsible operator and follow-up time in a private operations record. Escalate any discrepancy before another payout.

Payouts and refunds

In Admin → Finance, use Send to initiate the claimed Paystack transfer. Confirm the destination and amount before sending. Processing transfers settle through signed provider webhooks; verify the final provider state and Debelu record before treating a seller as paid. For a timeout or uncertain response, look up the existing transfer reference before initiating another transfer.

Mark as paid is for an offline transfer the operator has already completed and independently confirmed. It records the payment; it does not transfer money. For a rejected request, confirm the reversal in the seller's wallet/ledger. For a refund, use the implemented admin/dispute action and reconcile the provider refund state and Debelu ledger; do not bypass transaction functions with SQL.

Failed notifications and email health

Review the Admin Notifications delivery summary and Supabase Edge Function logs for deliver-notification. A delivery recorded as sent means the provider accepted it; check SES bounce/complaint health and suppression information for actual delivery failures. Verify Auth email separately from notification email.

WhatsApp requires an approved template, a valid production token, and explicit member opt-in. Members can turn it off in Profile → Notifications. Leave Meta's messages webhook field unsubscribed until inbound replies and receipts are persisted. Do not manually reset a delivery claim to resend an optional message.

Incident and application rollback

1. Record the incident time, release commit and affected service. Inspect Sentry, the relevant hosting logs, API readiness and provider status.
2. If marketplace money or account integrity is at risk, the owner can turn on maintenance mode in Admin Settings. Preserve evidence and provider references.
3. For an application regression, restore the last known-good production deployment in that service's hosting panel. Check configuration compatibility with the current database before rollback. Do not roll the marketing site back to a Next.js version affected by GHSA-vcvr-r3jv-pc5j.
4. For a DNS incident, compare the explicit affected record in Cloudflare with the DNS migration record. Correct a known incorrect target. A nameserver rollback involves caching and should be an owner-reviewed recovery decision.
5. After recovery, verify readiness, authentication and the affected user flow; reconcile any payments made during the incident and record the outcome.

Database and object recovery

The production backup API reported zero available backups and no PITR on 30 September. A tested database restore is still a launch prerequisite. Choose paid automatic backups or a secured logical backup route, then verify recovery in an isolated project before relying on it. Never test restoration on the live project. Restoring production requires an explicit recovery decision and downtime.

Supabase database backups do not contain Storage object bytes. Keep the existing Supabase product-image bucket for historical URLs and separately protect objects in Supabase Storage and R2. Do not delete either bucket as a migration cleanup.

Release evidence and maintenance

Keep the successful Pages workflow URL, commit SHA, migration numbers, provider deployment IDs and the matching sbom-<sha> CI artifact in the release record. CI retains SBOM artifacts for 90 days; copy them to the organization's private long-term release archive before expiry.

Production launch-hardening migrations 01–24 were owner-applied; 25 and 26 were applied individually through the Management API. Do not replay them or use supabase db push until the hosted migration history has been reconciled. The deletion, unpaid-checkout and auto-completion schedules were active with successful recent runs. Migration 26 fixes the old nightly notification/log retention job; inspect its next scheduled result before closing that incident.