Operations, retention and recovery

Health

/healthz checks the API process. /readyz checks PostgreSQL and Redis. Object-store failure degrades attachment/report/snapshot operations; monitor MinIO separately. Workers log scheduler, evaluator, retention and delivery failures to stdout/stderr.

Observe probe last-seen timestamps, due-job age, pending/failed deliveries, PostgreSQL storage/partition growth and Redis memory. Redis is configured with no eviction. Snapshot age is visible on the public page and is not a substitute for an external monitor of this system.

Retention

Free keeps raw checks/metrics seven days and daily/hourly aggregates 90 days. Business retention is configured by platform admin. Raw tables use monthly PostgreSQL partitions; the worker prepares adjacent months and deletes tenant-expired samples. Empty historical partitions can be removed during database maintenance after confirming no retained rows remain.

Backups

Use your platform’s encrypted backup target and retention policy. Example local logical backup:

docker compose exec -T postgres pg_dump -U postgres -Fc status > status.dump

Back up MinIO objects, versioning configuration and encryption keys separately. Redis persistence improves queue/cache recovery, but PostgreSQL is the authoritative source for jobs, outbox events and delivery attempts. Never include .env or raw production credentials in source control or diagnostic bundles.

Restore drill

Provision a separate PostgreSQL database and private MinIO bucket. Restore the dump with pg_restore --no-owner, restore bucket objects and secrets, then run the migrator. Start API/workers against the restored dependencies and check tenant isolation, published history, attachments and snapshot timestamps before changing traffic.

The scripts/backup-restore.sh script performs a disposable local PostgreSQL restore drill without replacing the live database. Production recovery still needs a separate object-store restore test and recovery-point review.

Failure behavior

When Redis restarts, workers rescan database pending rows. When a worker dies, expired job/delivery leases are reclaimed. If the public endpoint cannot verify the page in PostgreSQL, it bypasses Redis and reads the published MinIO snapshot. The response is always marked stale: true; its original updated_at, published branding and any suspension notice are preserved. Once the database recovers, the endpoint resumes serving live data.

Fallback has a separate seven-second budget after the live database/cache budget (up to five seconds). It makes at most two complete object-read attempts, each limited to three seconds, with a short delay between attempts. The storage SDK may retry HTTP exchanges within each attempt. Interrupted bodies, network timeouts and temporary storage errors can be retried; missing/forbidden objects and invalid snapshots fail immediately. The API validates the complete JSON, page slug, timestamp, status and required collections, with a 64 MiB size limit. It returns 503 if no valid snapshot can be read; the UI retains its last loaded data with an error.

Look for public snapshot read failed in API logs: error_kind, storage_code, storage_status, attempt, elapsed_ms and retry identify the failure without recording credentials or response bodies. public snapshot read recovered records a successful second attempt. Monitor these failures and snapshot age separately from /readyz. A concurrent image build is not proof of the cause of a storage error; check actual storage/network diagnostics and host capacity.

Capacity validation

Use the included benchmark command and load profile. Synthetic benchmark observations do not prove real Internet endpoint performance or a 24-hour production soak. Record machine resources, image digests, throughput, scheduling lag and API percentiles. Do not advertise a 10,000-monitor SLA before a representative 24-hour run completes.