Harden prod deploy: versioned secrets, healthchecks, migration lock, dry-run

Swarm stack
- Resource limits on all services, stop_grace_period 60s on api/worker/admin
- Dozzle bound to manager loopback only (ssh -L required for access)
- Worker health server on :6060, admin /api/health endpoint
- Redis 200M LRU cap, B2/S3 env vars wired through to api service

Deploy script
- DRY_RUN=1 prints plan + exits
- Auto-rollback on failed healthcheck, docker logout at end
- Versioned-secret pruning keeps last SECRET_KEEP_VERSIONS (default 3)
- PUSH_LATEST_TAG default flipped to false
- B2 all-or-none validation before deploy

Code
- cmd/api takes pg_advisory_lock on a dedicated connection before
  AutoMigrate, serialising boot-time migrations across replicas
- cmd/worker exposes an HTTP /health endpoint with graceful shutdown

Docs
- deploy/DEPLOYING.md: step-by-step walkthrough for a real deploy
- deploy/shit_deploy_cant_do.md: manual prerequisites + recurring ops
- deploy/README.md updated with storage toggle, worker-replica caveat,
  multi-arch recipe, connection-pool tuning, renumbered sections

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Trey t
2026-04-14 15:22:43 -05:00
parent ca818e8478
commit 33eee812b6
11 changed files with 908 additions and 30 deletions

View File

@@ -50,6 +50,27 @@ STORAGE_BASE_URL=/uploads
STORAGE_MAX_FILE_SIZE=10485760
STORAGE_ALLOWED_TYPES=image/jpeg,image/png,image/gif,image/webp,application/pdf
# Storage backend (S3-compatible: Backblaze B2 or MinIO)
#
# Leave all B2_* vars empty to use the local filesystem at STORAGE_UPLOAD_DIR.
# - Safe for single-node setups (dev / single-VPS prod).
# - NOT SAFE for multi-replica prod: named volumes are per-node in Swarm,
# so uploads written on one node are invisible to the other replicas.
#
# Set ALL FOUR of B2_ENDPOINT, B2_KEY_ID, B2_APP_KEY, B2_BUCKET_NAME to
# switch to S3-compatible storage. The deploy script enforces all-or-none.
#
# Example for Backblaze B2 (us-west-004):
# B2_ENDPOINT=s3.us-west-004.backblazeb2.com
# B2_USE_SSL=true
# B2_REGION=us-west-004
B2_ENDPOINT=
B2_KEY_ID=
B2_APP_KEY=
B2_BUCKET_NAME=
B2_USE_SSL=true
B2_REGION=us-east-1
# Feature flags
FEATURE_PUSH_ENABLED=true
FEATURE_EMAIL_ENABLED=true