Replaces the previous hand-rolled MigrateWithLock + GORM AutoMigrate path,
which had two compounding problems:
- AutoMigrate ran on every pod startup (~5 min over the transatlantic
link) even when no schema changes had landed
- pg_advisory_lock is session-scoped, which silently fails through
Neon's pgbouncer transaction-mode pooler — turns out this is a
known and documented limitation that bites golang-migrate too
Goose was chosen over golang-migrate (the other heavyweight) because:
- Goose wraps each migration file in a transaction by default, so a
failure rolls back cleanly instead of leaving a "dirty" version
state requiring manual force-reset (golang-migrate's known
weakness, per its own issue tracker — see #1001 + Atlas's writeup)
- Goose's locking is opt-in. We don't opt in: migrations run as a
single Kubernetes Job, which IS the singleton process. No advisory
lock needed at all.
Layout:
- migrations/000001_init.sql — schema-only pg_dump of the live Neon
DB at adoption, stripped of psql-only directives that block goose's
bookkeeping insert. Pre-goose hand-numbered migrations 002-022 had
their effects folded into this baseline; deleted from the live tree
but preserved in git history at 58e6997.
- Dockerfile installs `goose v3.22.1` at build time and copies the
binary into the api image. The migrate Job reuses the api image with
command=goose, so no separate image to build/push/version.
- deploy-k3s/manifests/migrate/job.yaml: a one-shot Job that strips
the -pooler segment from DB_HOST (advisory lock won't survive
pgbouncer transaction-mode), runs `goose up`, exits.
- deploy-k3s/scripts/03-deploy.sh: deletes any prior Job, applies the
fresh one, `kubectl wait --for=condition=complete --timeout=10m`,
then proceeds with api/worker rollout. Job failure aborts the deploy
before any new app pod sees a stale schema.
- internal/database/database.go::RequireSchemaApplied checks
goose_db_version on startup. api/worker refuse to boot if the
table is missing or its latest row has is_applied=false — the
fail-fast for "operator forgot to run migrate."
- Makefile: migrate-up / migrate-down / migrate-status / migrate-new
for local workflow.
Production DB was bootstrapped manually:
$ goose -dir migrations postgres "$DSN" version # creates table
$ psql ... -c "INSERT INTO goose_db_version (version_id, is_applied, tstamp) VALUES (1, true, NOW());"
Smoke test against fresh Postgres locally: 50 user tables created in
284ms via `goose up`, version_id=1 + is_applied=t recorded.
Verified the local goose CLI talks to prod successfully:
$ goose ... status
Applied At Migration
=======================================
Mon Apr 27 03:43:55 2026 -- 000001_init.sql
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The Echo gzip middleware was wrapping promhttp's pre-gzipped output, so
vmagent received double-compressed bytes that failed the Prometheus
parser with binary garbage. Skipping /metrics in the gzip Skipper.
Three deploy-script fixes uncovered while shipping this:
- _config.sh had backticks around \"kubectl get cm\" inside the python
heredoc, which bash treated as command substitution when KUBECONFIG
was set. Quoted the literal instead.
- 03-deploy.sh now passes --platform linux/amd64 to all docker builds
so arm64 Macs don't push images that fail with \"exec format error\"
on the Hetzner CX nodes.
- OBS_INGEST_TOKEN lookup was reading deploy-k3s/prod.env instead of
the actual deploy/prod.env at the repo root.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
vmagent.yaml lives under manifests/observability/; the deploy script now
substitutes the OBS_INGEST_TOKEN from deploy/prod.env into the manifest
before apply, and waits on the vmagent rollout. Manual kubectl apply is
no longer needed after the next deploy.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The Next.js 16 webapp in sibling repo honeyDueAPI-Web now runs
alongside api/worker/admin on the cluster. Uses a server-side proxy
pattern: browser hits app.myhoneydue.com, Next.js route handlers
forward to the Go API with an httpOnly cookie, so no CORS entry or
Allowed-Hosts change is needed on the API side.
Availability mirrors api (3 replicas, PDB minAvailable:2,
topologySpreadConstraints across nodes).
Changes:
- deploy-k3s/manifests/web/deployment.yaml: 3 replicas, readOnly root
FS, drops all caps, mounts emptyDir for /app/.next/cache and /tmp,
reads API_URL from honeydue-config.
- deploy-k3s/manifests/web/service.yaml: ClusterIP :3000.
- deploy-k3s/manifests/rbac.yaml: ServiceAccount web with
automountServiceAccountToken: false.
- deploy-k3s/manifests/pod-disruption-budgets.yaml: web-pdb
minAvailable: 2.
- deploy-k3s/manifests/ingress/ingress-simple.yaml: route
app.myhoneydue.com → web:3000.
- deploy-k3s/scripts/_config.sh: emit API_URL into the ConfigMap.
- deploy-k3s/scripts/03-deploy.sh: build + push + apply the web image
alongside api/worker/admin. Reads NEXT_PUBLIC_POSTHOG_KEY and
NEXT_PUBLIC_POSTHOG_HOST from the operator shell env (not committed).
Also adds the --build-arg NEXT_PUBLIC_API_URL wiring for the admin
image that was previously only done manually.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mirrors the prod deploy-k3s/ setup but runs all services in-cluster
on a single node: PostgreSQL (replaces Neon), MinIO S3-compatible
storage (replaces B2), Redis, API, worker, and admin.
Includes fully automated setup scripts (00-init through 04-verify),
server hardening (SSH, fail2ban, ufw), Let's Encrypt TLS via Traefik,
network policies, RBAC, and security contexts matching prod.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Trey t