Migrate prod deploy from Swarm to K3s; add full deployment book
Infrastructure:
- Stack now runs on K3s v1.34.6 HA (3 Hetzner CX33 nodes as managers)
- Traefik DaemonSet + hostNetwork replaces Caddy + ingress mesh
- All manifests in deploy-k3s/manifests/; Swarm config (deploy/) kept
temporarily for reference
Bug fixes surfaced during migration:
- Dockerfile: golang:1.24-alpine -> 1.25-alpine (go.mod requires 1.25)
- cache_service.go: remove sync.Once reassignment from inside Do()
callback (was causing 'unlock of unlocked mutex' fatal after
Redis Ping failure)
- router.go: relax CSP from 'default-src none' to 'default-src self'
+ allowlist fonts.googleapis.com so the marketing landing page CSS
actually loads in browsers
- deploy/scripts/deploy_prod.sh: use docker buildx with
--platform linux/amd64 so arm64 (Apple Silicon) dev machines produce
images runnable on x86_64 Hetzner nodes; fix array expansion under
set -u
- deploy/swarm-stack.prod.yml: fix secret source references to use
top-level aliases (the '\${X_SECRET}' form never actually resolved);
dozzle ports: long-form host_ip is rejected by Swarm, switched to
short-form (bound to 0.0.0.0 with UFW-based loopback restriction);
worker replicas 2 -> 1 (Asynq scheduler singleton)
- deploy-k3s/manifests/admin/deployment.yaml: probe path '/admin/' -> '/'
(Next.js serves at root; /admin/ returned 404 and killed pods);
startupProbe failureThreshold 12 -> 24
- deploy-k3s/manifests/pod-disruption-budgets.yaml: worker minAvailable
1 -> 0 (singleton)
- deploy-k3s/manifests/api/deployment.yaml: startupProbe failureThreshold
12 -> 48 (MigrateWithLock serializes across 3 replicas on first-boot;
real startup takes up to 240s)
- .gitignore: tighten 'api' -> '/api' (was matching deploy-k3s/manifests/api/
and admin/src/app/api/*, hiding legitimate files)
New files:
- deploy-k3s/manifests/traefik-helmchartconfig.yaml: DaemonSet +
hostNetwork override for k3s-bundled Traefik
- deploy-k3s/manifests/ingress/ingress-simple.yaml: plain Ingress
without TLS (CF Flexible SSL) and without middleware
- deploy-k3s/MIGRATION_NOTES.md: operator-facing migration log
Documentation:
- docs/deployment/ — full deployment book, 26 files, ~42k words:
- Part I Overview, infrastructure, orchestrator choice (Ch 0-2)
- Part II Networking, firewall, Cloudflare (Ch 3-4, 13)
- Part III Security, Traefik ingress (Ch 5-6)
- Part IV Services, DB, storage, secrets, registry (Ch 7-11)
- Part V Data flow, deploy process, observability, failures, runbook
(Ch 12, 14-17)
- Part VI Cost, Swarm postmortem, roadmap (Ch 18-20)
- Appendices: glossary, kubectl cheat sheet, file locations,
consolidated citations
- README.md: Production Deployment section replaced with pointer to
the book; Go version bumped to 1.25
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Trey t
@@ -0,0 +1,369 @@
|
||||
# 10 — Secrets & Config
|
||||
|
||||
## Summary
|
||||
|
||||
Non-sensitive config (hostnames, ports, feature flags, etc.) lives in
|
||||
`honeydue-config` ConfigMap. Sensitive values (DB password, signing
|
||||
keys, API keys) live in `honeydue-secrets` and `honeydue-apns-key`
|
||||
Secrets. Container registry auth lives in `gitea-credentials` (type
|
||||
`kubernetes.io/dockerconfigjson`). This chapter maps every env var to
|
||||
its source and explains what's stored where.
|
||||
|
||||
## Structure
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph SourceWorkstation[Operator workstation]
|
||||
ProdEnv[deploy/prod.env]
|
||||
Secrets[deploy/secrets/*.txt]
|
||||
Registry[deploy/registry.env]
|
||||
end
|
||||
|
||||
subgraph K8s[Kubernetes cluster]
|
||||
CM[honeydue-config<br/>ConfigMap]
|
||||
S1[honeydue-secrets<br/>Secret]
|
||||
S2[honeydue-apns-key<br/>Secret]
|
||||
S3[gitea-credentials<br/>Secret]
|
||||
end
|
||||
|
||||
subgraph Pods
|
||||
Api[api pod]
|
||||
Admin[admin pod]
|
||||
Worker[worker pod]
|
||||
end
|
||||
|
||||
ProdEnv -. kubectl create configmap<br/>--from-env-file .-> CM
|
||||
Secrets -. kubectl create secret<br/>--from-file/--from-literal .-> S1
|
||||
Secrets -. --from-file .-> S2
|
||||
Registry -. kubectl create secret docker-registry .-> S3
|
||||
|
||||
CM -- envFrom --> Api & Admin & Worker
|
||||
S1 -- env: secretKeyRef --> Api & Worker
|
||||
S2 -- volumeMounts --> Api & Worker
|
||||
S3 -- imagePullSecrets --> Api & Admin & Worker
|
||||
```
|
||||
|
||||
## ConfigMap: honeydue-config
|
||||
|
||||
Built from `deploy/prod.env` (minus sensitive keys). Contents (58 keys,
|
||||
abbreviated):
|
||||
|
||||
```
|
||||
ADMIN_PANEL_URL=https://admin.myhoneydue.com
|
||||
ALLOWED_HOSTS=api.myhoneydue.com,myhoneydue.com
|
||||
APNS_AUTH_KEY_ID=DISABLED01
|
||||
APNS_AUTH_KEY_PATH=/secrets/apns/apns_auth_key.p8
|
||||
APNS_PRODUCTION=false
|
||||
APNS_TEAM_ID=DISABLED01
|
||||
APNS_TOPIC=com.tt.honeyDue
|
||||
APNS_USE_SANDBOX=false
|
||||
BASE_URL=https://myhoneydue.com
|
||||
B2_BUCKET_NAME=honeyDueProd
|
||||
B2_ENDPOINT=s3.us-east-005.backblazeb2.com
|
||||
B2_REGION=us-east-005
|
||||
B2_USE_SSL=true
|
||||
CORS_ALLOWED_ORIGINS=https://myhoneydue.com,https://admin.myhoneydue.com
|
||||
DAILY_DIGEST_HOUR=3
|
||||
DB_HOST=ep-floral-truth-amttbc5a.c-5.us-east-1.aws.neon.tech
|
||||
DB_MAX_IDLE_CONNS=10
|
||||
DB_MAX_LIFETIME=600s
|
||||
DB_MAX_OPEN_CONNS=25
|
||||
DB_PORT=5432
|
||||
DB_SSLMODE=require
|
||||
DEBUG=false
|
||||
DEFAULT_FROM_EMAIL=noreply@myhoneydue.com
|
||||
EMAIL_HOST=smtp.fastmail.com
|
||||
EMAIL_HOST_USER=treytartt@fastmail.com
|
||||
EMAIL_PORT=587
|
||||
EMAIL_USE_TLS=true
|
||||
FEATURE_EMAIL_ENABLED=true
|
||||
FEATURE_ONBOARDING_EMAILS_ENABLED=true
|
||||
FEATURE_PDF_REPORTS_ENABLED=true
|
||||
FEATURE_PUSH_ENABLED=false
|
||||
FEATURE_WEBHOOKS_ENABLED=true
|
||||
FEATURE_WORKER_ENABLED=true
|
||||
NEXT_PUBLIC_API_URL=https://api.myhoneydue.com
|
||||
OVERDUE_REMINDER_HOUR=15
|
||||
PORT=8000
|
||||
POSTGRES_DB=honeyDue
|
||||
POSTGRES_USER=neondb_owner
|
||||
REDIS_DB=0
|
||||
REDIS_URL=redis://redis:6379/0
|
||||
STATIC_DIR=/app/static
|
||||
STORAGE_ALLOWED_TYPES=image/jpeg,image/png,image/gif,image/webp,application/pdf
|
||||
STORAGE_BASE_URL=/uploads
|
||||
STORAGE_MAX_FILE_SIZE=10485760
|
||||
STORAGE_UPLOAD_DIR=/app/uploads
|
||||
TASK_REMINDER_HOUR=14
|
||||
TIMEZONE=UTC
|
||||
```
|
||||
|
||||
Plus empty-but-declared keys for optional integrations (Apple/Google
|
||||
auth + IAP).
|
||||
|
||||
### How pods use it
|
||||
|
||||
```yaml
|
||||
envFrom:
|
||||
- configMapRef:
|
||||
name: honeydue-config
|
||||
```
|
||||
|
||||
Every key in the ConfigMap becomes an env var in the container.
|
||||
`envFrom` is bulk — no need to enumerate each one.
|
||||
|
||||
### Changing config
|
||||
|
||||
Edit `deploy/prod.env` locally, regenerate the ConfigMap:
|
||||
|
||||
```bash
|
||||
# Simplified; see scripts for the full version
|
||||
kubectl create configmap honeydue-config -n honeydue \
|
||||
--from-env-file=deploy/prod.env \
|
||||
--dry-run=client -o yaml | kubectl apply -f -
|
||||
|
||||
# Pods don't auto-reload env vars. Restart to pick up changes:
|
||||
kubectl rollout restart -n honeydue deploy/api deploy/admin deploy/worker
|
||||
```
|
||||
|
||||
## Secret: honeydue-secrets (Opaque)
|
||||
|
||||
9 keys:
|
||||
|
||||
| Key | Purpose |
|
||||
|---|---|
|
||||
| `POSTGRES_PASSWORD` | Neon DB password |
|
||||
| `SECRET_KEY` | Django-compat signing key (64 chars, base64) |
|
||||
| `EMAIL_HOST_PASSWORD` | Fastmail app password |
|
||||
| `FCM_SERVER_KEY` | FCM push key (currently placeholder, push disabled) |
|
||||
| `REDIS_PASSWORD` | Empty (no auth on in-cluster Redis) |
|
||||
| `B2_KEY_ID` | Backblaze B2 app key ID |
|
||||
| `B2_APP_KEY` | Backblaze B2 app key secret |
|
||||
| `ADMIN_EMAIL` | Next.js admin panel initial admin email |
|
||||
| `ADMIN_PASSWORD` | Next.js admin panel initial admin password |
|
||||
|
||||
### How pods use it
|
||||
|
||||
Individual `env:` entries wire specific Secret keys to env vars:
|
||||
|
||||
```yaml
|
||||
env:
|
||||
- name: POSTGRES_PASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: honeydue-secrets
|
||||
key: POSTGRES_PASSWORD
|
||||
- name: SECRET_KEY
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: honeydue-secrets
|
||||
key: SECRET_KEY
|
||||
# ... etc
|
||||
```
|
||||
|
||||
This pattern (vs. `envFrom: secretRef:`) is more explicit — you know
|
||||
exactly which secret keys a pod uses by reading the manifest.
|
||||
|
||||
### ADMIN_PASSWORD — one-time use
|
||||
|
||||
The Go app's `internal/database/database.go:519-538` reads
|
||||
`ADMIN_EMAIL` + `ADMIN_PASSWORD` at startup. If the `admin_users` table
|
||||
doesn't have a row for that email, it inserts one with a bcrypt hash of
|
||||
the password. Already-existing rows are **not** updated.
|
||||
|
||||
So:
|
||||
- First deploy: admin user created
|
||||
- Subsequent deploys: no-op
|
||||
- If you want to rotate the initial admin password: do it in the admin
|
||||
panel UI, not by changing `ADMIN_PASSWORD`
|
||||
|
||||
After first deploy you can technically blank `ADMIN_PASSWORD` in the
|
||||
Secret. Leaving it set is harmless but slightly messy.
|
||||
|
||||
## Secret: honeydue-apns-key (Opaque)
|
||||
|
||||
One file: `apns_auth_key.p8`. Mounted as a volume into api and worker
|
||||
pods at `/secrets/apns/apns_auth_key.p8` (read-only).
|
||||
|
||||
Push is currently **disabled** (`FEATURE_PUSH_ENABLED=false`), so
|
||||
this `.p8` is a throwaway EC P-256 private key generated by
|
||||
`openssl genpkey`. It passes the Go app's "does this file contain
|
||||
`BEGIN PRIVATE KEY`" validation but cannot authenticate against Apple.
|
||||
|
||||
When push is enabled:
|
||||
|
||||
1. Generate a real APNs auth key in Apple Developer console
|
||||
2. Replace `deploy/secrets/apns_auth_key.p8`
|
||||
3. Update `APNS_AUTH_KEY_ID`, `APNS_TEAM_ID`, `APNS_TOPIC` in ConfigMap
|
||||
4. `kubectl create secret generic honeydue-apns-key ... --dry-run=client -o yaml | kubectl apply -f -`
|
||||
5. Set `FEATURE_PUSH_ENABLED=true`
|
||||
6. `kubectl rollout restart` api and worker
|
||||
|
||||
## Secret: gitea-credentials (docker-registry)
|
||||
|
||||
Type `kubernetes.io/dockerconfigjson`. Contains a base64-encoded Docker
|
||||
config for Gitea registry auth.
|
||||
|
||||
Created via:
|
||||
|
||||
```bash
|
||||
kubectl create secret docker-registry gitea-credentials \
|
||||
--namespace=honeydue \
|
||||
--docker-server=gitea.treytartt.com \
|
||||
--docker-username=admin \
|
||||
--docker-password=<gitea PAT> \
|
||||
--dry-run=client -o yaml | kubectl apply -f -
|
||||
```
|
||||
|
||||
Referenced in every deployment that pulls from Gitea:
|
||||
|
||||
```yaml
|
||||
spec:
|
||||
imagePullSecrets:
|
||||
- name: gitea-credentials
|
||||
```
|
||||
|
||||
When a pod needs to pull an image, the kubelet reads this secret and
|
||||
uses it for the registry authentication.
|
||||
|
||||
## Source files — what's canonical
|
||||
|
||||
The Swarm-era files are still the **source of truth** for secrets:
|
||||
|
||||
| File | Contents | Canonical? |
|
||||
|---|---|---|
|
||||
| `deploy/prod.env` | All non-sensitive config | Yes |
|
||||
| `deploy/secrets/postgres_password.txt` | Neon DB password | Yes |
|
||||
| `deploy/secrets/secret_key.txt` | App signing key | Yes |
|
||||
| `deploy/secrets/email_host_password.txt` | Fastmail password | Yes |
|
||||
| `deploy/secrets/fcm_server_key.txt` | FCM key (placeholder) | Yes |
|
||||
| `deploy/secrets/apns_auth_key.p8` | APNs key (placeholder) | Yes |
|
||||
| `deploy/registry.env` | Gitea registry auth | Yes |
|
||||
| `deploy-k3s/manifests/secrets.yaml.example` | Template only (never committed with real values) | No — template |
|
||||
| In-cluster Secrets | Live state | Derived |
|
||||
|
||||
### Why canonical lives in `deploy/` not `deploy-k3s/`
|
||||
|
||||
Historical. We migrated from Swarm to k3s but kept the source files
|
||||
untouched. Rather than move them now (and break any remaining Swarm-era
|
||||
tooling), we use them from the k3s setup scripts as-is.
|
||||
|
||||
Future cleanup: move to `deploy-k3s/secrets/` for better provenance.
|
||||
|
||||
## Recreating the cluster secrets
|
||||
|
||||
If the k3s cluster is rebuilt, the Secrets need to be recreated from the
|
||||
local source files. Rough procedure:
|
||||
|
||||
```bash
|
||||
export KUBECONFIG=~/.kube/honeydue-k3s.yaml
|
||||
|
||||
# Namespace first
|
||||
kubectl create namespace honeydue
|
||||
|
||||
# Docker config secret for Gitea
|
||||
set -a; source deploy/registry.env; set +a
|
||||
kubectl create secret docker-registry gitea-credentials \
|
||||
-n honeydue \
|
||||
--docker-server="$REGISTRY" \
|
||||
--docker-username="$REGISTRY_USERNAME" \
|
||||
--docker-password="$REGISTRY_TOKEN"
|
||||
|
||||
# Main secrets bundle
|
||||
set -a; source deploy/prod.env; set +a
|
||||
kubectl create secret generic honeydue-secrets -n honeydue \
|
||||
--from-literal=POSTGRES_PASSWORD="$(tr -d '\n' < deploy/secrets/postgres_password.txt)" \
|
||||
--from-literal=SECRET_KEY="$(tr -d '\n' < deploy/secrets/secret_key.txt)" \
|
||||
--from-literal=EMAIL_HOST_PASSWORD="$(tr -d '\n' < deploy/secrets/email_host_password.txt)" \
|
||||
--from-literal=FCM_SERVER_KEY="$(tr -d '\n' < deploy/secrets/fcm_server_key.txt)" \
|
||||
--from-literal=REDIS_PASSWORD="" \
|
||||
--from-literal=B2_KEY_ID="$B2_KEY_ID" \
|
||||
--from-literal=B2_APP_KEY="$B2_APP_KEY" \
|
||||
--from-literal=ADMIN_EMAIL="$ADMIN_EMAIL" \
|
||||
--from-literal=ADMIN_PASSWORD="$ADMIN_PASSWORD"
|
||||
|
||||
# APNS key Secret
|
||||
kubectl create secret generic honeydue-apns-key -n honeydue \
|
||||
--from-file=apns_auth_key.p8=deploy/secrets/apns_auth_key.p8
|
||||
|
||||
# ConfigMap from prod.env (minus secret keys)
|
||||
# See deploy-k3s/scripts/02-setup-secrets.sh for the full version
|
||||
# Simplified:
|
||||
declare -a args
|
||||
secret_keys="POSTGRES_PASSWORD SECRET_KEY EMAIL_HOST_PASSWORD FCM_SERVER_KEY REDIS_PASSWORD B2_KEY_ID B2_APP_KEY ADMIN_EMAIL ADMIN_PASSWORD"
|
||||
while IFS='=' read -r k v; do
|
||||
[[ -z "$k" || "$k" =~ ^# ]] && continue
|
||||
for sk in $secret_keys; do [[ "$k" == "$sk" ]] && continue 2; done
|
||||
args+=(--from-literal="$k=$v")
|
||||
done < deploy/prod.env
|
||||
kubectl create configmap honeydue-config -n honeydue "${args[@]}"
|
||||
```
|
||||
|
||||
The full version with all edge cases is in
|
||||
`deploy-k3s/scripts/02-setup-secrets.sh` (which was written for the
|
||||
GHCR-era assumption; adapt for Gitea).
|
||||
|
||||
## Pitfalls
|
||||
|
||||
### Trailing newlines in secret files
|
||||
|
||||
Secret files created by text editors typically end with a newline. If
|
||||
we pass the content directly, the newline becomes part of the secret
|
||||
— a mismatch to what the app expects.
|
||||
|
||||
We strip trailing newlines with `tr -d '\n'` before creating Secrets.
|
||||
If you forget, your DB password will be silently wrong.
|
||||
|
||||
### Case sensitivity on POSTGRES_DB
|
||||
|
||||
`POSTGRES_DB=honeyDue` must be exactly `honeyDue`. `honeydue` (lowercase)
|
||||
fails with `database "honeydue" does not exist`. Postgres identifiers
|
||||
are case-sensitive if originally quoted at CREATE time.
|
||||
|
||||
### Placeholder detection
|
||||
|
||||
The Swarm-era deploy script rejected values containing `CHANGEME`,
|
||||
`your-`, `paste_here`, etc. When setting up the k3s cluster we had to
|
||||
strip those from `prod.env` first. If you ever see a pod error about
|
||||
"invalid host" or "invalid key id", check if a placeholder leaked
|
||||
through.
|
||||
|
||||
### B2_USE_SSL vs STORAGE_USE_SSL
|
||||
|
||||
The config has `B2_USE_SSL` but the Go code reads `STORAGE_USE_SSL`.
|
||||
See Chapter 9 §Vestigial variable. Setting `B2_USE_SSL=false` in the
|
||||
ConfigMap does nothing; SSL stays on.
|
||||
|
||||
## Operator cheat sheet
|
||||
|
||||
```bash
|
||||
# Print a ConfigMap as env-file format
|
||||
kubectl get cm honeydue-config -n honeydue -o jsonpath='{range .data}{"\n"}{end}'
|
||||
|
||||
# Edit a ConfigMap interactively (DOES NOT restart pods)
|
||||
kubectl edit cm honeydue-config -n honeydue
|
||||
|
||||
# After editing a ConfigMap, restart pods to pick up
|
||||
kubectl rollout restart -n honeydue deploy/api deploy/admin deploy/worker
|
||||
|
||||
# View a Secret (prints base64 — decode with base64 -d)
|
||||
kubectl get secret honeydue-secrets -n honeydue -o yaml
|
||||
|
||||
# Reveal a specific secret value (DANGER: plaintext to stdout)
|
||||
kubectl get secret honeydue-secrets -n honeydue \
|
||||
-o jsonpath='{.data.POSTGRES_PASSWORD}' | base64 -d
|
||||
|
||||
# Update a single secret key
|
||||
kubectl patch secret honeydue-secrets -n honeydue \
|
||||
--type=merge -p "{\"data\":{\"SECRET_KEY\":\"$(echo -n 'newvalue' | base64)\"}}"
|
||||
```
|
||||
|
||||
## References
|
||||
|
||||
- [Kubernetes ConfigMaps][cm]
|
||||
- [Kubernetes Secrets][secret]
|
||||
- [Secret types][secret-types]
|
||||
|
||||
[cm]: https://kubernetes.io/docs/concepts/configuration/configmap/
|
||||
[secret]: https://kubernetes.io/docs/concepts/configuration/secret/
|
||||
[secret-types]: https://kubernetes.io/docs/concepts/configuration/secret/#secret-types
|
||||
Reference in New Issue
Block a user