honeyDueAPI/docs/runbooks/secret-rotation.md

Runbook — Secret Rotation

Closes audit finding K3S-F12 (secrets unrotated since cluster bootstrap, no rotation cadence). See deploy-k3s/SECURITY.md Stage 2.

Cadence: rotate every secret at least annually. Rotate immediately on suspected exposure, on an operator-device loss, or when anyone who has seen a secret leaves the project.

Record keeping: after each rotation, annotate the secret so the age is visible:

kubectl -n honeydue annotate secret <name> \
  honeydue.dev/last-rotated="$(date -u +%Y-%m-%d)" --overwrite

---

How rotation works

Every secret has a source of truth on the operator workstation. The deploy scripts read those sources and (re)create the Kubernetes Secrets. Rotation is always: update the source → re-run 02-setup-secrets.sh → restart the pods that consume it → revoke the old credential at its provider.

02-setup-secrets.sh uses kubectl apply (via --dry-run=client -o yaml), so re-running it is idempotent and only changes what you changed.

Kubernetes Secret	Source of truth	Consumed by
honeydue-secrets → POSTGRES_PASSWORD	deploy-k3s/secrets/postgres_password.txt	api, worker
honeydue-secrets → SECRET_KEY	deploy-k3s/secrets/secret_key.txt	api, worker
honeydue-secrets → EMAIL_HOST_PASSWORD	deploy-k3s/secrets/email_host_password.txt	api, worker
honeydue-secrets → FCM_SERVER_KEY	deploy-k3s/secrets/fcm_server_key.txt	api, worker
honeydue-secrets → REDIS_PASSWORD	config.yaml key redis.password	api, worker, redis
honeydue-secrets → OBS_INGEST_TOKEN	deploy/prod.env	api, worker
honeydue-apns-key → apns_auth_key.p8	deploy-k3s/secrets/apns_auth_key.p8	api, worker
cloudflare-origin-cert	deploy-k3s/secrets/cloudflare-origin.{crt,key}	Traefik ingress
ghcr-credentials	config.yaml block registry.*	image pulls (all pods)
admin-basic-auth	config.yaml keys admin.basic_auth_user / ..._password	Traefik admin-auth middleware

The deploy-k3s/secrets/ directory and config.yaml are gitignored — never commit them.

---

Standard rotation procedure

cd honeyDueAPI-go
export KUBECONFIG="$(pwd)/deploy-k3s/kubeconfig"

# 1. Update the source (file under deploy-k3s/secrets/ or a config.yaml key)
# 2. Recreate the Kubernetes Secrets from sources
./deploy-k3s/scripts/02-setup-secrets.sh

# 3. Restart the consumers (see per-secret notes below for which)
kubectl -n honeydue rollout restart deploy/api deploy/worker

# 4. Confirm health
kubectl -n honeydue rollout status deploy/api
kubectl -n honeydue rollout status deploy/worker

# 5. Revoke the OLD credential at its provider (see per-secret notes)
# 6. Annotate the rotated secret with today's date

---

Per-secret notes

POSTGRES_PASSWORD

1. Rotate the role password in the Neon dashboard.
2. Write the new value to deploy-k3s/secrets/postgres_password.txt.
3. 02-setup-secrets.sh, then rollout restart deploy/api deploy/worker.
4. Watch logs for connection errors; the old password stops working the moment Neon applies the change, so do steps 2–3 promptly.

SECRET_KEY ⚠️ user-visible

This signs auth tokens. Rotating it logs every user out — all existing tokens become invalid and every client must re-authenticate.

1. Generate: openssl rand -hex 32.
2. Write to deploy-k3s/secrets/secret_key.txt (must be ≥32 chars — the script enforces this; the app refuses to start in production without it).
3. 02-setup-secrets.sh, then rollout restart deploy/api deploy/worker.

• Only rotate on a schedule or on suspected compromise — not casually.
• A future improvement (overlap window via a key-id header) would let old tokens validate during the transition; not implemented today.

EMAIL_HOST_PASSWORD

1. Generate a new app password in Fastmail; keep the old one alive briefly.
2. Write to deploy-k3s/secrets/email_host_password.txt.
3. 02-setup-secrets.sh, rollout restart deploy/api deploy/worker.
4. Delete the old Fastmail app password.

FCM_SERVER_KEY

1. Rotate the key in the Firebase console.
2. Write to deploy-k3s/secrets/fcm_server_key.txt.
3. 02-setup-secrets.sh, rollout restart deploy/api deploy/worker.

REDIS_PASSWORD

Source is config.yaml key redis.password (hex only — it is embedded in the REDIS_URL, so non-hex characters would break URL parsing).

1. Generate: openssl rand -hex 32.
2. Set redis.password in config.yaml.
3. 02-setup-secrets.sh.
4. Restart redis as well as api/worker so the new --requirepass and the new REDIS_URL land together: kubectl -n honeydue rollout restart deploy/redis deploy/api deploy/worker. Expect a few seconds where api/worker reconnect.

apns_auth_key.p8

1. Revoke the key in the Apple Developer console, generate a new .p8.
2. Replace deploy-k3s/secrets/apns_auth_key.p8.
3. 02-setup-secrets.sh, rollout restart deploy/api deploy/worker.
4. If the Key ID changed, update push.apns_key_id in config.yaml too.

cloudflare-origin-cert

1. Generate a new Origin CA certificate in the Cloudflare dashboard.
2. Replace deploy-k3s/secrets/cloudflare-origin.crt and .key.
3. 02-setup-secrets.sh. Traefik picks up the new TLS secret; no app restart needed. Verify the served cert with openssl s_client.

ghcr-credentials (Gitea registry)

1. Generate a new PAT in Gitea (scope: read:packages).
2. Update the registry.token value in config.yaml.
3. 02-setup-secrets.sh. No restart needed unless a pull is pending.
4. Revoke the old PAT in Gitea.

admin-basic-auth

Source is config.yaml keys admin.basic_auth_user / basic_auth_password.

1. Set a new password (e.g. openssl rand -hex 24).
2. 02-setup-secrets.sh regenerates the bcrypt htpasswd secret.
3. No app restart needed — Traefik reloads the admin-auth middleware.
4. Distribute the new credential to whoever uses the admin panel.

---

After any rotation

• Run ./deploy-k3s/scripts/04-verify.sh and confirm no ✗ lines.
• Annotate the rotated secret (see "Record keeping" above).
• If the rotation was due to a compromise, also follow the relevant playbook in deploy-k3s/SECURITY.md → Appendix (Incident response).