honeyDueAPI/docs/deployment/10-secrets-config.md

# 10 — Secrets & Config

> **Updated 2026-05-15 (security remediation):** `honeydue-secrets` now
> carries `REDIS_PASSWORD`; an `admin-basic-auth` Secret backs the admin
> ingress; rotation is documented in `docs/runbooks/secret-rotation.md`;
> and the Go config can read file-mounted secrets (`HONEYDUE_SECRETS_DIR`).
> `deploy-k3s/SECURITY.md` is the authoritative current-state record.

## Summary

Non-sensitive config (hostnames, ports, feature flags, etc.) lives in
`honeydue-config` ConfigMap. Sensitive values (DB password, signing
keys, API keys) live in `honeydue-secrets` and `honeydue-apns-key`
Secrets. Container registry auth lives in `gitea-credentials` (type
`kubernetes.io/dockerconfigjson`). This chapter maps every env var to
its source and explains what's stored where.

## Structure

```mermaid
flowchart LR
    subgraph SourceWorkstation[Operator workstation]
        ProdEnv[deploy/prod.env]
        Secrets[deploy/secrets/*.txt]
        Registry[deploy/registry.env]
    end

    subgraph K8s[Kubernetes cluster]
        CM[honeydue-config<br/>ConfigMap]
        S1[honeydue-secrets<br/>Secret]
        S2[honeydue-apns-key<br/>Secret]
        S3[gitea-credentials<br/>Secret]
    end

    subgraph Pods
        Api[api pod]
        Admin[admin pod]
        Worker[worker pod]
    end

    ProdEnv -. kubectl create configmap<br/>--from-env-file .-> CM
    Secrets -. kubectl create secret<br/>--from-file/--from-literal .-> S1
    Secrets -. --from-file .-> S2
    Registry -. kubectl create secret docker-registry .-> S3

    CM -- envFrom --> Api & Admin & Worker
    S1 -- env: secretKeyRef --> Api & Worker
    S2 -- volumeMounts --> Api & Worker
    S3 -- imagePullSecrets --> Api & Admin & Worker
```

## ConfigMap: honeydue-config

Built from `deploy/prod.env` (minus sensitive keys). Contents (58 keys,
abbreviated):

```
ADMIN_PANEL_URL=https://admin.myhoneydue.com
ALLOWED_HOSTS=api.myhoneydue.com,myhoneydue.com
APNS_AUTH_KEY_ID=DISABLED01
APNS_AUTH_KEY_PATH=/secrets/apns/apns_auth_key.p8
APNS_PRODUCTION=false
APNS_TEAM_ID=DISABLED01
APNS_TOPIC=com.myhoneydue.honeyDue
APNS_USE_SANDBOX=false
BASE_URL=https://myhoneydue.com
B2_BUCKET_NAME=honeyDueProd
B2_ENDPOINT=s3.us-east-005.backblazeb2.com
B2_REGION=us-east-005
B2_USE_SSL=true
CORS_ALLOWED_ORIGINS=https://myhoneydue.com,https://admin.myhoneydue.com
DAILY_DIGEST_HOUR=3
DB_HOST=ep-floral-truth-amttbc5a.c-5.us-east-1.aws.neon.tech
DB_MAX_IDLE_CONNS=10
DB_MAX_LIFETIME=600s
DB_MAX_OPEN_CONNS=25
DB_PORT=5432
DB_SSLMODE=require
DEBUG=false
DEFAULT_FROM_EMAIL=noreply@myhoneydue.com
EMAIL_HOST=smtp.fastmail.com
EMAIL_HOST_USER=treytartt@fastmail.com
EMAIL_PORT=587
EMAIL_USE_TLS=true
FEATURE_EMAIL_ENABLED=true
FEATURE_ONBOARDING_EMAILS_ENABLED=true
FEATURE_PDF_REPORTS_ENABLED=true
FEATURE_PUSH_ENABLED=false
FEATURE_WEBHOOKS_ENABLED=true
FEATURE_WORKER_ENABLED=true
NEXT_PUBLIC_API_URL=https://api.myhoneydue.com
OVERDUE_REMINDER_HOUR=15
PORT=8000
POSTGRES_DB=honeyDue
POSTGRES_USER=neondb_owner
REDIS_DB=0
REDIS_URL=redis://redis:6379/0
STATIC_DIR=/app/static
STORAGE_ALLOWED_TYPES=image/jpeg,image/png,image/gif,image/webp,application/pdf
STORAGE_BASE_URL=/uploads
STORAGE_MAX_FILE_SIZE=10485760
STORAGE_UPLOAD_DIR=/app/uploads
TASK_REMINDER_HOUR=14
TIMEZONE=UTC
```

Plus empty-but-declared keys for optional integrations (Apple/Google
auth + IAP).

### How pods use it

```yaml
envFrom:
  - configMapRef:
      name: honeydue-config
```

Every key in the ConfigMap becomes an env var in the container.
`envFrom` is bulk — no need to enumerate each one.

### Changing config

Edit `deploy/prod.env` locally, regenerate the ConfigMap:

```bash
# Simplified; see scripts for the full version
kubectl create configmap honeydue-config -n honeydue \
  --from-env-file=deploy/prod.env \
  --dry-run=client -o yaml | kubectl apply -f -

# Pods don't auto-reload env vars. Restart to pick up changes:
kubectl rollout restart -n honeydue deploy/api deploy/admin deploy/worker
```

## Secret: honeydue-secrets (Opaque)

9 keys:

| Key | Purpose |
|---|---|
| `POSTGRES_PASSWORD` | Neon DB password |
| `SECRET_KEY` | Django-compat signing key (64 chars, base64) |
| `EMAIL_HOST_PASSWORD` | Fastmail app password |
| `FCM_SERVER_KEY` | FCM push key (currently placeholder, push disabled) |
| `REDIS_PASSWORD` | Empty (no auth on in-cluster Redis) |
| `B2_KEY_ID` | Backblaze B2 app key ID |
| `B2_APP_KEY` | Backblaze B2 app key secret |
| `ADMIN_EMAIL` | Next.js admin panel initial admin email |
| `ADMIN_PASSWORD` | Next.js admin panel initial admin password |

### How pods use it

Individual `env:` entries wire specific Secret keys to env vars:

```yaml
env:
  - name: POSTGRES_PASSWORD
    valueFrom:
      secretKeyRef:
        name: honeydue-secrets
        key: POSTGRES_PASSWORD
  - name: SECRET_KEY
    valueFrom:
      secretKeyRef:
        name: honeydue-secrets
        key: SECRET_KEY
  # ... etc
```

This pattern (vs. `envFrom: secretRef:`) is more explicit — you know
exactly which secret keys a pod uses by reading the manifest.

### ADMIN_PASSWORD — one-time use

The Go app's `internal/database/database.go:519-538` reads
`ADMIN_EMAIL` + `ADMIN_PASSWORD` at startup. If the `admin_users` table
doesn't have a row for that email, it inserts one with a bcrypt hash of
the password. Already-existing rows are **not** updated.

So:
- First deploy: admin user created
- Subsequent deploys: no-op
- If you want to rotate the initial admin password: do it in the admin
  panel UI, not by changing `ADMIN_PASSWORD`

After first deploy you can technically blank `ADMIN_PASSWORD` in the
Secret. Leaving it set is harmless but slightly messy.

## Secret: honeydue-apns-key (Opaque)

One file: `apns_auth_key.p8`. Mounted as a volume into api and worker
pods at `/secrets/apns/apns_auth_key.p8` (read-only).

Push is currently **disabled** (`FEATURE_PUSH_ENABLED=false`), so
this `.p8` is a throwaway EC P-256 private key generated by
`openssl genpkey`. It passes the Go app's "does this file contain
`BEGIN PRIVATE KEY`" validation but cannot authenticate against Apple.

When push is enabled:

1. Generate a real APNs auth key in Apple Developer console
2. Replace `deploy/secrets/apns_auth_key.p8`
3. Update `APNS_AUTH_KEY_ID`, `APNS_TEAM_ID`, `APNS_TOPIC` in ConfigMap
4. `kubectl create secret generic honeydue-apns-key ... --dry-run=client -o yaml | kubectl apply -f -`
5. Set `FEATURE_PUSH_ENABLED=true`
6. `kubectl rollout restart` api and worker

## Secret: gitea-credentials (docker-registry)

Type `kubernetes.io/dockerconfigjson`. Contains a base64-encoded Docker
config for Gitea registry auth.

Created via:

```bash
kubectl create secret docker-registry gitea-credentials \
  --namespace=honeydue \
  --docker-server=gitea.treytartt.com \
  --docker-username=admin \
  --docker-password=<gitea PAT> \
  --dry-run=client -o yaml | kubectl apply -f -
```

Referenced in every deployment that pulls from Gitea:

```yaml
spec:
  imagePullSecrets:
    - name: gitea-credentials
```

When a pod needs to pull an image, the kubelet reads this secret and
uses it for the registry authentication.

## Source files — what's canonical

The Swarm-era files are still the **source of truth** for secrets:

| File | Contents | Canonical? |
|---|---|---|
| `deploy/prod.env` | All non-sensitive config | Yes |
| `deploy/secrets/postgres_password.txt` | Neon DB password | Yes |
| `deploy/secrets/secret_key.txt` | App signing key | Yes |
| `deploy/secrets/email_host_password.txt` | Fastmail password | Yes |
| `deploy/secrets/fcm_server_key.txt` | FCM key (placeholder) | Yes |
| `deploy/secrets/apns_auth_key.p8` | APNs key (placeholder) | Yes |
| `deploy/registry.env` | Gitea registry auth | Yes |
| `deploy-k3s/manifests/secrets.yaml.example` | Template only (never committed with real values) | No — template |
| In-cluster Secrets | Live state | Derived |

### Why canonical lives in `deploy/` not `deploy-k3s/`

Historical. We migrated from Swarm to k3s but kept the source files
untouched. Rather than move them now (and break any remaining Swarm-era
tooling), we use them from the k3s setup scripts as-is.

Future cleanup: move to `deploy-k3s/secrets/` for better provenance.

## Recreating the cluster secrets

If the k3s cluster is rebuilt, the Secrets need to be recreated from the
local source files. Rough procedure:

```bash
export KUBECONFIG=~/.kube/honeydue-k3s.yaml

# Namespace first
kubectl create namespace honeydue

# Docker config secret for Gitea
set -a; source deploy/registry.env; set +a
kubectl create secret docker-registry gitea-credentials \
  -n honeydue \
  --docker-server="$REGISTRY" \
  --docker-username="$REGISTRY_USERNAME" \
  --docker-password="$REGISTRY_TOKEN"

# Main secrets bundle
set -a; source deploy/prod.env; set +a
kubectl create secret generic honeydue-secrets -n honeydue \
  --from-literal=POSTGRES_PASSWORD="$(tr -d '\n' < deploy/secrets/postgres_password.txt)" \
  --from-literal=SECRET_KEY="$(tr -d '\n' < deploy/secrets/secret_key.txt)" \
  --from-literal=EMAIL_HOST_PASSWORD="$(tr -d '\n' < deploy/secrets/email_host_password.txt)" \
  --from-literal=FCM_SERVER_KEY="$(tr -d '\n' < deploy/secrets/fcm_server_key.txt)" \
  --from-literal=REDIS_PASSWORD="" \
  --from-literal=B2_KEY_ID="$B2_KEY_ID" \
  --from-literal=B2_APP_KEY="$B2_APP_KEY" \
  --from-literal=ADMIN_EMAIL="$ADMIN_EMAIL" \
  --from-literal=ADMIN_PASSWORD="$ADMIN_PASSWORD"

# APNS key Secret
kubectl create secret generic honeydue-apns-key -n honeydue \
  --from-file=apns_auth_key.p8=deploy/secrets/apns_auth_key.p8

# ConfigMap from prod.env (minus secret keys)
# See deploy-k3s/scripts/02-setup-secrets.sh for the full version
# Simplified:
declare -a args
secret_keys="POSTGRES_PASSWORD SECRET_KEY EMAIL_HOST_PASSWORD FCM_SERVER_KEY REDIS_PASSWORD B2_KEY_ID B2_APP_KEY ADMIN_EMAIL ADMIN_PASSWORD"
while IFS='=' read -r k v; do
  [[ -z "$k" || "$k" =~ ^# ]] && continue
  for sk in $secret_keys; do [[ "$k" == "$sk" ]] && continue 2; done
  args+=(--from-literal="$k=$v")
done < deploy/prod.env
kubectl create configmap honeydue-config -n honeydue "${args[@]}"
```

The full version with all edge cases is in
`deploy-k3s/scripts/02-setup-secrets.sh` (which was written for the
GHCR-era assumption; adapt for Gitea).

## Pitfalls

### Trailing newlines in secret files

Secret files created by text editors typically end with a newline. If
we pass the content directly, the newline becomes part of the secret
— a mismatch to what the app expects.

We strip trailing newlines with `tr -d '\n'` before creating Secrets.
If you forget, your DB password will be silently wrong.

### Case sensitivity on POSTGRES_DB

`POSTGRES_DB=honeyDue` must be exactly `honeyDue`. `honeydue` (lowercase)
fails with `database "honeydue" does not exist`. Postgres identifiers
are case-sensitive if originally quoted at CREATE time.

### Placeholder detection

The Swarm-era deploy script rejected values containing `CHANGEME`,
`your-`, `paste_here`, etc. When setting up the k3s cluster we had to
strip those from `prod.env` first. If you ever see a pod error about
"invalid host" or "invalid key id", check if a placeholder leaked
through.

### B2_USE_SSL vs STORAGE_USE_SSL

The config has `B2_USE_SSL` but the Go code reads `STORAGE_USE_SSL`.
See Chapter 9 §Vestigial variable. Setting `B2_USE_SSL=false` in the
ConfigMap does nothing; SSL stays on.

## Operator cheat sheet

```bash
# Print a ConfigMap as env-file format
kubectl get cm honeydue-config -n honeydue -o jsonpath='{range .data}{"\n"}{end}'

# Edit a ConfigMap interactively (DOES NOT restart pods)
kubectl edit cm honeydue-config -n honeydue

# After editing a ConfigMap, restart pods to pick up
kubectl rollout restart -n honeydue deploy/api deploy/admin deploy/worker

# View a Secret (prints base64 — decode with base64 -d)
kubectl get secret honeydue-secrets -n honeydue -o yaml

# Reveal a specific secret value (DANGER: plaintext to stdout)
kubectl get secret honeydue-secrets -n honeydue \
  -o jsonpath='{.data.POSTGRES_PASSWORD}' | base64 -d

# Update a single secret key
kubectl patch secret honeydue-secrets -n honeydue \
  --type=merge -p "{\"data\":{\"SECRET_KEY\":\"$(echo -n 'newvalue' | base64)\"}}"
```

## References

- [Kubernetes ConfigMaps][cm]
- [Kubernetes Secrets][secret]
- [Secret types][secret-types]

[cm]: https://kubernetes.io/docs/concepts/configuration/configmap/
[secret]: https://kubernetes.io/docs/concepts/configuration/secret/
[secret-types]: https://kubernetes.io/docs/concepts/configuration/secret/#secret-types