Migrate prod deploy from Swarm to K3s; add full deployment book
Infrastructure:
- Stack now runs on K3s v1.34.6 HA (3 Hetzner CX33 nodes as managers)
- Traefik DaemonSet + hostNetwork replaces Caddy + ingress mesh
- All manifests in deploy-k3s/manifests/; Swarm config (deploy/) kept
temporarily for reference
Bug fixes surfaced during migration:
- Dockerfile: golang:1.24-alpine -> 1.25-alpine (go.mod requires 1.25)
- cache_service.go: remove sync.Once reassignment from inside Do()
callback (was causing 'unlock of unlocked mutex' fatal after
Redis Ping failure)
- router.go: relax CSP from 'default-src none' to 'default-src self'
+ allowlist fonts.googleapis.com so the marketing landing page CSS
actually loads in browsers
- deploy/scripts/deploy_prod.sh: use docker buildx with
--platform linux/amd64 so arm64 (Apple Silicon) dev machines produce
images runnable on x86_64 Hetzner nodes; fix array expansion under
set -u
- deploy/swarm-stack.prod.yml: fix secret source references to use
top-level aliases (the '\${X_SECRET}' form never actually resolved);
dozzle ports: long-form host_ip is rejected by Swarm, switched to
short-form (bound to 0.0.0.0 with UFW-based loopback restriction);
worker replicas 2 -> 1 (Asynq scheduler singleton)
- deploy-k3s/manifests/admin/deployment.yaml: probe path '/admin/' -> '/'
(Next.js serves at root; /admin/ returned 404 and killed pods);
startupProbe failureThreshold 12 -> 24
- deploy-k3s/manifests/pod-disruption-budgets.yaml: worker minAvailable
1 -> 0 (singleton)
- deploy-k3s/manifests/api/deployment.yaml: startupProbe failureThreshold
12 -> 48 (MigrateWithLock serializes across 3 replicas on first-boot;
real startup takes up to 240s)
- .gitignore: tighten 'api' -> '/api' (was matching deploy-k3s/manifests/api/
and admin/src/app/api/*, hiding legitimate files)
New files:
- deploy-k3s/manifests/traefik-helmchartconfig.yaml: DaemonSet +
hostNetwork override for k3s-bundled Traefik
- deploy-k3s/manifests/ingress/ingress-simple.yaml: plain Ingress
without TLS (CF Flexible SSL) and without middleware
- deploy-k3s/MIGRATION_NOTES.md: operator-facing migration log
Documentation:
- docs/deployment/ — full deployment book, 26 files, ~42k words:
- Part I Overview, infrastructure, orchestrator choice (Ch 0-2)
- Part II Networking, firewall, Cloudflare (Ch 3-4, 13)
- Part III Security, Traefik ingress (Ch 5-6)
- Part IV Services, DB, storage, secrets, registry (Ch 7-11)
- Part V Data flow, deploy process, observability, failures, runbook
(Ch 12, 14-17)
- Part VI Cost, Swarm postmortem, roadmap (Ch 18-20)
- Appendices: glossary, kubectl cheat sheet, file locations,
consolidated citations
- README.md: Production Deployment section replaced with pointer to
the book; Go version bumped to 1.25
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Trey t
@@ -0,0 +1,216 @@
|
||||
# Appendix C — File Locations
|
||||
|
||||
Complete map of where every significant file lives — on the operator
|
||||
workstation, in the git repo, and on the Hetzner nodes.
|
||||
|
||||
## Operator workstation
|
||||
|
||||
### Kubernetes
|
||||
|
||||
| Path | Purpose |
|
||||
|---|---|
|
||||
| `~/.kube/honeydue-k3s.yaml` | kubeconfig for the k3s cluster. Contains an admin bearer token. Mode 0600. |
|
||||
| `~/.kube/config` | Default kubeconfig (points elsewhere, not our cluster). |
|
||||
|
||||
Set `KUBECONFIG=~/.kube/honeydue-k3s.yaml` before any `kubectl` command.
|
||||
|
||||
### SSH
|
||||
|
||||
| Path | Purpose |
|
||||
|---|---|
|
||||
| `~/.ssh/hetzner` | Private key for node SSH (ed25519). Mode 0600. |
|
||||
| `~/.ssh/hetzner.pub` | Public key corresponding to above. |
|
||||
| `~/.ssh/config` | Host aliases for hetzner1/hetzner2/hetzner3 → node IPs. |
|
||||
|
||||
Public key content:
|
||||
```
|
||||
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBU9xTTBD78tYUqHijgyU9PDqtmS4NuM/6uy8XgDzva+ hetzner2@myhoneydue.com
|
||||
```
|
||||
|
||||
### Docker
|
||||
|
||||
| Path | Purpose |
|
||||
|---|---|
|
||||
| `~/.docker/config.json` | Docker CLI config. After `docker login` to Gitea, contains creds. **Log out after each deploy** to not leave PATs on disk. |
|
||||
| `~/Library/Containers/com.docker.docker/` | Docker Desktop state (macOS). |
|
||||
|
||||
## Git repo (`/Users/treyt/Desktop/code/honeyDue/honeyDueAPI-go/`)
|
||||
|
||||
### Top-level
|
||||
|
||||
| Path | Purpose |
|
||||
|---|---|
|
||||
| `CLAUDE.md` | Project-wide instructions for Claude assistant. Never commit secrets here. |
|
||||
| `Dockerfile` | Multi-stage Docker build: api, worker, admin targets. |
|
||||
| `go.mod`, `go.sum` | Go module definition. |
|
||||
| `package.json` (admin-ui/) | Next.js dependencies. |
|
||||
|
||||
### Application code
|
||||
|
||||
| Path | Purpose |
|
||||
|---|---|
|
||||
| `cmd/api/main.go` | API server entry point. |
|
||||
| `cmd/worker/main.go` | Background worker entry point. |
|
||||
| `cmd/admin/main.go` | (may or may not exist for Go admin variant) |
|
||||
| `internal/config/` | Viper configuration loading. |
|
||||
| `internal/database/` | Postgres connection, migrations. |
|
||||
| `internal/handlers/` | HTTP handlers (one file per domain). |
|
||||
| `internal/services/` | Business logic. `cache_service.go` is where the sync.Once bug was (Chapter 19). |
|
||||
| `internal/repositories/` | GORM repositories. |
|
||||
| `internal/router/router.go` | Echo routes, including static file serving. CSP is set here. |
|
||||
| `internal/middleware/` | Echo middleware (auth, logging, etc.). |
|
||||
| `internal/task/` | Task predicates/scopes/categorization. See `docs/TASK_LOGIC_ARCHITECTURE.md`. |
|
||||
|
||||
### Deploy config (Swarm era — still exists, unused)
|
||||
|
||||
| Path | Purpose |
|
||||
|---|---|
|
||||
| `deploy/` | Legacy Swarm deploy root. |
|
||||
| `deploy/prod.env` | Non-secret config (ConfigMap source). **Gitignored.** |
|
||||
| `deploy/registry.env` | Gitea PAT + registry URL. **Gitignored.** |
|
||||
| `deploy/cluster.env` | Swarm cluster settings. Partly used for k3s too (manager host). **Gitignored.** |
|
||||
| `deploy/secrets/postgres_password.txt` | Neon password. **Gitignored.** |
|
||||
| `deploy/secrets/secret_key.txt` | App signing key (≥32 chars). **Gitignored.** |
|
||||
| `deploy/secrets/email_host_password.txt` | Fastmail password. **Gitignored.** |
|
||||
| `deploy/secrets/fcm_server_key.txt` | FCM key (placeholder, push off). **Gitignored.** |
|
||||
| `deploy/secrets/apns_auth_key.p8` | APNs key (placeholder, push off). **Gitignored.** |
|
||||
| `deploy/swarm-stack.prod.yml` | Swarm stack definition. Unused after migration. |
|
||||
| `deploy/Caddyfile` | Caddy config. Unused after migration. |
|
||||
| `deploy/scripts/deploy_prod.sh` | Swarm deploy script. Unused. |
|
||||
| `deploy/DEPLOYING.md`, `deploy/README.md`, `deploy/shit_deploy_cant_do.md` | Swarm-era docs. Historical reference. |
|
||||
|
||||
### Deploy config (k3s)
|
||||
|
||||
| Path | Purpose |
|
||||
|---|---|
|
||||
| `deploy-k3s/README.md` | k3s deployment README (scaffold version). |
|
||||
| `deploy-k3s/MIGRATION_NOTES.md` | Notes from Swarm → k3s migration. |
|
||||
| `deploy-k3s/SECURITY.md` | Security posture doc (scaffold). |
|
||||
| `deploy-k3s/config.yaml.example` | Template for a unified config.yaml (unused — we kept Swarm's file layout). |
|
||||
| `deploy-k3s/manifests/namespace.yaml` | Creates `honeydue` namespace. |
|
||||
| `deploy-k3s/manifests/rbac.yaml` | ServiceAccounts + `automountServiceAccountToken: false`. |
|
||||
| `deploy-k3s/manifests/pod-disruption-budgets.yaml` | PDBs for api (2/3) and worker (0/1). |
|
||||
| `deploy-k3s/manifests/network-policies.yaml` | Default-deny + allows. NOT currently applied. |
|
||||
| `deploy-k3s/manifests/api/deployment.yaml` | api Deployment. |
|
||||
| `deploy-k3s/manifests/api/service.yaml` | api ClusterIP Service. |
|
||||
| `deploy-k3s/manifests/api/hpa.yaml` | api HorizontalPodAutoscaler. NOT currently applied. |
|
||||
| `deploy-k3s/manifests/admin/deployment.yaml` | admin Deployment. |
|
||||
| `deploy-k3s/manifests/admin/service.yaml` | admin Service. |
|
||||
| `deploy-k3s/manifests/worker/deployment.yaml` | worker Deployment. |
|
||||
| `deploy-k3s/manifests/redis/deployment.yaml` | Redis Deployment. |
|
||||
| `deploy-k3s/manifests/redis/service.yaml` | Redis Service. |
|
||||
| `deploy-k3s/manifests/redis/pvc.yaml` | Redis PersistentVolumeClaim. |
|
||||
| `deploy-k3s/manifests/ingress/ingress.yaml` | Full Ingress with TLS + middleware (scaffold; needs CF origin cert). |
|
||||
| `deploy-k3s/manifests/ingress/ingress-simple.yaml` | Simple Ingress without TLS (what we actually apply). |
|
||||
| `deploy-k3s/manifests/ingress/middleware.yaml` | Traefik middleware CRDs. Not currently applied. |
|
||||
| `deploy-k3s/manifests/traefik-helmchartconfig.yaml` | Our DaemonSet + hostNetwork override for Traefik. |
|
||||
| `deploy-k3s/manifests/secrets.yaml.example` | Template (never deployed). |
|
||||
| `deploy-k3s/scripts/01-provision-cluster.sh` | hetzner-k3s provisioning (we didn't use it; existing nodes). |
|
||||
| `deploy-k3s/scripts/02-setup-secrets.sh` | Creates Secrets + ConfigMap (scaffold version; we ran commands manually). |
|
||||
| `deploy-k3s/scripts/03-deploy.sh` | Applies manifests (unused; we ran kubectl manually). |
|
||||
| `deploy-k3s/scripts/04-verify.sh` | Post-deploy verification. |
|
||||
| `deploy-k3s/scripts/rollback.sh` | Rollback helper. |
|
||||
|
||||
### Documentation
|
||||
|
||||
| Path | Purpose |
|
||||
|---|---|
|
||||
| `docs/deployment/` | **This book.** |
|
||||
| `docs/TASK_LOGIC_ARCHITECTURE.md` | Task logic internals. |
|
||||
| `docs/PUSH_NOTIFICATIONS.md` | Push notifications setup (for future). |
|
||||
| `docs/SUBSCRIPTION_WEBHOOKS.md` | Apple/Google subscription webhooks. |
|
||||
| `docs/Dokku_notes` | Pre-Swarm era deployment notes. Historical. |
|
||||
| `docs/server_2026_2_24.md` | Earlier architecture doc (predates k3s migration). |
|
||||
|
||||
## On the Hetzner nodes
|
||||
|
||||
### System
|
||||
|
||||
| Path | Purpose |
|
||||
|---|---|
|
||||
| `/etc/ssh/sshd_config` | SSH config — `PermitRootLogin no`, `PasswordAuthentication no`, `AllowUsers deploy`. |
|
||||
| `/etc/sudoers.d/deploy` | `deploy ALL=(ALL) NOPASSWD: ALL`. |
|
||||
| `/etc/ufw/` | UFW configuration. See Chapter 4 for rule inventory. |
|
||||
| `/etc/sysctl.d/99-unprivileged-ports.conf` | `net.ipv4.ip_unprivileged_port_start=0` for Traefik. |
|
||||
| `/home/deploy/.ssh/authorized_keys` | Our hetzner.pub. |
|
||||
|
||||
### K3s
|
||||
|
||||
| Path | Purpose |
|
||||
|---|---|
|
||||
| `/etc/rancher/k3s/k3s.yaml` | Kubeconfig (localhost-scoped; we copied to workstation). |
|
||||
| `/etc/systemd/system/k3s.service` | systemd service file. |
|
||||
| `/etc/systemd/system/k3s.service.env` | K3s install args (INSTALL_K3S_EXEC). |
|
||||
| `/var/lib/rancher/k3s/` | K3s state root (etcd, containerd, PVC storage). |
|
||||
| `/var/lib/rancher/k3s/server/node-token` | Token for joining additional nodes. |
|
||||
| `/var/lib/rancher/k3s/storage/` | local-path PVC storage. Redis data lives here. |
|
||||
| `/var/lib/rancher/k3s/agent/containerd/` | containerd state. |
|
||||
| `/var/log/containers/` | Container log files. |
|
||||
|
||||
### Commands installed
|
||||
|
||||
| Path | Purpose |
|
||||
|---|---|
|
||||
| `/usr/local/bin/k3s` | The k3s binary. |
|
||||
| `/usr/local/bin/kubectl` | Symlink to k3s (CLI for this cluster). |
|
||||
| `/usr/local/bin/crictl` | containerd CLI. |
|
||||
| `/usr/local/bin/k3s-killall.sh` | Emergency kill-all-k3s script. |
|
||||
| `/usr/local/bin/k3s-uninstall.sh` | Clean uninstall script. |
|
||||
|
||||
### Docker (legacy; disabled)
|
||||
|
||||
| Path | Purpose |
|
||||
|---|---|
|
||||
| `/etc/systemd/system/docker.service` | systemd unit (stopped + disabled). |
|
||||
| `/var/lib/docker/` | Docker state (unused on current cluster). |
|
||||
|
||||
## On Cloudflare
|
||||
|
||||
Not a filesystem, but worth noting the dashboard hierarchy:
|
||||
|
||||
```
|
||||
Websites → myhoneydue.com
|
||||
├── DNS → Records (A records for api, admin, @)
|
||||
├── SSL/TLS → Overview (SSL mode: Flexible)
|
||||
├── SSL/TLS → Edge Certificates (Always Use HTTPS: On)
|
||||
├── SSL/TLS → Origin Server (would live the Origin CA cert if we enabled it)
|
||||
├── Rules → Overview (where Origin Rules live if we had them)
|
||||
├── Rules → Page Rules (none)
|
||||
├── Security → WAF (managed rules only)
|
||||
├── Speed → Optimization (default)
|
||||
└── Analytics & Logs (read-only stats)
|
||||
```
|
||||
|
||||
## On Gitea (`gitea.treytartt.com`)
|
||||
|
||||
The image registry lives at:
|
||||
|
||||
```
|
||||
gitea.treytartt.com/admin/-/packages # UI listing of all packages
|
||||
gitea.treytartt.com/admin/-/packages/container/honeydue-api # API image
|
||||
gitea.treytartt.com/admin/-/packages/container/honeydue-worker # Worker image
|
||||
gitea.treytartt.com/admin/-/packages/container/honeydue-admin # Admin image
|
||||
```
|
||||
|
||||
Per-version tags visible in the UI with `docker pull` commands.
|
||||
|
||||
PATs at `gitea.treytartt.com/-/user/settings/applications`.
|
||||
|
||||
## On Neon
|
||||
|
||||
```
|
||||
console.neon.tech → project → Branches (production branch default)
|
||||
console.neon.tech → project → Monitoring (CU-hour usage, slow queries)
|
||||
console.neon.tech → project → Operations (history of schema changes)
|
||||
```
|
||||
|
||||
Connection strings at `console.neon.tech → project → Connection Details`.
|
||||
|
||||
## On Backblaze B2
|
||||
|
||||
```
|
||||
secure.backblaze.com/b2_buckets.htm # Buckets list
|
||||
secure.backblaze.com/b2_app_keys.htm # App keys
|
||||
```
|
||||
|
||||
`honeyDueProd` bucket → Files tab for browsing contents.
|
||||
Reference in New Issue
Block a user