Trey t
6f303dbbaa
Infrastructure:
- Stack now runs on K3s v1.34.6 HA (3 Hetzner CX33 nodes as managers)
- Traefik DaemonSet + hostNetwork replaces Caddy + ingress mesh
- All manifests in deploy-k3s/manifests/; Swarm config (deploy/) kept
temporarily for reference
Bug fixes surfaced during migration:
- Dockerfile: golang:1.24-alpine -> 1.25-alpine (go.mod requires 1.25)
- cache_service.go: remove sync.Once reassignment from inside Do()
callback (was causing 'unlock of unlocked mutex' fatal after
Redis Ping failure)
- router.go: relax CSP from 'default-src none' to 'default-src self'
+ allowlist fonts.googleapis.com so the marketing landing page CSS
actually loads in browsers
- deploy/scripts/deploy_prod.sh: use docker buildx with
--platform linux/amd64 so arm64 (Apple Silicon) dev machines produce
images runnable on x86_64 Hetzner nodes; fix array expansion under
set -u
- deploy/swarm-stack.prod.yml: fix secret source references to use
top-level aliases (the '\${X_SECRET}' form never actually resolved);
dozzle ports: long-form host_ip is rejected by Swarm, switched to
short-form (bound to 0.0.0.0 with UFW-based loopback restriction);
worker replicas 2 -> 1 (Asynq scheduler singleton)
- deploy-k3s/manifests/admin/deployment.yaml: probe path '/admin/' -> '/'
(Next.js serves at root; /admin/ returned 404 and killed pods);
startupProbe failureThreshold 12 -> 24
- deploy-k3s/manifests/pod-disruption-budgets.yaml: worker minAvailable
1 -> 0 (singleton)
- deploy-k3s/manifests/api/deployment.yaml: startupProbe failureThreshold
12 -> 48 (MigrateWithLock serializes across 3 replicas on first-boot;
real startup takes up to 240s)
- .gitignore: tighten 'api' -> '/api' (was matching deploy-k3s/manifests/api/
and admin/src/app/api/*, hiding legitimate files)
New files:
- deploy-k3s/manifests/traefik-helmchartconfig.yaml: DaemonSet +
hostNetwork override for k3s-bundled Traefik
- deploy-k3s/manifests/ingress/ingress-simple.yaml: plain Ingress
without TLS (CF Flexible SSL) and without middleware
- deploy-k3s/MIGRATION_NOTES.md: operator-facing migration log
Documentation:
- docs/deployment/ — full deployment book, 26 files, ~42k words:
- Part I Overview, infrastructure, orchestrator choice (Ch 0-2)
- Part II Networking, firewall, Cloudflare (Ch 3-4, 13)
- Part III Security, Traefik ingress (Ch 5-6)
- Part IV Services, DB, storage, secrets, registry (Ch 7-11)
- Part V Data flow, deploy process, observability, failures, runbook
(Ch 12, 14-17)
- Part VI Cost, Swarm postmortem, roadmap (Ch 18-20)
- Appendices: glossary, kubectl cheat sheet, file locations,
consolidated citations
- README.md: Production Deployment section replaced with pointer to
the book; Go version bumped to 1.25
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
10 KiB
20 — Roadmap
Summary
A consolidated list of known gaps, improvements, and scaling triggers. Items are grouped by category and roughly ordered by priority. This is the "if we had more time" list referenced throughout the book.
High priority (do soon)
Uptime monitoring
Why: Right now we find out the site is down when users complain.
How: Set up Uptime Kuma (self-hosted) or Better Stack Uptime
(free tier) to ping https://api.myhoneydue.com/api/health/ every
minute, with Slack/email alerts on failure.
Effort: ~30 min for Uptime Kuma deploy, ~10 min for Better Stack signup.
Cloudflare origin IP restriction
Why: UFW allows :80 from anywhere. If node IPs leak, direct-connect attackers bypass CF's WAF/DDoS protection.
How: Replace the anywhere-80 UFW rule with 15 IPv4 + 7 IPv6 CF ranges. See Chapter 13 §CF IP ranges.
Automation: a small script that refreshes the CF IP list monthly and re-applies UFW rules.
Effort: 1 hour.
Enable network policies in k3s
Why: Currently pods can freely egress anywhere. A compromised pod could exfiltrate data or attack lateral services.
How: kubectl apply -f deploy-k3s/manifests/network-policies.yaml.
The scaffold defines default-deny + explicit allows for:
- DNS egress for all pods
- Traefik → api (port 8000)
- Traefik → admin (port 3000)
- api/worker → Redis
- api/worker → external services (Postgres, B2, Fastmail)
Then test that nothing breaks (might need to adjust allow rules).
Effort: 1-2 hours including testing.
Apply Traefik security middleware
Why: Our current Ingress has no rate limiting or security headers beyond what Traefik adds by default.
How: Apply deploy-k3s/manifests/ingress/middleware.yaml, annotate
Ingresses to use them:
metadata:
annotations:
traefik.ingress.kubernetes.io/router.middlewares: honeydue-security-headers@kubernetescrd,honeydue-rate-limit@kubernetescrd
Effort: 15 min.
Medium priority
Upgrade to CF Full (strict) SSL
Why: Currently CF↔origin is plain HTTP. An attacker between CF and Hetzner could read traffic. Full (strict) mode encrypts this leg with a CF-issued origin cert.
How:
- Generate Origin CA cert in CF dashboard → SSL/TLS → Origin Server
- Create
cloudflare-origin-certSecret in k8s - Add
tls:block to Ingresses - Switch CF SSL mode to Full (strict)
Effort: 30 min.
Citations: [Cloudflare Origin CA docs][cf-origin-ca]
Migration Job for schema changes
Why: Currently every api pod runs MigrateWithLock() on startup,
serializing on a Postgres advisory lock. Adds 90-240s to cold startup
and caused bug #13 in Chapter 19.
How: Create a Kubernetes Job resource that runs the api image
with a --migrate-only flag. Job runs once per deploy, completes when
schema is current. api pods get an initContainer that waits for the
Job to complete.
Requires Go code change to support --migrate-only flag.
Effort: 3-4 hours (code + job manifest + testing).
Redis password
Why: Redis runs in the cluster with no auth. Any compromised pod could read cache or queue state.
How: Set REDIS_PASSWORD in honeydue-secrets, update api/worker
env, update Redis command to include --requirepass. Already partially
wired up in the manifests.
Effort: 20 min.
Image signing with cosign
Why: No guarantee that an image pulled from Gitea is the one we built. Gitea compromise = arbitrary code execution in cluster.
How:
- Install cosign on build machine
- Sign images as part of deploy:
cosign sign gitea.treytartt.com/admin/honeydue-api:<sha> - Deploy Kyverno (or Connaisseur) to cluster
- Apply cluster policy requiring all images have valid cosign signatures
Effort: 4-6 hours.
etcd encryption at rest
Why: Kubernetes Secrets are stored in etcd unencrypted by default. Node disk compromise = plaintext secrets.
How: K3s supports --secrets-encryption flag at server install.
Need to recreate cluster or re-install k3s server on each node.
Effort: 1 hour.
Automated unattended-upgrades
Why: Currently OS patches require manual apt upgrade. Security
patches can be delayed.
How:
sudo apt install unattended-upgrades
# Configure /etc/apt/apt.conf.d/50unattended-upgrades for security-only
sudo dpkg-reconfigure -plow unattended-upgrades
Effort: 30 min per node.
fail2ban
Why: SSH is open to the world. No rate limiting on failed attempts. Bot noise is constant.
How: sudo apt install fail2ban; sudo systemctl enable --now fail2ban.
Default config bans IPs after 5 failed attempts for 10 min.
Effort: 15 min per node.
Move SSH off port 22
Why: Port 22 attracts constant scanner noise. Moving to a non-default port cuts >90% of attempts.
How:
- Edit
/etc/ssh/sshd_configon each node:Port 2222 - UFW rule:
sudo ufw allow 2222/tcp - Update
~/.ssh/configon operator:Port 2222 - Restart sshd:
sudo systemctl restart ssh - Remove UFW rule for port 22 after verifying
Effort: 30 min (and pray).
Lower priority
Prometheus + Grafana
Why: Historical metrics, dashboards, alerting.
How: kube-prometheus-stack Helm chart. Adds ~500 MB RAM across
cluster.
Effort: 4-6 hours including dashboard setup.
Loki log aggregation
Why: Cross-pod log queries, longer retention.
How: grafana/loki + promtail DaemonSet. Integrates with existing
Grafana.
Effort: 2-3 hours.
OpenTelemetry tracing
Why: Request-level profiling. Show which hop dominates p99 latency.
How: Add OpenTelemetry SDK to Go app; export to Jaeger/Tempo.
Effort: 8-12 hours including tuning.
Hetzner private network
Why: Currently all inter-node traffic (including Flannel overlay) goes over public network. Private network = less attack surface, no bandwidth costs (if metered in future).
How: Attach Hetzner vswitch to the 3 nodes, reconfigure Flannel to advertise private IPs, update UFW rules to allow from private IP range instead of specific public IPs.
Effort: 2-3 hours including testing Flannel reconfig.
Move secrets to Vault
Why: Kubernetes Secrets are base64-encoded etcd values. Vault is purpose-built for secret management with audit logs, dynamic secrets, rotation policies.
How: Deploy Vault in the cluster (or external), migrate secret values, use Vault Agent Injector or External Secrets Operator.
Effort: 6-8 hours.
Not high priority until we have multiple engineers who shouldn't see every secret, or compliance requirements.
Automated backups to B2
Why: Neon's backup is Neon's problem. If Neon-as-a-company disappeared, we'd lose everything.
How: Nightly pg_dump | gzip | aws s3 cp (via s3cmd for B2) as a
CronJob in the cluster.
Effort: 2 hours.
Multi-region
Why: ~100 ms CF→origin hop could be reduced by having origins in multiple regions. Not needed at current scale.
How: Add 2 more Hetzner nodes in ash (Ashburn, US). Separate k3s cluster (or one stretched cluster — painful). Cloudflare Load Balancing for geo-based routing.
Effort: Days of work, doubling cost. Don't until traffic justifies.
CF Workers for static + caching
Why: Certain endpoints (the marketing landing page, public API lookups) could serve from CF Workers with near-zero origin load.
How: Move static pages to Cloudflare Pages; cache API responses
with Cache-Control: public, max-age=300.
Effort: 4-6 hours.
WireGuard-encrypted overlay
Why: Current Flannel VXLAN is plaintext between nodes. An attacker with Hetzner-internal network access could read pod-to-pod traffic.
How: K3s supports --flannel-backend=wireguard-native. Reinstall
k3s server on each node with the new backend.
Effort: 2-3 hours (requires brief downtime).
Scaling triggers
| Trigger | Action |
|---|---|
| p99 latency > 500ms sustained | Investigate with tracing; consider CF Workers for cached paths |
| API CPU > 70% sustained | HPA already configured; may need more nodes |
| DB connections at Neon limit | Upgrade Neon Scale or reduce DB_MAX_OPEN_CONNS |
| Redis memory > 80% | Scale Redis memory; consider cache sharding |
| B2 storage > 500 GB | Evaluate if R2 (free egress) is cheaper overall |
| Active users > 100k | Evaluate multi-region, CF Pro, paid monitoring |
| Revenue > $5k/mo | Hire ops help; this document assumes solo operator |
Known gaps we accept
- No canary deploys: all-or-nothing rollouts via
kubectl set image - No feature flags (app-level): code is deployed as-is. Can't toggle features without re-deploying
- No A/B testing infra: out of scope for current product stage
- No Windows/tablet-specific CDN rules: CF serves everyone the same responses
- No explicit blue-green: rolling updates only
Stuff to delete when brave
deploy/(the Swarm era) — once we've been on k3s 30 days- Legacy UFW rules from the Swarm era (2377, 7946, 4789, ESP, 500, 3000) — they don't hurt but they're confusing
deploy-k3s/manifests/secrets.yaml.example— we don't use this pattern, we create secrets imperatively
Stuff that could go wrong and we should plan for
- Hetzner price hike: 2026-04-01 already happened. If another one comes, we could migrate to Netcup or OVH for savings.
- Neon EOL free tier: Neon could change pricing policy. Fallback: self-host Postgres on a Hetzner box or migrate to Supabase.
- Cloudflare Free plan changes: CF could restrict Free features. Fallback: BunnyCDN, or raw nodes without CDN.
- Gitea host outage: If Gitea is down, deploys can't pull new images. Existing pods continue. For long outages, we'd cache images locally or temporarily push to Docker Hub.
Progress tracker
As items are done, mark them here. Think of this as a running changelog.
- k3s migration from Swarm (2026-04-24)
- Traefik DaemonSet + hostNetwork
- Admin seed via ADMIN_EMAIL + ADMIN_PASSWORD
- Documentation book (this doc set)
- All other items above