honeyDueAPI/docs/deployment/20-roadmap.md

20 — Roadmap

Summary

A consolidated list of known gaps, improvements, and scaling triggers. Items are grouped by category and roughly ordered by priority. This is the "if we had more time" list referenced throughout the book.

High priority (do soon)

Uptime monitoring

Why: Right now we find out the site is down when users complain.

How: Set up Uptime Kuma (self-hosted) or Better Stack Uptime (free tier) to ping https://api.myhoneydue.com/api/health/ every minute, with Slack/email alerts on failure.

Effort: ~30 min for Uptime Kuma deploy, ~10 min for Better Stack signup.

Cloudflare origin IP restriction ✓ DONE (2026-04-24)

Both :80 and :443 Anywhere rules removed on all 3 nodes. Only CF's 15 IPv4 + 7 IPv6 ranges allowed on :443. Direct-connect attempts from non-CF IPs time out.

Still TODO: monthly automated refresh of the CF IP list. Ranges change rarely; manual re-run of scripts/ufw-cf-refresh.sh (not yet written) on cadence is acceptable for now.

Enable network policies in k3s ✓ DONE (2026-04-24)

Applied with one scaffold correction: Traefik runs as a DaemonSet with hostNetwork: true, so traffic from it arrives with the node IP as source rather than a pod IP. The original scaffold used namespaceSelector: kube-system which doesn't match hostNetwork traffic. Fixed by using an ipBlock list of the three node IPs plus the cluster pod CIDR 10.42.0.0/16.

Also added policies for web (missing from the original scaffold).

Apply Traefik security middleware ✓ DONE (2026-04-24)

security-headers + rate-limit attached to all three ingresses (api, admin, web). admin-auth is defined but not attached (needs an admin-basic-auth secret we haven't created). cloudflare-only IP allowlist exists but is redundant with the UFW-level CF restriction — keep for defense in depth if we ever expose another layer.

One scaffold correction: the Content-Security-Policy header in security-headers.customResponseHeaders was stripped. The Go API sets its own CSP in internal/router/router.go, and two CSP headers combine via intersection (most restrictive wins), which would break the Google Fonts on the marketing landing page. Next.js apps set their own via middleware.

Medium priority

Upgrade to CF Full (strict) SSL ✓ DONE (2026-04-24)

Origin CA cert (*.myhoneydue.com + myhoneydue.com, 15-year validity) stored as cloudflare-origin-cert TLS secret. All three ingresses reference it via tls: blocks. CF mode flipped from Flexible to Full (strict). Verified by:

• direct-connect to origin on :443 serves the Origin cert (subject CN=CloudFlare Origin Certificate)
• CF edge continues to serve its own Let's Encrypt cert to browsers
• both layers now TLS-encrypted

Migration Job for schema changes — done (2026-04-26, commit 12b2f9d)

What shipped: pressly/goose as the migration tool, run as a one-shot Kubernetes Job from deploy-k3s/manifests/migrate/job.yaml before api/worker rollout. The Job uses the api image (goose CLI is baked in during the Dockerfile build), strips -pooler from DB_HOST for the direct-endpoint connection migrations need, and exits in seconds when there's nothing to apply. RequireSchemaApplied in the api/worker startup checks goose_db_version and fails fast on a stale schema.

The Go-code-with---migrate-only shape originally proposed here was rejected in favor of using the upstream goose binary directly — see Chapter 8 §Schema management for the trade-offs.

Pre-goose MigrateWithLock is gone; ch19 §13 has the historical postmortem context.

Redis password

Why: Redis runs in the cluster with no auth. Any compromised pod could read cache or queue state.

How: Set REDIS_PASSWORD in honeydue-secrets, update api/worker env, update Redis command to include --requirepass. Already partially wired up in the manifests.

Effort: 20 min.

Image signing with cosign

Why: No guarantee that an image pulled from Gitea is the one we built. Gitea compromise = arbitrary code execution in cluster.

How:

1. Install cosign on build machine
2. Sign images as part of deploy: cosign sign gitea.treytartt.com/admin/honeydue-api:<sha>
3. Deploy Kyverno (or Connaisseur) to cluster
4. Apply cluster policy requiring all images have valid cosign signatures

Effort: 4-6 hours.

etcd encryption at rest

Why: Kubernetes Secrets are stored in etcd unencrypted by default. Node disk compromise = plaintext secrets.

How: K3s supports --secrets-encryption flag at server install. Need to recreate cluster or re-install k3s server on each node.

Effort: 1 hour.

Automated unattended-upgrades

Why: Currently OS patches require manual apt upgrade. Security patches can be delayed.

How:

sudo apt install unattended-upgrades
# Configure /etc/apt/apt.conf.d/50unattended-upgrades for security-only
sudo dpkg-reconfigure -plow unattended-upgrades

Effort: 30 min per node.

fail2ban

Why: SSH is open to the world. No rate limiting on failed attempts. Bot noise is constant.

How: sudo apt install fail2ban; sudo systemctl enable --now fail2ban. Default config bans IPs after 5 failed attempts for 10 min.

Effort: 15 min per node.

Move SSH off port 22

Why: Port 22 attracts constant scanner noise. Moving to a non-default port cuts >90% of attempts.

How:

1. Edit /etc/ssh/sshd_config on each node: Port 2222
2. UFW rule: sudo ufw allow 2222/tcp
3. Update ~/.ssh/config on operator: Port 2222
4. Restart sshd: sudo systemctl restart ssh
5. Remove UFW rule for port 22 after verifying

Effort: 30 min (and pray).

Lower priority

Prometheus + Grafana

Why: Historical metrics, dashboards, alerting.

How: kube-prometheus-stack Helm chart. Adds ~500 MB RAM across cluster.

Effort: 4-6 hours including dashboard setup.

Loki log aggregation

Why: Cross-pod log queries, longer retention.

How: grafana/loki + promtail DaemonSet. Integrates with existing Grafana.

Effort: 2-3 hours.

OpenTelemetry tracing

Why: Request-level profiling. Show which hop dominates p99 latency.

How: Add OpenTelemetry SDK to Go app; export to Jaeger/Tempo.

Effort: 8-12 hours including tuning.

Hetzner private network

Why: Currently all inter-node traffic (including Flannel overlay) goes over public network. Private network = less attack surface, no bandwidth costs (if metered in future).

How: Attach Hetzner vswitch to the 3 nodes, reconfigure Flannel to advertise private IPs, update UFW rules to allow from private IP range instead of specific public IPs.

Effort: 2-3 hours including testing Flannel reconfig.

Move secrets to Vault

Why: Kubernetes Secrets are base64-encoded etcd values. Vault is purpose-built for secret management with audit logs, dynamic secrets, rotation policies.

How: Deploy Vault in the cluster (or external), migrate secret values, use Vault Agent Injector or External Secrets Operator.

Effort: 6-8 hours.

Not high priority until we have multiple engineers who shouldn't see every secret, or compliance requirements.

Automated backups to B2

Why: Neon's backup is Neon's problem. If Neon-as-a-company disappeared, we'd lose everything.

How: Nightly pg_dump | gzip | aws s3 cp (via s3cmd for B2) as a CronJob in the cluster.

Effort: 2 hours.

Multi-region

Why: ~100 ms CF→origin hop could be reduced by having origins in multiple regions. Not needed at current scale.

How: Add 2 more Hetzner nodes in ash (Ashburn, US). Separate k3s cluster (or one stretched cluster — painful). Cloudflare Load Balancing for geo-based routing.

Effort: Days of work, doubling cost. Don't until traffic justifies.

CF Workers for static + caching

Why: Certain endpoints (the marketing landing page, public API lookups) could serve from CF Workers with near-zero origin load.

How: Move static pages to Cloudflare Pages; cache API responses with Cache-Control: public, max-age=300.

Effort: 4-6 hours.

WireGuard-encrypted overlay

Why: Current Flannel VXLAN is plaintext between nodes. An attacker with Hetzner-internal network access could read pod-to-pod traffic.

How: K3s supports --flannel-backend=wireguard-native. Reinstall k3s server on each node with the new backend.

Effort: 2-3 hours (requires brief downtime).

Scaling triggers

Trigger	Action
p99 latency > 500ms sustained	Investigate with tracing; consider CF Workers for cached paths
API CPU > 70% sustained	HPA already configured; may need more nodes
DB connections at Neon limit	Upgrade Neon Scale or reduce DB_MAX_OPEN_CONNS
Redis memory > 80%	Scale Redis memory; consider cache sharding
B2 storage > 500 GB	Evaluate if R2 (free egress) is cheaper overall
Active users > 100k	Evaluate multi-region, CF Pro, paid monitoring
Revenue > $5k/mo	Hire ops help; this document assumes solo operator

Known gaps we accept

• No canary deploys: all-or-nothing rollouts via kubectl set image
• No feature flags (app-level): code is deployed as-is. Can't toggle features without re-deploying
• No A/B testing infra: out of scope for current product stage
• No Windows/tablet-specific CDN rules: CF serves everyone the same responses
• No explicit blue-green: rolling updates only

Stuff to delete when brave

• deploy/ (the Swarm era) — once we've been on k3s 30 days
• Legacy UFW rules from the Swarm era (2377, 7946, 4789, ESP, 500, 3000) — they don't hurt but they're confusing
• deploy-k3s/manifests/secrets.yaml.example — we don't use this pattern, we create secrets imperatively

Stuff that could go wrong and we should plan for

• Hetzner price hike: 2026-04-01 already happened. If another one comes, we could migrate to Netcup or OVH for savings.
• Neon EOL free tier: Neon could change pricing policy. Fallback: self-host Postgres on a Hetzner box or migrate to Supabase.
• Cloudflare Free plan changes: CF could restrict Free features. Fallback: BunnyCDN, or raw nodes without CDN.
• Gitea host outage: If Gitea is down, deploys can't pull new images. Existing pods continue. For long outages, we'd cache images locally or temporarily push to Docker Hub.

Progress tracker

As items are done, mark them here. Think of this as a running changelog.

• k3s migration from Swarm (2026-04-24)
• Traefik DaemonSet + hostNetwork (2026-04-24)
• Admin seed via ADMIN_EMAIL + ADMIN_PASSWORD (2026-04-24)
• Documentation book (this doc set) (2026-04-24)
• Web client deployed at app.myhoneydue.com (2026-04-24) — Next.js 16 standalone, 3 replicas with PDB, proxy pattern to api, see Chapter 7.
• Admin URL-baking fix (2026-04-24) — Dockerfile ARG NEXT_PUBLIC_API_URL, .dockerignore hardening for admin/.env.*.
• Auto-seed initial data on first API boot (2026-04-24) — 20260414_seed_initial_data migration populates lookups, admin user, task templates. See commit 4ec4bbb.
• APNs wired up (2026-04-24) — Key ID 5L5BVF5G48, Team ID X86BR9WTLD, sandbox mode. Secret honeydue-apns-key, FEATURE_PUSH_ENABLED=true.
• Traefik middleware: security-headers + rate-limit attached to all three ingresses (2026-04-24). CSP is stripped from the middleware because the Go API sets its own.
• Admin liveness probe path fix (2026-04-24) — was hitting /admin/ (404) and crashlooping every ~90s for 6 hours before the bug was caught. Fixed to /.
• Network policies applied (2026-04-24) — default-deny + explicit allows. Traefik hostNetwork is matched via node IP ipBlocks, not namespaceSelector. See Chapter 5.
• Cloudflare Full (strict) SSL (2026-04-24) — Origin CA cert installed as cloudflare-origin-cert secret, ingresses have tls: blocks, CF mode flipped from Flexible. Both user↔CF and CF↔origin now TLS.
• UFW CF-IP allowlist on all 3 nodes (2026-04-24) — 15 IPv4 + 7 IPv6 CF ranges allow :443; Anywhere rules for :80 and :443 deleted. Direct-connect from non-CF IPs times out.
• All other items above