# honeyDue Production Deployment — The Book

This is the complete reference for the honeyDue production deployment as it
exists on **2026-04-24**. It serves two audiences:

1. **A new engineer** learning the system for the first time. Start at
   Chapter 0 (Overview) and read in order. Concepts are built up; nothing is
   assumed beyond "you've deployed web apps before."
2. **The operator** (future-you) needing a specific fact fast. Every chapter
   opens with a one-paragraph summary and has an operator runbook at its end.
   The appendices are a cheat sheet.

The deployment is non-trivial. It's a 3-node HA Kubernetes cluster running
a Go API, a Next.js admin panel, a background worker, Redis, and Traefik —
all fronted by Cloudflare, integrated with Neon Postgres, Backblaze B2, and
a self-hosted Gitea registry. This book explains **why each of those pieces
was chosen** (often over two or three alternatives we tried first), what
they do, and how to operate them.

## Table of Contents

### Part I — The System

- [00 — Overview](./00-overview.md) — what's running, at a glance
- [01 — Infrastructure](./01-infrastructure.md) — Hetzner nodes, specs, cost, region
- [02 — Orchestrator Choice](./02-orchestrator-choice.md) — why k3s (and not Swarm, full k8s, or Nomad)

### Part II — Networking

- [03 — Networking](./03-networking.md) — flannel, CoreDNS, kube-proxy, the overlay story
- [04 — Firewall](./04-firewall.md) — every UFW rule on every node, rationale
- [13 — Cloudflare](./13-cloudflare.md) — DNS, SSL modes, round-robin origin pool

### Part III — Security

- [05 — Security](./05-security.md) — RBAC, Pod Security, secrets, TLS chain
- [06 — Traefik Ingress](./06-traefik-ingress.md) — host-network DaemonSet, cert plan

### Part IV — Workloads

- [07 — Services](./07-services.md) — api, admin, worker, redis per-service deep dive
- [08 — Database](./08-database.md) — Neon Postgres, advisory-lock migrations
- [09 — Storage](./09-storage.md) — Backblaze B2, minio-go, presigned-URL direct uploads
- [10 — Secrets & Config](./10-secrets-config.md) — ConfigMap, Secret, env mapping
- [11 — Registry](./11-registry.md) — Gitea container registry, multi-arch builds

### Part V — Operation

- [12 — Data Flow](./12-data-flow.md) — end-to-end request lifecycle
- [14 — Deployment Process](./14-deployment-process.md) — how to roll new code
- [15 — Observability](./15-observability.md) — VictoriaMetrics + Jaeger + Grafana on `obs.88oakapps.com`, vmagent in-cluster, Prometheus histograms in the Go API
- [16 — Failure Modes](./16-failure-modes.md) — what happens when X dies
- [17 — Runbook](./17-runbook.md) — common ops tasks

### Part VI — Context

- [18 — Cost](./18-cost.md) — what this costs to run, per service
- [19 — Swarm Postmortem](./19-postmortem-swarm.md) — the story of why we migrated from Docker Swarm
- [20 — Roadmap](./20-roadmap.md) — known TODOs and scaling triggers

### Appendices

- [A — Glossary](./appendices/a-glossary.md)
- [B — kubectl Cheat Sheet](./appendices/b-commands.md)
- [C — File Locations](./appendices/c-file-locations.md)
- [D — References & Citations](./appendices/d-references.md)

## Quick Facts

| Field | Value |
|---|---|
| Orchestrator | K3s v1.34.6+k3s1 (3 nodes, HA control plane) |
| Ingress | Traefik v3 (DaemonSet, hostNetwork) |
| Nodes | 3× Hetzner Cloud CX33 (4 vCPU, 8 GB RAM, 80 GB SSD) in `nbg1` (Nuremberg) |
| DNS & Edge | Cloudflare (Free plan), SSL=Flexible, round-robin 3 node A records |
| Database | Neon Postgres, `ep-floral-truth-amttbc5a.c-5.us-east-1.aws.neon.tech` |
| Cache + Queue | Redis 7-alpine, in-cluster, 1 replica, PVC-backed, pinned to `nbg1-2` |
| Object Storage | Backblaze B2, `honeyDueProd` bucket, `us-east-005` region |
| Image Registry | Self-hosted Gitea v1.25.5 at `gitea.treytartt.com` |
| Transactional Email | Fastmail SMTP (`smtp.fastmail.com:587`) |
| Domains | `api.myhoneydue.com`, `admin.myhoneydue.com`, `myhoneydue.com` |
| Monthly Cost (current) | ~$30–40 (3× Hetzner + Neon Launch + B2 + Cloudflare Free + Gitea free) |
| kubeconfig | `~/.kube/honeydue-k3s.yaml` on operator workstation |
| Repo | `honeyDueAPI-go/deploy-k3s/` for manifests, `deploy/` is the legacy Swarm config |

## How to Read This Book

- **"Why did we…?"** answers are in the chapter covering that component. Every
  major design choice has an explicit rejection of 1–3 alternatives.
- **Historical bugs** are in Chapter 19. The rest of the book describes the
  current (fixed) state; 19 is the forensic record of what was broken and
  how we figured it out.
- **Operator commands** you'll run regularly are in Appendix B. Chapter 17
  has longer procedures (cert rotation, DB migration, etc.).
- **Citations** throughout use footnote-style links to the canonical source
  (k3s docs, moby issues, Cloudflare docs, etc.). Appendix D collects them.

## Conventions

- Kubernetes namespace for the app is `honeydue`.
- SSH aliases are `hetzner1`, `hetzner2`, `hetzner3` in your `~/.ssh/config`.
- Node hostnames in the cluster are `ubuntu-8gb-nbg1-{1,2,3}` (Hetzner-assigned).
- The mapping is non-obvious because the Hetzner hostname suffix order does
  not match SSH alias order:

| SSH alias | Public IP | Hostname in k3s |
|---|---|---|
| hetzner1 | 178.104.247.152 | `ubuntu-8gb-nbg1-2` |
| hetzner2 | 178.105.32.198 | `ubuntu-8gb-nbg1-1` |
| hetzner3 | 178.104.249.189 | `ubuntu-8gb-nbg1-3` |

When a chapter refers to "hetzner1" it means the box at 178.104.247.152 / `nbg1-2`.