Trey t
77cfcc0b27
ch15 is now an account of what's actually running, not a roadmap for what we'd add: VictoriaMetrics + Jaeger + Grafana on 88oakappsUpdate fronted by Cloudflare and bearer-gated nginx, vmagent in-cluster, the internal/prom histogram set, the rollout's NetworkPolicy footprint, the obs.88oakapps.com endpoint shape, the ~$0/700MB resource budget, and a token-rotation runbook. The "what we still don't have" section keeps log aggregation, alerting, and full distributed tracing as the honest gap list. Other touched docs: - 00-overview: \"deliberately absent\" no longer claims we have no metrics — calls out the cross-cluster shape instead. - 14-deployment-process: TL;DR now points at deploy-k3s/scripts/03-deploy.sh (full build + push + apply + obs vmagent), with the manual kubectl-set-image flow kept as the single-service path. Notes the IfNotPresent gotcha that bit us during the rollout. - 16-failure-modes: adds vmagent-can't-reach-obs and Grafana-no-data. - 18-cost: $0 line item for the obs stack on 88oakappsUpdate, with the CX32 migration trigger. - 17/18 README + appendix b: link the new ch15, add the obs cheat sheet block. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
113 lines
5.4 KiB
Markdown
113 lines
5.4 KiB
Markdown
# honeyDue Production Deployment — The Book
|
||
|
||
This is the complete reference for the honeyDue production deployment as it
|
||
exists on **2026-04-24**. It serves two audiences:
|
||
|
||
1. **A new engineer** learning the system for the first time. Start at
|
||
Chapter 0 (Overview) and read in order. Concepts are built up; nothing is
|
||
assumed beyond "you've deployed web apps before."
|
||
2. **The operator** (future-you) needing a specific fact fast. Every chapter
|
||
opens with a one-paragraph summary and has an operator runbook at its end.
|
||
The appendices are a cheat sheet.
|
||
|
||
The deployment is non-trivial. It's a 3-node HA Kubernetes cluster running
|
||
a Go API, a Next.js admin panel, a background worker, Redis, and Traefik —
|
||
all fronted by Cloudflare, integrated with Neon Postgres, Backblaze B2, and
|
||
a self-hosted Gitea registry. This book explains **why each of those pieces
|
||
was chosen** (often over two or three alternatives we tried first), what
|
||
they do, and how to operate them.
|
||
|
||
## Table of Contents
|
||
|
||
### Part I — The System
|
||
|
||
- [00 — Overview](./00-overview.md) — what's running, at a glance
|
||
- [01 — Infrastructure](./01-infrastructure.md) — Hetzner nodes, specs, cost, region
|
||
- [02 — Orchestrator Choice](./02-orchestrator-choice.md) — why k3s (and not Swarm, full k8s, or Nomad)
|
||
|
||
### Part II — Networking
|
||
|
||
- [03 — Networking](./03-networking.md) — flannel, CoreDNS, kube-proxy, the overlay story
|
||
- [04 — Firewall](./04-firewall.md) — every UFW rule on every node, rationale
|
||
- [13 — Cloudflare](./13-cloudflare.md) — DNS, SSL modes, round-robin origin pool
|
||
|
||
### Part III — Security
|
||
|
||
- [05 — Security](./05-security.md) — RBAC, Pod Security, secrets, TLS chain
|
||
- [06 — Traefik Ingress](./06-traefik-ingress.md) — host-network DaemonSet, cert plan
|
||
|
||
### Part IV — Workloads
|
||
|
||
- [07 — Services](./07-services.md) — api, admin, worker, redis per-service deep dive
|
||
- [08 — Database](./08-database.md) — Neon Postgres, advisory-lock migrations
|
||
- [09 — Storage](./09-storage.md) — Backblaze B2, minio-go client details
|
||
- [10 — Secrets & Config](./10-secrets-config.md) — ConfigMap, Secret, env mapping
|
||
- [11 — Registry](./11-registry.md) — Gitea container registry, multi-arch builds
|
||
|
||
### Part V — Operation
|
||
|
||
- [12 — Data Flow](./12-data-flow.md) — end-to-end request lifecycle
|
||
- [14 — Deployment Process](./14-deployment-process.md) — how to roll new code
|
||
- [15 — Observability](./15-observability.md) — VictoriaMetrics + Jaeger + Grafana on `obs.88oakapps.com`, vmagent in-cluster, Prometheus histograms in the Go API
|
||
- [16 — Failure Modes](./16-failure-modes.md) — what happens when X dies
|
||
- [17 — Runbook](./17-runbook.md) — common ops tasks
|
||
|
||
### Part VI — Context
|
||
|
||
- [18 — Cost](./18-cost.md) — what this costs to run, per service
|
||
- [19 — Swarm Postmortem](./19-postmortem-swarm.md) — the story of why we migrated from Docker Swarm
|
||
- [20 — Roadmap](./20-roadmap.md) — known TODOs and scaling triggers
|
||
|
||
### Appendices
|
||
|
||
- [A — Glossary](./appendices/a-glossary.md)
|
||
- [B — kubectl Cheat Sheet](./appendices/b-commands.md)
|
||
- [C — File Locations](./appendices/c-file-locations.md)
|
||
- [D — References & Citations](./appendices/d-references.md)
|
||
|
||
## Quick Facts
|
||
|
||
| Field | Value |
|
||
|---|---|
|
||
| Orchestrator | K3s v1.34.6+k3s1 (3 nodes, HA control plane) |
|
||
| Ingress | Traefik v3 (DaemonSet, hostNetwork) |
|
||
| Nodes | 3× Hetzner Cloud CX33 (4 vCPU, 8 GB RAM, 80 GB SSD) in `nbg1` (Nuremberg) |
|
||
| DNS & Edge | Cloudflare (Free plan), SSL=Flexible, round-robin 3 node A records |
|
||
| Database | Neon Postgres, `ep-floral-truth-amttbc5a.c-5.us-east-1.aws.neon.tech` |
|
||
| Cache + Queue | Redis 7-alpine, in-cluster, 1 replica, PVC-backed, pinned to `nbg1-2` |
|
||
| Object Storage | Backblaze B2, `honeyDueProd` bucket, `us-east-005` region |
|
||
| Image Registry | Self-hosted Gitea v1.25.5 at `gitea.treytartt.com` |
|
||
| Transactional Email | Fastmail SMTP (`smtp.fastmail.com:587`) |
|
||
| Domains | `api.myhoneydue.com`, `admin.myhoneydue.com`, `myhoneydue.com` |
|
||
| Monthly Cost (current) | ~$30–40 (3× Hetzner + Neon Launch + B2 + Cloudflare Free + Gitea free) |
|
||
| kubeconfig | `~/.kube/honeydue-k3s.yaml` on operator workstation |
|
||
| Repo | `honeyDueAPI-go/deploy-k3s/` for manifests, `deploy/` is the legacy Swarm config |
|
||
|
||
## How to Read This Book
|
||
|
||
- **"Why did we…?"** answers are in the chapter covering that component. Every
|
||
major design choice has an explicit rejection of 1–3 alternatives.
|
||
- **Historical bugs** are in Chapter 19. The rest of the book describes the
|
||
current (fixed) state; 19 is the forensic record of what was broken and
|
||
how we figured it out.
|
||
- **Operator commands** you'll run regularly are in Appendix B. Chapter 17
|
||
has longer procedures (cert rotation, DB migration, etc.).
|
||
- **Citations** throughout use footnote-style links to the canonical source
|
||
(k3s docs, moby issues, Cloudflare docs, etc.). Appendix D collects them.
|
||
|
||
## Conventions
|
||
|
||
- Kubernetes namespace for the app is `honeydue`.
|
||
- SSH aliases are `hetzner1`, `hetzner2`, `hetzner3` in your `~/.ssh/config`.
|
||
- Node hostnames in the cluster are `ubuntu-8gb-nbg1-{1,2,3}` (Hetzner-assigned).
|
||
- The mapping is non-obvious because the Hetzner hostname suffix order does
|
||
not match SSH alias order:
|
||
|
||
| SSH alias | Public IP | Hostname in k3s |
|
||
|---|---|---|
|
||
| hetzner1 | 178.104.247.152 | `ubuntu-8gb-nbg1-2` |
|
||
| hetzner2 | 178.105.32.198 | `ubuntu-8gb-nbg1-1` |
|
||
| hetzner3 | 178.104.249.189 | `ubuntu-8gb-nbg1-3` |
|
||
|
||
When a chapter refers to "hetzner1" it means the box at 178.104.247.152 / `nbg1-2`.
|