Trey t
1347ffadf5
09-storage.md:
- Replaced the "Upload flow" section. The previous text described the
multipart-via-API path that was removed in b7f8329. Now documents
the three-step direct-to-B2 flow (presign → POST to B2 → attach
via upload_ids[]) with an ASCII diagram and a server-side
enforcement-points table.
- Replaced the "Future: signed URLs" placeholder (since presigned
URLs are now the present, not the future).
- Added "Lifecycle and retention" subsections covering the
pending_uploads cleanup cron (worker, 30 * * * *), the B2 bucket
lifecycle as backstop (uploads/ prefix, 7-day hide + 1-day delete),
and the still-open user-deletion cascade gap.
14-deployment-process.md:
- Added a "One-time B2 bucket lifecycle (manual)" section explaining
why the rule can't live in the deploy script (B2's S3 lifecycle
API is partial), the exact rule to apply via the Backblaze
console, and a verification command.
docs/deployment/README.md:
- Updated the chapter 9 description to mention presigned-URL uploads.
README.md (root):
- Added a paragraph under "Object storage" pointing to the new
upload architecture and the relevant deployment-book chapters.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
113 lines
5.5 KiB
Markdown
113 lines
5.5 KiB
Markdown
# honeyDue Production Deployment — The Book
|
||
|
||
This is the complete reference for the honeyDue production deployment as it
|
||
exists on **2026-04-24**. It serves two audiences:
|
||
|
||
1. **A new engineer** learning the system for the first time. Start at
|
||
Chapter 0 (Overview) and read in order. Concepts are built up; nothing is
|
||
assumed beyond "you've deployed web apps before."
|
||
2. **The operator** (future-you) needing a specific fact fast. Every chapter
|
||
opens with a one-paragraph summary and has an operator runbook at its end.
|
||
The appendices are a cheat sheet.
|
||
|
||
The deployment is non-trivial. It's a 3-node HA Kubernetes cluster running
|
||
a Go API, a Next.js admin panel, a background worker, Redis, and Traefik —
|
||
all fronted by Cloudflare, integrated with Neon Postgres, Backblaze B2, and
|
||
a self-hosted Gitea registry. This book explains **why each of those pieces
|
||
was chosen** (often over two or three alternatives we tried first), what
|
||
they do, and how to operate them.
|
||
|
||
## Table of Contents
|
||
|
||
### Part I — The System
|
||
|
||
- [00 — Overview](./00-overview.md) — what's running, at a glance
|
||
- [01 — Infrastructure](./01-infrastructure.md) — Hetzner nodes, specs, cost, region
|
||
- [02 — Orchestrator Choice](./02-orchestrator-choice.md) — why k3s (and not Swarm, full k8s, or Nomad)
|
||
|
||
### Part II — Networking
|
||
|
||
- [03 — Networking](./03-networking.md) — flannel, CoreDNS, kube-proxy, the overlay story
|
||
- [04 — Firewall](./04-firewall.md) — every UFW rule on every node, rationale
|
||
- [13 — Cloudflare](./13-cloudflare.md) — DNS, SSL modes, round-robin origin pool
|
||
|
||
### Part III — Security
|
||
|
||
- [05 — Security](./05-security.md) — RBAC, Pod Security, secrets, TLS chain
|
||
- [06 — Traefik Ingress](./06-traefik-ingress.md) — host-network DaemonSet, cert plan
|
||
|
||
### Part IV — Workloads
|
||
|
||
- [07 — Services](./07-services.md) — api, admin, worker, redis per-service deep dive
|
||
- [08 — Database](./08-database.md) — Neon Postgres, advisory-lock migrations
|
||
- [09 — Storage](./09-storage.md) — Backblaze B2, minio-go, presigned-URL direct uploads
|
||
- [10 — Secrets & Config](./10-secrets-config.md) — ConfigMap, Secret, env mapping
|
||
- [11 — Registry](./11-registry.md) — Gitea container registry, multi-arch builds
|
||
|
||
### Part V — Operation
|
||
|
||
- [12 — Data Flow](./12-data-flow.md) — end-to-end request lifecycle
|
||
- [14 — Deployment Process](./14-deployment-process.md) — how to roll new code
|
||
- [15 — Observability](./15-observability.md) — VictoriaMetrics + Jaeger + Grafana on `obs.88oakapps.com`, vmagent in-cluster, Prometheus histograms in the Go API
|
||
- [16 — Failure Modes](./16-failure-modes.md) — what happens when X dies
|
||
- [17 — Runbook](./17-runbook.md) — common ops tasks
|
||
|
||
### Part VI — Context
|
||
|
||
- [18 — Cost](./18-cost.md) — what this costs to run, per service
|
||
- [19 — Swarm Postmortem](./19-postmortem-swarm.md) — the story of why we migrated from Docker Swarm
|
||
- [20 — Roadmap](./20-roadmap.md) — known TODOs and scaling triggers
|
||
|
||
### Appendices
|
||
|
||
- [A — Glossary](./appendices/a-glossary.md)
|
||
- [B — kubectl Cheat Sheet](./appendices/b-commands.md)
|
||
- [C — File Locations](./appendices/c-file-locations.md)
|
||
- [D — References & Citations](./appendices/d-references.md)
|
||
|
||
## Quick Facts
|
||
|
||
| Field | Value |
|
||
|---|---|
|
||
| Orchestrator | K3s v1.34.6+k3s1 (3 nodes, HA control plane) |
|
||
| Ingress | Traefik v3 (DaemonSet, hostNetwork) |
|
||
| Nodes | 3× Hetzner Cloud CX33 (4 vCPU, 8 GB RAM, 80 GB SSD) in `nbg1` (Nuremberg) |
|
||
| DNS & Edge | Cloudflare (Free plan), SSL=Flexible, round-robin 3 node A records |
|
||
| Database | Neon Postgres, `ep-floral-truth-amttbc5a.c-5.us-east-1.aws.neon.tech` |
|
||
| Cache + Queue | Redis 7-alpine, in-cluster, 1 replica, PVC-backed, pinned to `nbg1-2` |
|
||
| Object Storage | Backblaze B2, `honeyDueProd` bucket, `us-east-005` region |
|
||
| Image Registry | Self-hosted Gitea v1.25.5 at `gitea.treytartt.com` |
|
||
| Transactional Email | Fastmail SMTP (`smtp.fastmail.com:587`) |
|
||
| Domains | `api.myhoneydue.com`, `admin.myhoneydue.com`, `myhoneydue.com` |
|
||
| Monthly Cost (current) | ~$30–40 (3× Hetzner + Neon Launch + B2 + Cloudflare Free + Gitea free) |
|
||
| kubeconfig | `~/.kube/honeydue-k3s.yaml` on operator workstation |
|
||
| Repo | `honeyDueAPI-go/deploy-k3s/` for manifests, `deploy/` is the legacy Swarm config |
|
||
|
||
## How to Read This Book
|
||
|
||
- **"Why did we…?"** answers are in the chapter covering that component. Every
|
||
major design choice has an explicit rejection of 1–3 alternatives.
|
||
- **Historical bugs** are in Chapter 19. The rest of the book describes the
|
||
current (fixed) state; 19 is the forensic record of what was broken and
|
||
how we figured it out.
|
||
- **Operator commands** you'll run regularly are in Appendix B. Chapter 17
|
||
has longer procedures (cert rotation, DB migration, etc.).
|
||
- **Citations** throughout use footnote-style links to the canonical source
|
||
(k3s docs, moby issues, Cloudflare docs, etc.). Appendix D collects them.
|
||
|
||
## Conventions
|
||
|
||
- Kubernetes namespace for the app is `honeydue`.
|
||
- SSH aliases are `hetzner1`, `hetzner2`, `hetzner3` in your `~/.ssh/config`.
|
||
- Node hostnames in the cluster are `ubuntu-8gb-nbg1-{1,2,3}` (Hetzner-assigned).
|
||
- The mapping is non-obvious because the Hetzner hostname suffix order does
|
||
not match SSH alias order:
|
||
|
||
| SSH alias | Public IP | Hostname in k3s |
|
||
|---|---|---|
|
||
| hetzner1 | 178.104.247.152 | `ubuntu-8gb-nbg1-2` |
|
||
| hetzner2 | 178.105.32.198 | `ubuntu-8gb-nbg1-1` |
|
||
| hetzner3 | 178.104.249.189 | `ubuntu-8gb-nbg1-3` |
|
||
|
||
When a chapter refers to "hetzner1" it means the box at 178.104.247.152 / `nbg1-2`.
|