docs: rewrite ch15 observability + cross-refs for the live obs stack
Backend CI / Test (push) Has been cancelled
Backend CI / Contract Tests (push) Has been cancelled
Backend CI / Build (push) Has been cancelled
Backend CI / Lint (push) Has been cancelled
Backend CI / Secret Scanning (push) Has been cancelled

ch15 is now an account of what's actually running, not a roadmap for
what we'd add: VictoriaMetrics + Jaeger + Grafana on 88oakappsUpdate
fronted by Cloudflare and bearer-gated nginx, vmagent in-cluster, the
internal/prom histogram set, the rollout's NetworkPolicy footprint,
the obs.88oakapps.com endpoint shape, the ~$0/700MB resource budget,
and a token-rotation runbook. The "what we still don't have" section
keeps log aggregation, alerting, and full distributed tracing as the
honest gap list.

Other touched docs:
- 00-overview: \"deliberately absent\" no longer claims we have no
  metrics — calls out the cross-cluster shape instead.
- 14-deployment-process: TL;DR now points at deploy-k3s/scripts/03-deploy.sh
  (full build + push + apply + obs vmagent), with the manual
  kubectl-set-image flow kept as the single-service path. Notes the
  IfNotPresent gotcha that bit us during the rollout.
- 16-failure-modes: adds vmagent-can't-reach-obs and Grafana-no-data.
- 18-cost: $0 line item for the obs stack on 88oakappsUpdate, with the
  CX32 migration trigger.
- 17/18 README + appendix b: link the new ch15, add the obs cheat
  sheet block.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Trey t
2026-04-25 15:05:06 -05:00
parent d3708e6c72
commit 77cfcc0b27
8 changed files with 414 additions and 187 deletions
+15
View File
@@ -58,6 +58,20 @@ honeyDue.
|---|---:|
| Gitea container registry | **$0** |
### Observability (88oakappsUpdate)
VictoriaMetrics + Jaeger + Grafana co-tenant on the existing Linode
VPS that hosts PostHog. ~700 MB RAM, 21 GB disk — fits inside the
existing instance. Not charged to honeyDue.
| Item | Monthly |
|---|---:|
| Self-hosted obs stack on `88oakappsUpdate` | **$0** |
Migration trigger: when the obs stack starts pressuring PostHog or
needs hard isolation, move to a dedicated Hetzner CX32 (~$8/mo).
See [Chapter 15 — When to move off](./15-observability.md).
### Total infrastructure
| Category | Monthly |
@@ -67,6 +81,7 @@ honeyDue.
| Storage | ~$0.30 |
| Edge | $0 |
| Registry | $0 |
| Observability | $0 |
| **Total** | **~$30** |
## External SaaS