honeyDueAPI/deploy-k3s/RUNBOOK.md

# k3s Cluster Operations Runbook

Living document for honeyDue k3s cluster operations. Add entries when you
hit something non-obvious so future-you (or your replacement) doesn't have
to rediscover it.

---

## Deployment

The canonical deploy path is `deploy-k3s/scripts/03-deploy.sh`. It applies
everything in `deploy-k3s/manifests/` in the right order.

What it touches (in order):

1. `namespace.yaml`
2. `network-policies.yaml` — **all** NetPols including the vmagent ones
3. `redis/`
4. `ingress/`
5. `migrate/job.yaml` (with image substitution; blocks on success)
6. `api/deployment.yaml`, `api/service.yaml`, `api/hpa.yaml` (image-subbed)
7. `worker/deployment.yaml` (image-subbed)
8. `admin/deployment.yaml`, `admin/service.yaml` (image-subbed)
9. `web/deployment.yaml`, `web/service.yaml` (image-subbed; optional dir)
10. `observability/kube-state-metrics.yaml`
11. `observability/vmagent.yaml` (with `TOKEN_PLACEHOLDER` sed-substituted from `deploy/prod.env`)

If you add a new manifest, also add a `kubectl apply -f` line to
`03-deploy.sh` — there's no kustomization or `apply -R`. **A manifest
that exists in the repo but isn't applied by the script will silently
not deploy.**

### Pre-deploy checklist

- [ ] `deploy/prod.env` exists and contains `OBS_INGEST_TOKEN=...`
       (otherwise vmagent gets skipped with a warning)
- [ ] `KUBECONFIG` points at the right cluster
- [ ] The Gitea image registry is reachable from k3s nodes
- [ ] Schema migrations in `migrations/` are tested locally first
       (the deploy aborts if `honeydue-migrate` Job fails)

---

## Known gotchas

### vmagent SD broken on fresh deploy ("0 pods up" in Grafana)

**Symptoms:**
- Grafana panels using `kube_*` metrics or `up{job=...}` show 0
- vmagent logs: `dial tcp 10.43.0.1:443: connect: connection refused`
       repeating every ~30s
- Direct test from a pod also refused: `kubectl -n honeydue exec deploy/vmagent
       -- wget --no-check-certificate -qO- -T 3 https://10.43.0.1:443/livez`

**Cause:** k3s's built-in NetworkPolicy controller evaluates egress rules
**after** kube-proxy's DNAT, not before (contrary to the k8s spec).
Traffic from a pod to the `kubernetes` Service (ClusterIP `10.43.0.1:443`)
gets DNAT'd to `<node_public_ip>:6443`, and **then** the policy check
runs. Without an explicit egress rule for `:6443`, the packet is rejected
with a TCP RST → "connection refused".

The `allow-egress-from-vmagent` NetPol in `network-policies.yaml` includes
both rules:

```yaml
# Pre-DNAT view (correct per spec; harmless if unused)
- to:
    - ipBlock: { cidr: 10.43.0.0/16 }
  ports:
    - { port: 443, protocol: TCP }
# Post-DNAT path (what k3s NetPol enforcer actually sees) — REQUIRED
- to:
    - ipBlock:
        cidr: 0.0.0.0/0
        except: [10.42.0.0/16]
  ports:
    - { port: 6443, protocol: TCP }
```

**If this happens on a fresh deploy:** confirm `network-policies.yaml`
was applied:
```bash
kubectl -n honeydue get netpol allow-egress-from-vmagent -o yaml
```
Look for the port-6443 egress rule. If missing, the apply step in
`03-deploy.sh` was skipped or the file was edited and the rule got
dropped.

**Counter-evidence that confirms diagnosis:** kube-state-metrics in
`kube-system` works fine, because `kube-system` has no NetPols. So if
ksm is healthy but workloads in `honeydue` can't reach the apiserver
ClusterIP, this gotcha is the cause.

---

### vmagent appears healthy but no data in Grafana

vmagent's `/-/healthy` endpoint returns 200 as long as the process is
alive and remote-write is functional (TCP-level) — it does **not**
check whether scrapes are succeeding. We saw this fail once: vmagent
was "healthy" for 17 days while having zero healthy targets due to a
broken k8s SD watch.

The liveness probe in `vmagent.yaml` queries the agent's `/api/v1/targets`
endpoint and fails the pod if no target is in state `up`. After 3
consecutive failures (~3 min), kubelet recycles the pod and SD restarts
clean.

**Verify it's working:** `kubectl -n honeydue describe pod -l app.kubernetes.io/name=vmagent`
should show `Liveness: exec [sh -c ...]`. If you ever see vmagent running
for weeks but no metrics in Grafana, the probe was disabled or the exec
command broke.

---

### vmagent's bearer token got blown away after `kubectl apply -f vmagent.yaml`

The committed `vmagent.yaml` has `bearer_token: TOKEN_PLACEHOLDER`. The
real token is sed-substituted at deploy time by `03-deploy.sh`. If you
ever apply `vmagent.yaml` directly:

```bash
kubectl apply -f deploy-k3s/manifests/observability/vmagent.yaml   # WRONG
```

the Secret gets overwritten with the literal string `TOKEN_PLACEHOLDER`
and all remote-writes start returning 401 from obs.88oakapps.com.

**To restore without a full redeploy** (the safe inline path):

```bash
export KUBECONFIG=...
OBS_TOKEN_B64=$(kubectl -n honeydue get secret honeydue-secrets \
                  -o jsonpath='{.data.OBS_INGEST_TOKEN}')
kubectl -n honeydue patch secret vmagent-remote-write --type=json \
  -p="[{\"op\":\"replace\",\"path\":\"/data/bearer_token\",\"value\":\"${OBS_TOKEN_B64}\"}]"
kubectl -n honeydue rollout restart deploy/vmagent
```

The OBS token also lives in `honeydue-secrets.OBS_INGEST_TOKEN` because
the api pods use it for traces — same secret, same value.

**Or just re-run the deploy:** `./deploy-k3s/scripts/03-deploy.sh`. The
sed step handles the substitution correctly.

---

### Node kubeconfig is world-readable

`/etc/rancher/k3s/k3s.yaml` is mode `0644` per the `--write-kubeconfig-mode=644`
k3s install flag. Any process on the host (including any container that
mounts the host filesystem) can read full cluster-admin credentials.

This is intentional for the deploy user but worth knowing — any container
escape becomes immediate cluster-admin. Tracked as finding **F4** in
`k3_audit_5_12.md`.

To tighten (if you ever turn this knob): change to `--write-kubeconfig-mode=600`
in the k3s install command, then re-fetch `deploy-k3s/kubeconfig`.

---

## Common operations

### Fetch a working kubectl tunnel (if `deploy-k3s/kubeconfig` is missing or stale)

```bash
ssh -i ~/.ssh/hetzner deploy@hetzner1 'sudo cat /etc/rancher/k3s/k3s.yaml' \
  | sed 's|server: https://127.0.0.1:6443|server: https://178.104.247.152:6443|' \
  > deploy-k3s/kubeconfig
chmod 600 deploy-k3s/kubeconfig
```

If the public :6443 is firewalled from your IP (the default — only
Cloudflare ranges are allowed for app traffic; admin is locked down):

```bash
# SSH tunnel — leave running in another terminal
ssh -fN -o ExitOnForwardFailure=yes -o ServerAliveInterval=30 \
    -i ~/.ssh/hetzner \
    -L 127.0.0.1:6443:127.0.0.1:6443 \
    deploy@hetzner1

# Then use a kubeconfig pointing at localhost
cp deploy-k3s/kubeconfig deploy-k3s/kubeconfig.tunnel
sed -i.bak 's|https://178.104.247.152:6443|https://127.0.0.1:6443|' \
  deploy-k3s/kubeconfig.tunnel
export KUBECONFIG="$(pwd)/deploy-k3s/kubeconfig.tunnel"
```

### Restore vmagent after a "0 targets" incident

```bash
export KUBECONFIG="$(pwd)/deploy-k3s/kubeconfig.tunnel"

# 1. Confirm the diagnosis
kubectl -n honeydue logs deploy/vmagent --tail=20 | grep -i "connect: connection refused"

# 2. Check the NetPol has the :6443 rule
kubectl -n honeydue get netpol allow-egress-from-vmagent -o yaml | grep -A 5 6443

# 3. If missing, re-apply
kubectl apply -f deploy-k3s/manifests/network-policies.yaml

# 4. Restart vmagent
kubectl -n honeydue rollout restart deploy/vmagent

# 5. Verify targets after ~60s
kubectl -n honeydue port-forward deploy/vmagent 8429:8429 &
curl -s http://localhost:8429/api/v1/targets \
  | python3 -c "import json,sys; d=json.load(sys.stdin); \
                a=d['data']['activeTargets']; \
                print(f'targets={len(a)} up={sum(1 for t in a if t[\"health\"]==\"up\")}')"
```

### Verify NetPols match the repo

If you suspect drift between cluster and repo:

```bash
diff <(kubectl -n honeydue get netpol -o name | sort) \
     <(grep -E '^\s*name: ' deploy-k3s/manifests/network-policies.yaml \
       | sed 's/.*name: /networkpolicy.networking.k8s.io\//' | sort)
```

Empty output = match. Any differences need investigation — either the
cluster has policies that aren't in repo (manual `kubectl apply` did it)
or repo has policies that didn't apply.

---

## Disaster recovery notes

### "I have to redeploy the whole stack"

The deploy path is designed to be re-runnable. From a fresh cluster:

1. Install k3s on all 3 nodes (use existing `deploy-k3s/scripts/01-install-k3s.sh`)
2. Fetch a kubeconfig (see "Common operations" above)
3. Confirm `deploy/prod.env` has all required secrets:
   - `POSTGRES_PASSWORD`, `SECRET_KEY`, `EMAIL_HOST_PASSWORD`,
     `FCM_SERVER_KEY`, `B2_KEY_ID`, `B2_APP_KEY`, `OBS_INGEST_TOKEN`,
     `OBS_TRACES_URL`, `REDIS_PASSWORD` (optional), `ADMIN_EMAIL`, `ADMIN_PASSWORD`
4. Run `./deploy-k3s/scripts/02-setup-secrets.sh` (creates `honeydue-secrets`)
5. Run `./deploy-k3s/scripts/03-deploy.sh` (deploys everything; sed-injects
   the obs token into vmagent at apply time)
6. Verify: `kubectl -n honeydue get pods` should show all workloads Running

### Post-redeploy verification checklist

- [ ] `kubectl -n honeydue get netpol` shows **12 NetPols** (default-deny +
       6 egress + 5 ingress)
- [ ] `kubectl -n honeydue get netpol allow-egress-from-vmagent -o yaml | grep 6443`
       returns the rule (if missing → see "vmagent SD broken" gotcha)
- [ ] `kubectl -n kube-system get pod -l app.kubernetes.io/name=kube-state-metrics`
       shows 1 Running pod
- [ ] `kubectl -n honeydue port-forward deploy/vmagent 8429:8429` + curl
       `localhost:8429/api/v1/targets` shows 4+ targets, all `up`
- [ ] Grafana panel "pods up" in `honeydue` namespace populates within 60s

If any of those fail, this runbook entry tells you exactly which gotcha
you hit.