feat(observability): ship pod logs to Loki via Grafana Alloy
Backend CI / Test (push) Has been cancelled
Backend CI / Contract Tests (push) Has been cancelled
Backend CI / Lint (push) Has been cancelled
Backend CI / Secret Scanning (push) Has been cancelled
Backend CI / Build (push) Has been cancelled

Adds a Grafana Alloy DaemonSet that tails honeydue-namespace pod logs
from /var/log/pods and pushes them to Loki at obs.88oakapps.com,
reusing the existing OBS_INGEST_TOKEN (14-day retention).

- deploy-k3s/manifests/observability/alloy-logs.yaml — DaemonSet + RBAC
  + token Secret + Alloy config. Runs as root (/var/log/pods is 0750
  root:root) but otherwise locked down: all caps dropped, read-only
  root filesystem, seccomp RuntimeDefault, read-only hostPath mount.
- network-policies.yaml — allow-egress-from-alloy-logs (DNS + k8s API
  + obs HTTPS), mirroring the vmagent egress policy.
- 03-deploy.sh — applies alloy-logs with the OBS_INGEST_TOKEN
  substitution and waits for the DaemonSet rollout.

The Loki container, nginx /loki/api/v1/push route, and Grafana Loki
datasource live on the obs server and are not repo-managed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Trey t
2026-05-17 20:04:09 -05:00
parent c77ff07ce9
commit 93fddc3769
3 changed files with 320 additions and 1 deletions
@@ -372,3 +372,57 @@ spec:
ports:
- port: 8000
protocol: TCP
---
# alloy-logs egress — Grafana Alloy discovers honeydue pods via the k8s API
# and pushes their logs to Loki at obs.88oakapps.com. Same k3s NetworkPolicy
# DNAT gotcha as vmagent: API-server traffic is policy-checked as
# dst=<node_public_ip>:6443, so an explicit :6443 rule is required.
# Alloy reads log FILES from a hostPath, so it needs no ingress and no
# egress to pod :8000/:8080 — only DNS, the API server, and obs HTTPS.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-egress-from-alloy-logs
namespace: honeydue
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: alloy-logs
policyTypes:
- Egress
egress:
# DNS (cluster-internal)
- to:
- namespaceSelector: {}
ports:
- port: 53
protocol: UDP
- port: 53
protocol: TCP
# k8s API server via ClusterIP (pre-DNAT view)
- to:
- ipBlock:
cidr: 10.43.0.0/16
ports:
- port: 443
protocol: TCP
# k8s API server post-DNAT (real path k3s NetPol enforcer sees) — REQUIRED
- to:
- ipBlock:
cidr: 0.0.0.0/0
except:
- 10.42.0.0/16
ports:
- port: 6443
protocol: TCP
# HTTPS to public (log push to obs.88oakapps.com via Cloudflare)
- to:
- ipBlock:
cidr: 0.0.0.0/0
except:
- 10.42.0.0/16
- 10.43.0.0/16
ports:
- port: 443
protocol: TCP