admin/honeyDueAPI
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(observability): ship pod logs to Loki via Grafana Alloy
Backend CI / Test (push) Has been cancelled
Details
Backend CI / Contract Tests (push) Has been cancelled
Details
Backend CI / Lint (push) Has been cancelled
Details
Backend CI / Secret Scanning (push) Has been cancelled
Details
Backend CI / Build (push) Has been cancelled
Details

Browse Source 
Adds a Grafana Alloy DaemonSet that tails honeydue-namespace pod logs
from /var/log/pods and pushes them to Loki at obs.88oakapps.com,
reusing the existing OBS_INGEST_TOKEN (14-day retention).

- deploy-k3s/manifests/observability/alloy-logs.yaml — DaemonSet + RBAC
  + token Secret + Alloy config. Runs as root (/var/log/pods is 0750
  root:root) but otherwise locked down: all caps dropped, read-only
  root filesystem, seccomp RuntimeDefault, read-only hostPath mount.
- network-policies.yaml — allow-egress-from-alloy-logs (DNS + k8s API
  + obs HTTPS), mirroring the vmagent egress policy.
- 03-deploy.sh — applies alloy-logs with the OBS_INGEST_TOKEN
  substitution and waits for the DaemonSet rollout.

The Loki container, nginx /loki/api/v1/push route, and Grafana Loki
datasource live on the obs server and are not repo-managed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Trey t
2026-05-17 20:04:09 -05:00
parent c77ff07ce9
commit 93fddc3769
3 changed files with 320 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
deploy-k3s/scripts/03-deploy.sh
+9 -1
View File
@@ -253,9 +253,14 @@ if [[ -d "${MANIFESTS}/observability" ]]; then
# under deploy-k3s/. It's gitignored — operator copies values there once.
OBS_TOKEN="$(grep -E '^OBS_INGEST_TOKEN=' "${REPO_DIR}/deploy/prod.env" 2>/dev/null | cut -d= -f2- || true)"
if [[ -z "${OBS_TOKEN}" ]]; then
warn "OBS_INGEST_TOKEN not found in deploy/prod.env — skipping vmagent apply"
warn "OBS_INGEST_TOKEN not found in deploy/prod.env — skipping vmagent + alloy-logs apply"
else
sed "s|TOKEN_PLACEHOLDER|${OBS_TOKEN}|" "${MANIFESTS}/observability/vmagent.yaml" | kubectl apply -f -
# alloy-logs — DaemonSet that tails honeydue pod logs and pushes them to
# Loki at obs.88oakapps.com. Same OBS_INGEST_TOKEN as vmagent.
if [[ -f "${MANIFESTS}/observability/alloy-logs.yaml" ]]; then
sed "s|TOKEN_PLACEHOLDER|${OBS_TOKEN}|" "${MANIFESTS}/observability/alloy-logs.yaml" | kubectl apply -f -
fi
fi
fi
@@ -273,6 +278,9 @@ fi
if kubectl -n "${NAMESPACE}" get deployment vmagent >/dev/null 2>&1; then
kubectl rollout status deployment/vmagent -n "${NAMESPACE}" --timeout=120s
fi
if kubectl -n "${NAMESPACE}" get daemonset alloy-logs >/dev/null 2>&1; then
kubectl rollout status daemonset/alloy-logs -n "${NAMESPACE}" --timeout=120s
fi
# --- Done ---