honeyDueAPI/docs/deployment/15-observability.md

15 — Observability

Summary

We have minimal observability today: kubectl logs, kubectl top, Cloudflare Analytics, and the Neon dashboard. No Prometheus, no Grafana, no centralized log aggregator, no APM. This is adequate for the current traffic volume (low) but is a known gap. This chapter documents what we have and what we'd add as traffic grows.

What we have

1. kubectl logs

Every container's stdout/stderr is captured by containerd and readable via kubectl:

# Live tail from all api pods
kubectl logs -n honeydue -l app.kubernetes.io/name=api -f --prefix

# Last 100 lines
kubectl logs -n honeydue -l app.kubernetes.io/name=api --tail=100

# Previous pod's logs (before the most recent restart)
kubectl logs -n honeydue <pod-name> --previous

# Events (not logs — k8s-level state changes)
kubectl get events -n honeydue --sort-by=.lastTimestamp

Retention: containerd rotates logs when they exceed 10 MB (default). Only the last ~20 MB of logs is retained per container, on-disk on the node. Once a pod is deleted, its logs are gone.

For persistent log access we'd need aggregation (see §what we'd add).

2. kubectl top

Pod and node resource usage via metrics-server:

kubectl top nodes
# NAME                CPU(cores)   CPU(%)   MEMORY(bytes)   MEMORY(%)
# ubuntu-8gb-nbg1-1   169m         4%       748Mi           9%
# ubuntu-8gb-nbg1-2   229m         5%       1043Mi          13%
# ubuntu-8gb-nbg1-3   124m         3%       770Mi           9%

kubectl top pods -n honeydue

Retention: In-memory only. Last few minutes of data. No historical view.

3. Cloudflare Analytics

CF Dashboard → Analytics & Logs. Per-zone stats:

• Requests per second
• Bandwidth
• Cache hit ratio
• Top HTTP status codes
• Top request paths
• Bot traffic score

All aggregated, no individual request traces. Good for spotting macro trends ("suddenly 10× more 502s today"), poor for debugging specific issues.

Free tier retention: 7 days of aggregate stats. Pro extends this.

4. Neon dashboard

Neon console → project → Monitoring:

• Compute utilization (CU-hours consumed)
• Query performance (slow queries)
• Active connections
• Storage usage

Good for "is the DB busy?" and "am I close to my free tier limit?" Not real-time.

5. Kubernetes events

kubectl get events shows cluster-level state changes: pod scheduling, failures, image pulls, probe failures. Useful for post-mortem on deploys.

Retention: events are stored in etcd but default to 1 hour.

What we don't have (the gap)

No log aggregation

Individual pod logs are on the node. For multi-pod debugging ("show me all api pod logs for user X") we have to:

# Query all at once with stern (if installed)
stern -n honeydue api

# Or for specific pod
kubectl logs -n honeydue <pod> | grep user_id=12345

This works but doesn't scale. Grep across 3 pods for a specific user_id is OK. Across 30 pods, intractable.

What we'd add: Loki — a lightweight log aggregator designed for k8s. ~$0 to self-host; integrates with Grafana for queries. Or Betterstack ($10/mo, hosted).

No metrics/dashboards

kubectl top tells us "is this pod hot right now?" but not "has CPU been climbing over the past hour?" We'd need:

• Prometheus — scrapes metrics from kubelet and pods' /metrics endpoints, stores time series
• Grafana — queries Prometheus, renders dashboards

K3s can install these via Helm in ~10 minutes. Adds ~500MB RAM to the cluster. Stability and operational load: moderate.

Alternative: Kubernetes Dashboard bundled with k3s (disabled by default). Minimal UI over the existing metrics API. Cheaper than Prometheus but less queryable.

No distributed tracing

"This request took 800ms — which hop was slow?" is currently unanswerable beyond "the DB query, probably." A real trace would show:

• TLS handshake time
• Traefik routing time
• Go handler time
• Postgres query time
• Redis call time
• Each B2 request time

We'd add OpenTelemetry to the Go app and export to Jaeger/Tempo. Work is moderate; value kicks in when we have complex request flows.

No alerting

No PagerDuty, no Slack webhooks, no email on "api is returning 500s." The operator finds out when users complain.

Cheapest fix: Uptime Kuma (self-hosted) or Better Stack Uptime (free for small teams). Ping https://api.myhoneydue.com/api/health/ every minute; alert if it fails.

No APM (Application Performance Monitoring)

No request-level profiling. We can't see "which endpoint has the highest p99 latency?" or "which SQL query is hot this week?"

Options: Datadog, New Relic, Honeycomb, self-hosted Tempo+Grafana. All are meaningful work to set up and cost $$$.

The app's logging conventions

The Go app uses zerolog and emits structured JSON:

{
  "level": "info",
  "time": "2026-04-24T05:29:40Z",
  "caller": "/app/cmd/api/main.go:189",
  "addr": ":8000",
  "message": "HTTP server listening"
}

Log levels: debug, info, warn, error, fatal. Controlled by DEBUG=true|false in ConfigMap (true sets level to debug, false sets level to info).

Every request is logged with:

• Method, path, status code
• Request ID (for correlating logs across pods)
• User ID (if authenticated)
• Latency

{
  "level": "info",
  "method": "GET",
  "path": "/api/tasks/",
  "status": 200,
  "latency_ms": 42,
  "user_id": 123,
  "request_id": "a6b5db35-..."
}

This is queryable by grep. Better with log aggregation.

Health endpoints

Each service exposes a health endpoint:

Service	Endpoint	What it checks
api	/api/health/	Process alive (doesn't verify DB)
admin	/	Next.js is up
worker	(none public)	Internal Asynq status

Health endpoints are shallow — they return 200 if the process is running and listening. They don't try to reach Postgres/Redis/etc. Rationale: if Postgres is briefly down, we don't want all api pods to start failing liveness and cascade-restart.

Dozzle (deprecated)

The Swarm era had Dozzle — a lightweight web UI for Docker logs. Accessible via SSH tunnel to the manager node. Not deployed on k3s; kubectl logs + stern fills the niche.

Kubernetes metrics the k8s API exposes

Even without Prometheus, these are queryable:

# Resource metrics (via metrics-server)
kubectl get --raw /apis/metrics.k8s.io/v1beta1/nodes
kubectl get --raw /apis/metrics.k8s.io/v1beta1/namespaces/honeydue/pods

# Core API (k8s state)
kubectl get --raw /api/v1/namespaces/honeydue/pods/<name>

# Kubelet metrics (per-node; requires tunneling)
kubectl get --raw /api/v1/nodes/<node>/proxy/metrics

If we ever spin up Prometheus, these are the endpoints it would scrape.

Future: what to add and when

Trigger	Add
10k+ daily users	Loki + Grafana for logs
100+ req/s sustained	Prometheus + Grafana for metrics
Performance incidents	OpenTelemetry tracing
Revenue > $5k/mo	Paid monitoring (Datadog or similar)
First production outage	Alerting to phone/Slack

The overall philosophy: observability is an investment that compounds. Add it before you need it, not after. But also don't over-invest at idle.

Next quarter: set up Uptime Kuma + Loki at minimum.

Checking what's installed

# In kube-system namespace
kubectl get pods -n kube-system
# Should see: coredns, metrics-server, traefik, local-path-provisioner,
# and some k3s-related helm install jobs

# In honeydue namespace
kubectl get pods -n honeydue
# api, admin, worker, redis

# No monitoring namespace (yet)
kubectl get namespaces
# default, honeydue, kube-node-lease, kube-public, kube-system

Operator cheat sheet

# Tail all logs in the namespace
kubectl logs -n honeydue --all-containers=true --tail=50 -l app.kubernetes.io/part-of=honeydue

# With stern (if installed: brew install stern)
stern -n honeydue .

# Follow specific pod, including previous runs
kubectl logs -n honeydue <pod> -f --previous=false

# Pod resource usage
kubectl top pods -n honeydue --sort-by=memory
kubectl top pods -n honeydue --sort-by=cpu

# Events (cluster-wide)
kubectl get events -A --sort-by=.lastTimestamp | tail -20

# Full state dump for a pod (debugging)
kubectl describe pod -n honeydue <pod> > /tmp/pod-dump.txt
kubectl logs -n honeydue <pod> > /tmp/pod-logs.txt

References

• Kubernetes metrics-server
• K3s metrics
• Loki
• Stern (multi-pod log tail)