Add Prometheus metrics + vmagent push to obs.88oakapps.com
Backend CI / Test (push) Has been cancelled
Backend CI / Contract Tests (push) Has been cancelled
Backend CI / Build (push) Has been cancelled
Backend CI / Lint (push) Has been cancelled
Backend CI / Secret Scanning (push) Has been cancelled

Adds internal/prom package with histograms for HTTP, GORM, B2, APNs, and
FCM, wired into the Echo router (HTTPMiddleware + /metrics) and GORM via
statement-level callbacks (no ctx plumbing needed). Storage and push
clients call ObserveB2Upload / ObserveAPNsSend / ObserveFCMSend at the
network round-trip points.

Existing internal/monitoring metrics move to /metrics/legacy so the
canonical /metrics emits proper histogram buckets for p50/p95/p99 rollups.

deploy-k3s/manifests/observability/vmagent.yaml deploys a single-replica
vmagent in the honeydue namespace that scrapes api Pods on :8000/metrics
every 15s and remote-writes to https://obs.88oakapps.com/api/v1/write
with a bearer token (substituted at deploy time from OBS_INGEST_TOKEN
in deploy/prod.env). NetworkPolicies allow vmagent egress to api Pods
and to the public obs endpoint over :443; the obs side runs
VictoriaMetrics + Jaeger + Grafana on 88oakappsUpdate.

docs/observability-plan.md captures the full plan including resource
budget, instrumentation table, 4-step rollout, and migration triggers.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Trey t
2026-04-25 14:16:17 -05:00
parent 1cd6cafa9d
commit df78d9ccd8
10 changed files with 622 additions and 3 deletions
+9 -1
View File
@@ -13,6 +13,7 @@ import (
"github.com/rs/zerolog/log"
"github.com/treytartt/honeydue-api/internal/config"
"github.com/treytartt/honeydue-api/internal/prom"
)
// StorageService handles file uploads, validation, encryption, and URL generation.
@@ -149,11 +150,18 @@ func (s *StorageService) Upload(file *multipart.FileHeader, category string) (*U
}
}
// Write to backend
// Write to backend (B2/S3 round trip — instrumented for Prometheus)
bucket := s.cfg.S3Bucket
if bucket == "" {
bucket = "local"
}
uploadStart := time.Now()
if err := s.backend.Write(key, fileData); err != nil {
prom.ObserveB2Upload(bucket, "error", time.Since(uploadStart), 0)
return nil, fmt.Errorf("failed to save file: %w", err)
}
written := int64(len(fileData))
prom.ObserveB2Upload(bucket, "ok", time.Since(uploadStart), written)
// Generate URL (always uses the original filename without .enc suffix)
url := fmt.Sprintf("%s/%s/%s", s.cfg.BaseURL, subdir, newFilename)