fix(observability): unbreak vmagent SD on fresh deploy + ship kube-state-metrics

vmagent's k8s service discovery has been silently broken for 17+ days
because k3s's NetworkPolicy controller evaluates egress AFTER kube-proxy's
DNAT (contrary to the k8s spec). Pod → ClusterIP 10.43.0.1:443 was
DNAT'd to <node_public_ip>:6443, and the resulting :6443 destination
matched none of vmagent's egress rules → TCP RST → "connection refused"
on every SD watch attempt. Grafana panels using kube_* or up{} metrics
returned empty as a result.

Changes:

- network-policies.yaml: commit the previously-cluster-only NetPols
  (allow-egress-from-vmagent, allow-vmagent-to-api) so a fresh deploy
  produces a working cluster. The vmagent egress rule now includes :6443
  to public IPs (the post-DNAT path) and :8080 to the pod CIDR (for
  scraping kube-state-metrics).

- observability/kube-state-metrics.yaml: new manifest. Provides the
  kube_pod_*, kube_deployment_*, kube_service_* metrics that Grafana
  panels need to count pods, replicas, etc. Runs in kube-system with
  cluster-scoped RBAC.

- observability/vmagent.yaml:
  * add kube-state-metrics scrape job to the ConfigMap
  * add vmagent-kube-system Role+RoleBinding so cross-namespace SD works
  * replace the misleading liveness probe (was /-/healthy, which lies
    while SD is broken) with an exec probe that checks /api/v1/targets
    for at least one healthy target — automatic recovery from future
    stale-SD incidents

- scripts/03-deploy.sh: actually apply network-policies.yaml (was
  committed but never applied) and apply kube-state-metrics.yaml.

- RUNBOOK.md (new): documents the post-DNAT gotcha, the liveness probe
  trap, bearer-token recovery procedure, drift-detection diff, and a
  post-redeploy verification checklist.

- .gitignore: cover kubeconfig.tunnel (created during SSH-tunnelled
  kubectl sessions) so admin client cert can't be committed by accident.

Verified via kubectl --dry-run on all three modified manifests.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

deploy-k3s/manifests/observability/vmagent.yaml

@@ -42,6 +42,21 @@ data:
           - target_label: service
             replacement: api
 
+      # kube-state-metrics — cluster object state (kube_pod_*, kube_deployment_*,
+      # etc.) needed for Grafana panels that count pods/replicas/etc.
+      - job_name: kube-state-metrics
+        kubernetes_sd_configs:
+          - role: endpoints
+            namespaces:
+              names: [kube-system]
+        relabel_configs:
+          - source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_name]
+            action: keep
+            regex: kube-state-metrics
+          - source_labels: [__meta_kubernetes_endpoint_port_name]
+            action: keep
+            regex: http-metrics
+
       # honeyDue worker — also exposes /metrics if/when we add it.
       # Keep this stanza commented until the worker has a /metrics endpoint;
       # uncommented form drops scrapes silently.
@@ -104,6 +119,35 @@ roleRef:
   name: vmagent
   apiGroup: rbac.authorization.k8s.io
 
+---
+# Allow vmagent to discover the kube-state-metrics Service/Endpoints in
+# kube-system so the kube-state-metrics scrape job can find its target.
+# Cross-namespace SD needs an explicit RoleBinding here.
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: vmagent-kube-system
+  namespace: kube-system
+rules:
+  - apiGroups: [""]
+    resources: [services, endpoints, pods]
+    verbs: [get, list, watch]
+
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: vmagent-kube-system
+  namespace: kube-system
+subjects:
+  - kind: ServiceAccount
+    name: vmagent
+    namespace: honeydue
+roleRef:
+  kind: Role
+  name: vmagent-kube-system
+  apiGroup: rbac.authorization.k8s.io
+
 ---
 apiVersion: apps/v1
 kind: Deployment
@@ -162,12 +206,31 @@ spec:
               readOnly: true
             - name: buffer
               mountPath: /tmp/vmagent
-          livenessProbe:
+          # Process startup gate. /-/healthy returns 200 once vmagent has
+          # parsed config — gives the agent up to 2 min to come up before
+          # liveness starts evaluating.
+          startupProbe:
             httpGet:
               path: /-/healthy
               port: http
-            initialDelaySeconds: 10
-            periodSeconds: 30
+            initialDelaySeconds: 5
+            periodSeconds: 5
+            failureThreshold: 24
+          # Real liveness check: are scrapes actually succeeding?
+          # /-/healthy was the old probe and returned 200 for 17 days even
+          # while vmagent had zero healthy targets (stale k8s SD watch).
+          # This exec probe queries vmagent's own targets API and fails if
+          # NO target is in state "up". Three consecutive failures (3 min)
+          # → kubelet kills the pod → fresh SD watch.
+          livenessProbe:
+            exec:
+              command:
+                - sh
+                - -c
+                - 'n=$(wget -qO- http://localhost:8429/api/v1/targets 2>/dev/null | grep -c ''"health":"up"''); [ "$n" -gt 0 ]'
+            initialDelaySeconds: 120
+            periodSeconds: 60
+            failureThreshold: 3
           readinessProbe:
             httpGet:
               path: /-/healthy