The nervous system of the vessel: metrics, logs, and the alert loop
Thirteen scrape targets feed one Prometheus, one Loki holds the logs, one Grafana renders both and evaluates the alert rules — and every firing rule is routed not to a pager, but back through relayd's webhook into the model, so a critical alert gets judgment applied before it ever reaches Jean.
Data flow, exporter to admin
Open-Meteo,
NWS / weather.gov, WiGLE — not scraped directly, they write
textfile .prom collectors that ride the node job), host
(node, router), lab services
(qdrant, kafka, email_embed_worker, email_search,
govee_monitor, relayd), kubernetes (kube_state_metrics,
k8s_cadvisor via the node's kubelet proxy), and the observability stack's own
self-scrape (prometheus, grafana, loki) — 13 jobs total,
all healthy at last check. Node-level custom metrics ride node_exporter's textfile collector
from /var/lib/prometheus/node-exporter/*.prom (ssh_auth, tailscale, nvme, weather,
backup, kafka_consumer_lag, and more — over 20 small scripts). Promtail ships logs to Loki
independently of the metrics path. Grafana is the only consumer of both Prometheus and Loki,
and it alone evaluates alert rules; firing/resolved alerts are POSTed to relayd's
/webhook/grafana, loopback-only, which injects a prompt into the live Claude Code
session rather than paging Jean directly — the model decides urgency before anything reaches him.
Metrics pipeline
Prometheus (prometheus.service, active since 2026‑07‑17, ~830 MB RSS) scrapes 13 jobs every 15s (5s for its own self-scrape) per /etc/prometheus/prometheus.yml, retaining 400 days / 20GB on local TSDB.
Recording rules (vessel-recording-rules.yml) pre-aggregate SLOs: vessel_slo:up:avg7d,
vessel_slo:tailscale_self:avg7d/avg30d_from_weekly, vessel_slo:node_reboots:count7d/30d, and
weather derivatives (vessel:indoor_outdoor_temp_diff_fahrenheit) so dashboards and alert
conditions don't recompute long-window aggregates on every panel load.
Not every metric originates on this box. Three cron-run collector scripts call public APIs and
write straight into node_exporter's textfile directory, so their output rides the plain
node scrape rather than getting its own job: Open-Meteo
(api.open-meteo.com/forecast + archive, and air-quality-api.open-meteo.com)
via weather-groundtruth and aqi_check into
weather_groundtruth.prom / air_quality.prom — outdoor temp, feels-like,
humidity, US AQI; NWS / weather.gov (api.weather.gov/alerts/active) via
weather_alerts_check into weather_alerts.prom — severe-weather and
advisory alerts; and WiGLE (api.wigle.net) via wigle_locate into
wigle_location.prom.
Log pipeline (Loki)
loki.service (active since 2026‑07‑17) runs single-binary, filesystem storage under
/var/lib/loki, listening on :3100 (HTTP) / :9096 (gRPC). promtail.service tails
journald and files, pushing structured log lines with labels (e.g. job="usb_events").
Grafana's Loki datasource is queried both for dashboards and directly inside alert rule conditions —
e.g. usb-device-connect-disconnect evaluates
count_over_time({job="usb_events"} | logfmt [1m]) once a minute, so log-only events (no metric
series exists) can still fire an alert through the same rule engine as PromQL-based ones.
Dashboards
Grafana (grafana-server.service, restarted 2026‑08‑09, provisioned dashboards under
/etc/grafana/provisioning/dashboards/json/) is the single pane of glass over both datasources
— PromQL panels for metrics, LogQL panels for logs, plus an alert-history table
(vessel-alert-history.json) and a weather dashboard. Datasources and alert config are all
file-provisioned (not clicked-together), so the stack rebuilds itself from git/config on a fresh box.
The alert / nervous-system loop
Alert rules live in /etc/grafana/provisioning/alerting/, folder Vessel Alerts,
group vessel-critical, evaluated every 1m. A single contact point,
relay-webhook, POSTs every firing/resolved alert to
http://127.0.0.1:9210/webhook/grafana — loopback only, since Grafana and relayd share
this host. Notification policy groups by alertname; severity=critical gets
10s group-wait / 1h repeat, severity=degraded uses the looser default.
relayd's handler (cmd/relayd/main.go, serveGrafanaWebhook) doesn't page Jean
directly — it builds a prompt ("[Grafana alert, status=…, severity=…] name: summary … use your
judgment on how urgently to surface this…") and injects it into the live Claude Code session
via the same buffered-inject path used for scheduled triggers. The model decides whether a firing
critical alert needs an immediate ping, a resolved one just a brief note, or folds it into an
investigation already underway — then calls the relay reply tool itself. This is the
loop this whole document exists to describe: exporters → Prometheus/Loki → Grafana evaluates →
relayd webhook → model judgment → Jean.
A representative slice of the ~34 configured rules (grouped by folder,
not all currently loaded — some rotate through .bak staging files during edits):