Manthan Testnet Observability Dashboard
Per Manthan G-012. The metrics emitted at each node'sGET /metricsendpoint power this dashboard. Configure your Prometheus + Grafana stack to scrape/metricsevery 15s.
Top-line panels
1. Network health
| Panel | Query | What it shows |
|---|---|---|
| Active sessions | sum by (node) (manthan_session_count) | Live MWP sessions per node |
| Peer set size | sum by (node) (manthan_peer_count) | Bounded peer set after PEER_EXCHANGE absorption |
| Charter freshness | manthan_charter_age_seconds / 86400 (in days) | Days since Charter was signed |
| Uptime | manthan_node_uptime_seconds / 3600 (in hours) | Process uptime |
2. Gossip throughput
| Panel | Query | What it shows |
|---|---|---|
| Envelopes received/sec | rate(manthan_envelopes_received_total[1m]) | Inbound gossip rate |
| Envelopes sent/sec | rate(manthan_envelopes_sent_total[1m]) | Origination rate |
| Dedupe ratio | rate(manthan_envelopes_deduped_total[5m]) / rate(manthan_envelopes_received_total[5m]) | Fraction of inbound gossip already-seen |
| Rate-limit drops | rate(manthan_envelopes_rate_limited_total[5m]) | Per-peer fanout drops due to rate cap |
3. Ledger state
| Panel | Query | What it shows |
|---|---|---|
| Envelope state distribution | manthan_ledger_state (group by state) | How many envelopes are in each state machine bucket |
| Quarantine count | sum(manthan_ledger_state{state="quarantined"}) | Should be 0 in a healthy testnet |
| Verified rate | rate(manthan_envelopes_verified_total[5m]) | How fast inbound envelopes pass verification |
Alert priorities
Severities map directly to LOTS §8 incident taxonomy:
- SEV-1 (log integrity affected):
ManthanCharterStale,ManthanPeerCollapse,ManthanReadinessFail,ManthanLivenessLost. PagerDuty 24/7. - SEV-2 (availability impacted):
ManthanRepeatedVerifierFailures,ManthanLedgerCorruption. PagerDuty business hours. - SEV-3 (operational warning):
ManthanCharterStaleSoon,ManthanRateLimitSaturation,ManthanLowSessionCount,ManthanDedupeRateHigh. Slack only.
Sample Grafana JSON
A starter dashboard JSON lives at services/testnet/observability/grafana-dashboard.json (TBD — produce from the panels above when wiring AWS Managed Grafana). For the local-service testnet, run Grafana locally and import the JSON.
What to watch during incidents
Per incident-response/runbook.md:
- Charter refresh:
manthan_charter_age_secondsshould drop to ~0 after re-sign. - Peer collapse:
manthan_peer_countandmanthan_session_countshould recover within 5 minutes after fixing connectivity. - Verifier failures:
manthan_envelopes_quarantined_totalrate should drop to 0 after ejecting the offending peer. - Ledger corruption:
manthan_ledger_state{state="quarantined"}should be 0 after restore from backup.