Stale per-partition gauge series persist after partition migrates off a node #4802

Summary

Per-partition gauge metrics are not cleaned up when a partition stops or migrates off a node. The Prometheus recorder retains the last-written value indefinitely, causing stale series to accumulate and producing incorrect per-node aggregations.

Observed on main.

Affected metrics

All of these will retain stale label combinations after a partition moves off a node:

• restate.partition.is_effective_leader{partition=N} -- frozen at last value (0 or 1)
• restate.partition.time_since_last_status_update{partition=N} -- frozen at last elapsed duration
• restate.partition.applied_lsn_lag{partition=N} -- frozen at last lag value
• restate.partition.snapshot_age.seconds{partition=N} -- frozen at last snapshot age
• restate.partition.blocked_flare{partition=N,...} -- could be stuck at 1

restate.num_active_partitions is not affected because it is a scalar gauge with no per-partition labels, recomputed directly from the live processor map.

Root cause

get_state() in crates/worker/src/partition_processor_manager.rs:806 iterates only over self.processor_states (currently-running processors) and calls gauge!(...).set(value) for each. When a partition stops, the entry is removed from processor_states (lines 573, 584, 1302-1333), but no code ever zeroes or unregisters the associated per-partition gauges.

metrics-exporter-prometheus v0.18.1 (used here) holds the last-written value in memory permanently unless explicitly cleared or an idle_timeout is configured. Since neither is done, every partition that ever ran on a node leaves a permanent ghost series.

Observable symptom

On a node that has seen partition migration, the count of is_effective_leader == 1 series via Prometheus exceeds the number of partitions currently led. For example, a node leading 20 partitions and following 28 (48 active total) was observed emitting 29 is_effective_leader=1 series and 30 is_effective_leader=0 series -- 11 stale series from partitions that had previously been on the node.

Any dashboard or alert using sum(restate_partition_is_effective_leader == 1) by (node_name) or similar will overcount leader assignments on nodes that have experienced partition movement.

Fix options

Option A -- PrometheusBuilder::idle_timeout (recommended):

In crates/tracing-instrumentation/src/prometheus_metrics.rs:41, add an idle timeout to the builder:

let builder = PrometheusBuilder::default()
    .add_global_label(...)
    ...
    .idle_timeout(MetricKindMask::GAUGE, Some(Duration::from_secs(120)))

The upkeep task already calls prometheus_handle.run_upkeep() every 5 seconds, which is what sweeps idle gauges. Setting the timeout to 2x the scrape interval (typically 30s scrape => 60-120s timeout) is sufficient. This is a one-line fix that covers all current and future per-partition gauges without touching each metric call site.

Option B -- explicit zero-on-stop:

In EventKind::Stopped (line 582) and on_replica_set_state_changes (line 1302), emit .set(0.0) for every per-partition gauge before removing the partition from processor_states. More surgical, but requires updating every call site when new per-partition gauges are added in future.

Option A is preferable unless there is a reason to retain zeroed-but-present series (e.g. for absence detection in alerting).