fix: Always log snapshot succession

[sergerad]

Summary

Why:

SnapshotGuard only emitted the snapshot.superseded_for_ms field on the warn-level event fired when a snapshot outlived its supersession by more than the 10s threshold; the healthy release path logged at debug without the field. Since Honeycomb triggers can only be built against columns that already exist in the dataset, this made it impossible to set up alerting on snapshot retention before the pathology had already occurred — and debug events are filtered out of the export pipeline anyway.

How:

Every snapshot release now logs at info level with the full field set (block.number, snapshot.lifetime_ms, snapshot.superseded_for_ms, snapshots.live), escalating the same event to warn when time held past supersession exceeds SNAPSHOT_SUPERSEDED_WARN_THRESHOLD. A never-superseded snapshot (the latest generation at shutdown) reports zero rather than omitting the field, so triggers never see missing values. Volume is roughly one event per block interval. This also gives us baseline data to chart p95/p99 retention and pick alert thresholds empirically.

Changelog

changelog = "none"
reason    = "Observability-only logging change."

[Mirko-von-Leipzig]

I don't think this should be logged on info.

I think the way to think about this, is that we shouldn't be logging this at all - this should be information attached to the query span. But the problem is that this is tracked within the object itself, and not within the query so this is difficult to now do correctly.

Could we instead drop this tracking entirely, and instead simply use the RPC queries/handlers durations directly?

[sergerad]

I would like to know how long snapshots are alive for in a manner that takes into account whether a new snapshot has been created or not.

Yea refactoring this into a span field is the right idea. But I might need to impl that as part of #2452 for per-block build metrics.

[Mirko-von-Leipzig]

Similarly, I'm not sure why we care about the snapshot being superseded specifically. We care about it being held for a long time in general - whether or not the chain kept going or not is somewhat irrelevant?

[sergerad]

Because its at least theoretically possible that a snapshot is around for too long because a new snapshot has not been created. Thats why we take into account whether it was succeeded or not.

Granted we should have other alerts that relate to that slowdown - in block building. I would still rather have this duration as a span field.

[sergerad]

I think it would be fine to close this PR and integrate a relevant span field in #2452