RW 2.0-rc.5: allow exemplar-only time series

[thc1006]

This changes the 2.0 specification so that a TimeSeries carrying only exemplars is described rather than forbidden, and keeps the expectation that those exemplars travel with the series they belong to.

Why the current wording does not cover what a sender produces

Prometheus stores exemplars per series, not per sample. populateV2TimeSeries in storage/remote/queue_manager.go writes one output TimeSeries per queue item, and for an exemplar item it appends only an exemplar, having already reset that entry's samples and histograms. So an exemplar can leave the sender in a TimeSeries of its own.

The specification currently says:

At least one element in samples or in histograms MUST be provided. A TimeSeries MUST NOT include both samples and histograms.

That shape therefore has no wording behind it, and a receiver has nothing to rely on when it arrives.

What this changes

Three normative sentences, plus the proto copy this document carries:

• the "at least one element" rule accepts exemplars as well, and says what a TimeSeries carrying only exemplars means;
• the #### Exemplars section says the same thing from the exemplar's side;
• it adds the SHOULD that exemplars travel in the same request as the samples or histograms of their series, which is the second half of the decision and the part that keeps this from becoming a licence to detach them freely;
• the copied proto comments say the same as the rules above them, so the two do not disagree inside one document.

That last one matters because a sender does usually keep them together. QueueManager.Append and AppendExemplars are called separately by the WAL watcher, so the batch can be cut between a sample and its exemplar, but that is the exception rather than the intent. SHOULD says exactly that.

No field, field number or wire layout changes. The SHOULD does ask something of a sender, but it asks for what one already does when batching allows it.

What this leaves alone

The receiver tests in prometheus/compliance only cover exemplars sent alongside a sample. Covering a TimeSeries that carries only exemplars is a separate change. The canonical proto comments are in prometheus/prometheus#19530, which carries the source version bump to rc.5 as well.

On versioning

Retitled with the version, the way rc.4 was done in #2762, so the title is the record of what went into the release. #3080 still carries the single bump and #3081 is the other change in this one.

Related to prometheus/prometheus#17857 and prometheus/prometheus#16944.

[bwplotka]

This looks what wanted as per https://docs.google.com/document/d/1ZcyKVF87Xf6FhaGkb33LgVLb27FH5NIlRXWEGO2Cc4Q/edit?tab=t.0#bookmark=id.nllt8bbnl0i decision. However we need strict versioning.

I initially planned to pack this and async change under rc.5 for an easy reference -- we don't use git tags so it's a bit challenging to pack two separate changes under same version (:

[thc1006]

Retitled with the version, the way rc.4 was done in #2762, so the title carries the record that git tags would otherwise hold. #3080 keeps the single bump and #3081 is the other change in rc.5, so nothing has to be packed together beyond the two titles saying the same version.

Also added the second half of the decision, which I had left out: exemplars SHOULD be sent in the same request as the samples or histograms of their series, including when they travel in a TimeSeries of their own. Without it the wording read as a licence to detach them freely, which is not what the sender does today.

Out of draft now.

[bwplotka]

I changed base to #3085 so we can squash it all as a rc.5, hope that's ok

[bwplotka]

Ups I think you need to manually rebase, sorry @thc1006 - do you mind doing this?

[krajorama]

This would be updated once the proto is updated then?

[bwplotka]

Let's update it at one go and prep a PR.

[thc1006]

Both done: the copy here in this PR, the original in prometheus/prometheus#19530.

[krajorama]

LGTM, except the question about the copied protobuf

[thc1006]

Rebased onto rw2-spec-rc5, so this is one commit against that branch again. No trouble at all, and moving the base was the right call for keeping rc.5 in one place.

@krajorama good catch on the copied protobuf. The comment you pointed at turned out to be one of three saying the same thing: that one says a TimeSeries specifies samples or histograms, the exemplars field says they belong to the series' samples, and the Exemplar message says it too. All three are in this PR now.

The originals live in prompb/io/prometheus/write/v2/types.proto, so those go in prometheus/prometheus#19530. That one does not depend on this PR: populateV2TimeSeries clears Samples and Histograms and appends only an exemplar when the queue item is a tExemplar, so the comment already disagrees with the sender sitting next to it.

One thing I would rather ask than guess. In prometheus/prometheus#17857 back in January you proposed two halves, "Add SHOULD for per sample, but allow per series by spec". This PR has the second one. The SHOULD I added after your first comment here is about sending exemplars in the same request, not about preferring per sample association. Should the per sample SHOULD go in as well, or does the August decision replace it? Your 26 August comment on that issue reads as moving away from the per sample model for now, which is why I left it out.

Last thing, unrelated to the diff: prometheus/prometheus#18014 has been open since February taking the other route, dropping exemplars that arrive without a matching sample. @AftAb-25 wrote it against the spec as it read then. If this lands, they deserve to hear that the direction moved. The bot has already marked it stale.

[bwplotka]

General thought when I look a the changes: It feels this change is a step back when it comes to compatibility with systems oriented around OpenTelemetry model. As-in, you can use PRW2 to have a reliable transport against those, but it requires being strict on various elements beyond what protocol requires from sender e.g.:

• Avoid unknown metadata.
• Use NHCB instead of classic histograms.

Now with this change it also technically needs:

• Exemplar per sample (or at least within same request).

In the same time this relaxation of rules for exemplars allows (slightly) simpler protocol for existing Prometheus systems, and it's unlikely the exemplar part would change for those given the current APIs.

I wonder how to tackle this redundancy of implementing PRW2 around those intentions to battle ossification/proliferation and being too lean on sending so "I implement PRW2 sending" is not ambiguous.

Perhaps what we need is to introduce some "mode" or consider OTLP export again given advances we have in the storage.

[bwplotka]

We need to think about other elements like metadata. I don't think we want to allow them hanging without samples.

[thc1006]

The at-least-one rule above names samples, histograms and exemplars, so metadata alone does not satisfy it and a metadata-only TimeSeries stays invalid. If this line reads as an opening for the other fields, I can say so where the rule is rather than leaving it to be inferred.

[thc1006]

One piece of data first, since it may change the shape of the question.

Prometheus already sends metadata in a message with no samples. AppendExemplars reads seriesMetadata when it enqueues, and populateV2TimeSeries fills Metadata before it looks at what kind of item it has, so an exemplar-only TimeSeries goes out carrying type, unit and help. That is today's traffic rather than something this change introduces.

The same holds for the OTel side. A receiver reading real Prometheus traffic already had to decide what to do with exemplar-only messages, and the specification told it they were invalid. This does not hand it a new case, it names one it already had.

What the change does add is the SHOULD about the same request, and that one is mine to defend or drop. It is there so a receiver has a bounded window to associate within. If the cost is another implicit requirement on senders, dropping it leaves the MAY describing the wire format and nothing more.

On modes and on OTLP export I do not have a view worth your time. What would help me is knowing whether you want this narrowed to the wire-format statement, held until that larger question settles, or something else.

[bwplotka]

Prometheus already sends metadata in a message with no samples. AppendExemplars reads seriesMetadata when it enqueues, and populateV2TimeSeries fills Metadata before it looks at what kind of item it has, so an exemplar-only TimeSeries goes out carrying type, unit and help. That is today's traffic rather than something this change introduces.

That is true, and it's the tech debt due to abandoning the idea of using prometheus/prometheus#15911 - if we want to bring this back, we need to fix it.

The path using __type__ and __unit__ feature leverages metadata per samples correctly, so we do have path to use it.

[thc1006]

Thanks, that framing helps, and the __type__/__unit__ pointer is one I had not followed.

Correcting something in what I wrote above: I leaned on enableTypeAndUnitLabels as though it described how things run, but it sits behind --enable-feature=type-and-unit-labels, so it is the path you are pointing at rather than the current state. The argument I built on it does not carry the weight I gave it.

What is left is narrower. Exemplars are enqueued separately from their samples whichever way metadata goes, so that part does not resolve with it.

Happy to drop the SHOULD and leave this describing only the wire format, or to park it until the metadata direction settles. Your call.