Add SpeechRecognitionResult Timestamps

[alan33d]

Description

The Web Speech API currently does not expose the start and end timestamps of the source audio corresponding to a given transcription result (SpeechRecognitionResult). This limitation creates two major challenges for API clients and end users:

• Timeline Association: Developers cannot readily associate transcribed text with specific segments of the source audio stream, making it difficult to map generated captions to media timelines, audio tracks, or live video frames.
• Latency Tracking & Cloud Failover: With the adoption of on-device Automatic Speech Recognition (ASR) to improve privacy and reduce server costs, processing performance becomes heavily dependent on local client hardware resources. The Web Speech API acts as a "black box" regarding local processing delays. Developers cannot programmatically calculate transcription latency or detect when on-device models fall behind real-time. This leads to poor user experiences (e.g. caption lag during live video conferencing) and deprives applications of the signal needed to seamlessly fail over to a cloud ASR backend provider.

Specification

Expose source audio timestamps on the SpeechRecognitionResult interface in seconds (relative to the start of the audio stream consumed by the recognizer, starting at $0.0\text{s}$).

Web IDL

partial interface SpeechRecognitionResult {
    // Start timestamp of the audio segment in seconds relative to the start of the audio stream (0.0s).
    readonly attribute double audioStartTime;

    // End timestamp of the audio segment in seconds relative to the start of the audio stream.
    readonly attribute double audioEndTime;
};

Usage Example

Developers can compute real-time processing latency by mapping the stream timestamp to the document timeline and comparing with Event.timeStamp:

const recognition = new SpeechRecognition();
recognition.continuous = true;
recognition.interimResults = true;

let audioOriginMs = 0;

// 1. Capture stream start timestamp on the document timeline
recognition.onaudiostart = (event) => {
  audioOriginMs = event.timeStamp;
};

recognition.onresult = (event) => {
  const result = event.results[event.resultIndex];

  // 2. Convert stream-relative seconds to document timeline milliseconds
  const absoluteAudioEndMs = audioOriginMs + (result.audioEndTime * 1000);
  const latencyMs = event.timeStamp - absoluteAudioEndMs;

  // 3. If local processing latency exceeds acceptable threshold, switch to cloud provider
  if (latencyMs > LATENCY_THRESHOLD_MS) {
    console.warn(`ASR processing lag detected (${latencyMs.toFixed(0)}ms). Transitioning to cloud provider.`);
    switchToCloudBackend();
  }
};

recognition.start();

Security & Privacy Considerations

• Fingerprinting Risk: High-precision timing can expose micro-architectural hardware and CPU performance characteristics for device fingerprinting.
• Mitigation: In accordance with W3C High Resolution Time Level 3 and HTMLMediaElement.currentTime, user agents may coarsen timer resolution or introduce jitter to mitigate side-channel risks in non-isolated contexts.

Alternatives Considered

• Browser Warning Events (e.g. onprocessinglag): Difficult to standardize a single threshold across applications with varying latency tolerances (e.g., real-time captions require <200ms, while dictation tolerates multiple seconds).
• Internal Queue Metric (e.g. queueDepth): Difficult to standardize across fragmented engine architectures, model types, and buffering models.
• Binary Status Flag (e.g. isRealTime): Lacks numerical flexibility for applications to measure progressive degradation trends.
• Existing Events (speechstart / speechend): Fire only once per continuous session, cannot be correlated to ongoing stream results, and event.timeStamp reflects DOM dispatch rather than acoustic audio boundaries.

Links & References

• Explainer: speech-recognition-result-timestamps.md (PR #205)
• Spec PR: PR #192
• Interactive Demo: Speech Latency & Audio Timestamp Demo (by @msw)
• ChromeStatus: https://chromestatus.com/feature/5811907077472256

[padenot]

https://webaudio.github.io/web-speech-api/#speechreco-result:~:text=For%20all%20these,endpointer%29%2E

In Firefox, this accounts for input latency (for direct microphone access), various buffering stages e.g. in Media playback, IPC boundaries, and the ASR model inherent latency. Output latency is available on the web platform to have super tight synchronization for e.g. auto-captionning.

This is workable, and we have built neat demos that show that alignment works well (e.g. putting words below the wave form). It does however require doing a little bit of arithmetic, but you get high flexibility in return.

@alan33d, @evanbliu, what does Chrome do?

[michaelwasserman]

Can you point to the demos? I'm not too experienced here and more of a visual learner, so I vibe coded a demo (source) to try learning about the status quo.

I tried testing the same in FF with about:config media.webspeech.* flags all set true, but it wasn't working; lmk if you have tips.

I'm hoping we can continue to explore the viability of using the existing API surface, but new timestamps may prove useful if not necessary.

[padenot]

Our implementation isn't merged.

[alan33d]

https://webaudio.github.io/web-speech-api/#speechreco-result:~:text=For%20all%20these,endpointer%29%2E

In Firefox, this accounts for input latency (for direct microphone access), various buffering stages e.g. in Media playback, IPC boundaries, and the ASR model inherent latency. Output latency is available on the web platform to have super tight synchronization for e.g. auto-captionning.

This is workable, and we have built neat demos that show that alignment works well (e.g. putting words below the wave form). It does however require doing a little bit of arithmetic, but you get high flexibility in return.

@alan33d, @evanbliu, what does Chrome do?

Thanks for the suggestion. The current proposal addresses a developer request to detect speech recognition latency degradation mid-audiostream. From @michaelwasserman 's demo using the existing API surface, we noticed the existing event updates will only allow websites to accurately detect the latency of speech recognition at the beginning of the stream which is not enough granularity for the developer's use case. Thus we are proposing this change to get us the latency updates on an individual SpeechRecognitionResult call basis.

[michaelwasserman]

I published an improved demo and Chrome seems to fire speechstart, speechend, etc. just once over the course of the recognition session. While the API might permit firing those events around each recognition result, clients would have a hard time correlating separate events and results. It seems like including the timing information in the results is a vast ergonomic improvement.