Observations are specified as context thrashing

[gterzian]

https://webmachinelearning.github.io/webmcp/#observation is specified as an implementation-defined data structure containing at least a tool map, which implies the entire tool map should be sent along in each observation, which seems like a great way to fill up the context of the agent with duplicate data.

I'd say the "containing at least a tool map" can be dropped, so that implementations can include such a map only in an observation of tool changes (which can be made explicit).

[domfarolino]

which implies the entire tool map should be sent along in each observation, which seems like a great way to fill up the context of the agent with duplicate data.

Realistically, agent implementations are responsible for their own observation diffing, or other absorption techniques. So I think that defining a standard observation as containing at least a tool map is the right model for illustrative purposes, and real-world agents will do with that whatever they please at their harness and filtering layer. I don't think the spec is incompatible with this.

If you want to submit a PR clarifying that real-world agent implementations will likely need to be "smart" about how repeated observations play with their context, I'd welcome it.

[gterzian]

Realistically, agent implementations are responsible for their own observation diffing, or other absorption techniques.

If we take the below are a starting point, which is from https://html.spec.whatwg.org/#history-2:

specifications need to be detailed enough that implementations can achieve complete interoperability without reverse-engineering each other

Then I would say the kind of stuff mentioned in this issue should be deal with in more details in the spec itself, otherwise implementations could end-up with completely different behavior based on what the UA chooses to do to avoid thrashing the context.

By this I do not mean specifying a diffing algorithm, but just be more specific about when observations are sent, and with what content.

For example:

• send an observation containing a new tool when that new tool is registered (so only the new tool, not other stuff that you might want to observe at other times), which is discussed at #dom-modelcontext-registertool could send an observation #230
• have the app send an observation whenever it deems it necessary, containing whatever data it deems necessary (discussed at Add method to ModelContext for application driven observation #229). yes that means the app can thrash the context, but that could be dealt with by making compaction part of the spec, as in expose agent sessions and their compaction #232

But for the ua to be responsible for deciding when to send observations, what data they contain, and what to do to avoid duplicating data in the agent's context would seem quite bad for compat in my opinion.

[domfarolino]

otherwise implementations could end-up with completely different behavior based on what the UA chooses to do to avoid thrashing the context.

The implementation you're talking about is the agent implementation, which isn't governed by web platform specifications like ours. WebMCP, HTML, etc., can only specify constraints on an implementation of a web platform. But exactly how browsers, AI agents (that live in or outside the browser), or other computer programs respond to state on the web (including tools), is not something we can constrain.

I'll be more concrete. Gemini-in-Chrome has this concept of page observations that exists far outside of WebMCP. It uses parts of the Google Chrome product to construct a Chrome-specific "observation" of the page. Since it's not web-exposed, it is not specified by any web specification. Copilot-in-Edge has a similar mechanism, but differs in ways that make the most sense to Copilot's agent harness. Same goes for AI mode in Chrome's side-by-side experience—it handles, absorbs, and responds to information on the page in ways that make sense for the AI mode product, not the web platform.

All we can do on the platform is describe some reasonable surface area of shared data structures that each agent will likely implement to absorb tools on a page, and specify how tools interact with that mechanism. We can't specify how that mechanism interacts with agent loops, or when the agent should "pull" observations, etc. Those are all agent product decisions, not platform decisions. All we can specify is what tool data is exposed to the mechanism that agents interact with however they want.

But for the ua to be responsible for deciding when to send observations, what data they contain, and what to do to avoid duplicating data in the agent's context would seem quite bad for compat in my opinion.

This is necessary. Each agent is a different product, and we cannot define in our specification when Gemini-in-Chrome chooses to "observe" page state, any more than we can can specify when Claude Code's in-app browser pulls information from a site it's hosting, or observes state changes on it. Those are product differentiators that must be left to the product.

[gterzian]

Thanks for the additional background.

So what you describe with Gemini I would indeed see as browser specific and not part of the standard web platform, and I also think it might be kept separate from the standard.

Just like there is nothing stopping Chrome from shipping it's own devtools, it can ship it's own agent that can have access to whatever it wants on the page.

But where I disagree is that I don't think WebMCP has to be a kind of partial API to that Chrome "super" agent.

You could also see it as just an api to "an agent", and then you could expose things like sessions and other primitives, in a way that can be Web compatible and also respects user privacy (for example if you need cross origin agentic workflows, a given session could be explicitly shared between two origins, as opposed to relying on an unspecified browser specific super agent).

Then if someone wants to implement an agent using the page via WebMCP, well then if the agent needs to be Web compatible it just has to be implemented in a way that follows the constraints of the spec; that is the same for any other features of the web, and is unrelated to other stuff like how Claude uses a webview or something like that (which is indeed outside of the scope of standards).

Anyway, outside of the scope of this issue, but thanks for the back and forth.