Field notes · WebMCP Teach
WebMCP vs. APIs, MCP, and browser automation
Five rungs separate a direct API call from a model squinting at pixels. I built an extension and ran three experiments to feel out the difference.
Last verified . Hosts and protocols move fast, so details here may be stale. Check the primary sources before you build.
The fastest run in these notes took about 1.1 seconds: an agent refreshed a ChatGPT connection and sent a prompt through a tool I taught the browser myself. The slowest way to do the same job is a screenshot and a guess about where the button is. Both work eventually. That is the trap.
There is no best way for an agent to operate a web application. There is a reliability ladder, and the mistake is treating the rungs as interchangeable because every one of them can eventually click the same button. The right rung depends on where the agent runs, whether the site is already open, whether the action must stay visible, and whether the owner will publish a contract. Top to bottom:
A direct API is the strongest contract
An authenticated API exposes business operations with no browser involved: fast, testable, indifferent to layout. The costs are integration work (credentials, client code, a policy layer) and blindness to the tab: drafts, local selections, and unsaved changes live above the backend contract.
An MCP server makes the contract discoverable
MCP makes those operations discoverable: agents find and call tools wrapping a first-party API, database, or workflow from desktop, web, mobile, or cloud clients, no tab open anywhere. It does not make an existing web interface legible to a browser agent. Chrome's line is clean: MCP is backend and persistent, WebMCP is frontend and bound to the current page.
WebMCP names the operation inside the live page
A page can annotate an HTML form declaratively or register a JavaScript tool through document.modelContext.registerTool(): name, description, JSON Schema, an implementation wired to the current page. The site declares what the operation means; the agent stops inferring that a rectangle with an icon is probably submit. Tools use the user's live session and visible state, die with the tab, and sit behind a Chrome origin trial and a flag; the proposal can still change.
DOM automation operates what already exists
Playwright and its relatives find elements by role, label, text, or selector, then click, fill, and inspect, with no cooperation from the site. The weakness is coupling: selectors rot after redesigns, ambiguous labels grab the wrong control, one cross-document navigation can wipe an in-flight workflow. Accessibility semantics soften this without becoming a first-party promise.
Visual computer use reads the pixels
A visual agent reasons from screenshots and drives the mouse and keyboard: websites, native apps, canvases, anything a person could see. It also reinterprets the presentation at every step, paying in time, tokens, and fresh chances for ambiguity. WebMCP was designed to remove that guesswork for known application functions, not to retire computer use everywhere else.
The evidence
Teaching a browser tools its sites never shipped
Most websites will not adopt WebMCP soon. So I built WebMCP Teach, an experimental Chrome extension, as a temporary bridge: you define a domain-scoped tool as JSON, with a name, description, JSON Schema inputs, and a declarative sequence of browser steps. The pack lives in chrome.storage.local for the registrable domain; on later visits the extension registers it on the page's native document.modelContext (a navigator.modelContext fallback covers older experimental Chromium builds).
{
"name": "search_specialists",
"description": "Search specialists by specialty and city",
"inputSchema": {
"type": "object",
"properties": {
"specialty": { "type": "string" },
"city": { "type": "string" }
},
"required": ["specialty"]
},
"steps": [
{ "type": "fetchDoc", "url": "...{{args.specialty}}..." },
{ "type": "extractList", "itemSelector": "...", "as": "specialists" }
]
}The registered callback runs in the page's main JavaScript world; execution crosses a nonce-protected postMessage bridge into the extension's isolated content script, which interprets the steps: waitFor, click, fill, select, press, submit, same-origin fetchDoc, structured extraction. No arbitrary remote JavaScript, domain matching enforced, same-origin fetches only, and requiresConfirmation gates agent-initiated calls behind a user prompt. Edit a pack and it re-registers without a reload. None of it competes with the standard: the day a site ships first-party tools, bound to stable application logic and real authorization, the local pack retires.
Doctoralia: ten results without touching the tab
The Doctoralia tool takes a specialty and an optional city, fetches the results page same-origin, parses it in a detached document, and extracts up to ten specialist cards: name, profile URL, specialty, address. The live tab never navigates, scrolls, or flickers. A search for dermatologista in São Paulo returned ten structured results in Arc, Brave, and Chrome test sessions. For a server-rendered site, the browser is just a very well-authenticated HTTP client.
iFood: the SPA demands its own event handlers
That trick has no purchase on iFood, which renders client-side. Its tool works the page instead: wait for the search field, fill the query, submit through requestSubmit(), wait for client-rendered results, extract up to twelve restaurant cards. A live sushi test returned names, URLs, ratings, categories, delivery times, and fees. Two sites, two strategies; one mechanism was never going to be enough.
ChatGPT: nested frames, 1.1 seconds
The most demanding pack targeted a test account on ChatGPT: open a new chat, send a prompt, read visible page state, open an installed app's management panel, refresh a connection, register and uninstall a private MCP app, operate controls inside nested MCP App frames.
Hard-coded account identifiers were useless there, so the extension gained semantic targeting by visible text and accessible name: a tool finds "Open Gmail Compose Message" knowing nothing about the account. Child-frame actions run through chrome.scripting; the native WebMCP registration stays the single agent-facing entrypoint. Refresh-and-send completed in about 1.1 seconds, and a nested Gmail app control was operated without a single runtime computer-use click.
What survived the experiments
Structure ends re-interpretation
The agent chooses one named operation and supplies validated arguments instead of rediscovering the interface on every run.
Semantic targets age better
Accessible names and visible control text outlive account-specific selectors. Redesigns can still break them; they break less often.
Architecture picks the strategy
Server-rendered pages suit a detached same-origin fetch. SPAs need their live event handlers. Embedded apps need frame-aware actions.
The layers compose
The ChatGPT pack used WebMCP to operate a website that itself hosts MCP Apps. Each layer handled a different boundary.
How I would choose
- A direct API for stable backend operations you control.
- An MCP server to reach those operations from any client, no tab open.
- WebMCP to act inside the user's live session and visible state.
- DOM automation when no contract exists and the workflow can tolerate maintenance.
- Visual computer use when nothing programmatic is reliable, or the job spans native apps.
- A user-taught bridge for a site you do not control, retired the day native tools arrive.
The long-term architecture is a stack with graceful fallback: start at the most explicit contract available, step down only when forced. WebMCP Teach is scaffolding for the adoption gap, useful precisely because it is built to be torn down. WebMCP itself matters because it gives the open web a first-party semantic layer without turning every website interaction into a remote server call. Stand on the highest rung the site will let you.