Testing whether browser agents still work when the web changes
Drift testing
Part of Chief of Staff
A harness, built but not yet merged, that records a browser agent's run, is meant to replay it against the live site later and name what changed and why the agent broke, for the failures benchmarks never catch. So far it has been tested only against a local fixture.
This build is part of the chief of staff's test management system. It is the part that tests whether a browser agent still works when the web changes.
Problem
A browser agent that passes every benchmark can still fail in production, for reasons no benchmark tests. A redesign moves a button, a cookie banner covers the page, a login quietly expires, or the site starts limiting requests. I wanted to know when one of my own agents breaks for one of those reasons, and which one. The general tools for watching AI agents score their traces in general terms rather than looking for these browser-specific failures.
Approach
Record a real run once, replay it against the live site later, and compare. The harness has five parts: a recorder, a replayer, a report that names what changed and which kind of failure it is, a small library of sites chosen because they change, and a pilot on real agents. Research comes first, through my /build pipeline, with an adversarial review after every round, so the design is argued over before any code is written.
Architecture
The reference agent follows the common pattern for browser agents: at each step it lists the page's clickable elements, asks a model which to use, and acts. Each of the four failure kinds breaks that loop in its own recognisable way, which is what the report reads. The review established that the recorder and the agent have to share one way of driving the browser, because the leading open-source agent library dropped the tool the recorder relies on. Drift is measured first on a test site deliberately changed between two versions, because the obvious practice sites never actually change. Login sessions stay outside the repository, and a test fails if a token ever reaches a trace.
What I built
The research and its reviews came first. The research found that none of my existing projects contains a browser agent to pilot against, only fixed test scripts, so the pilot needs reference agents built for it. The second review held the plan back until three questions are settled: which model the reference agents may use and at what cost, a single design for driving the browser, and real sources of drift. A four-stage build has since produced a working harness on a separate branch, with a scripted stand-in agent and a small site library. Its 155 tests pass and its pilot classifies all eight injected failures as expected, but only against a local fixture, and the branch is not merged.
Results
The most useful result so far was a correction. The market statistic that prompted the project, quoted in my own research as Cleanlab's finding on agent monitoring, does not appear in the report it cites. What the report does support is that fewer than one in three teams are satisfied with their observability tools.
What didn't work
The first candidate for the pilot, a part of another of my projects, turned out not to drive a browser at all, which the first research round caught before any design rested on it. The final review's three judges each scored the build 3 out of 10 at first, because the browser tests passed without ever launching a browser, which was then fixed.
Stack
- Python
- Playwright
- Claude Code
How these figures are sourced
The outcome figure is agent-reported: it is taken from a test log or report written by the agents that built and ran the system. The status was read from the build's final review. Every figure and claim on this page is tied to a source file or a CV line in the site's content.