Essay
When a coding agent says it works
Justin Ahinon
I use coding agents on my own products and on client projects. I keep having to ask them to open the app, sign in, and try the flow they just changed.
Reading the code and running a type checker leaves too much untested, especially on the frontend. Giving the agent a browser seemed like an obvious improvement.
It helped. But when I went back through my Codex sessions, I found failures I couldn't explain by a skipped test. The agent had used the browser and collected accurate observations, then given me the wrong answer.
One agent told me a broken checkbox worked. Another reported a loading bug that didn't exist. Their explanations sounded reasonable until I looked at what they'd measured.
I can't infer a failure rate from a few debugging sessions. But they've changed what I ask for when an agent says it tested something, and what I think a tool I've been considering would need to do.
The checkbox passed the test the agent chose
In July, I had a checkbox on the onboarding page of NextGoodRole that still looked empty after clicking it. Codex changed the callback that kept the control and form data in sync, then ran the project checks.
I asked whether it had tested with Agent Browser. It hadn't.
It went back, created a disposable local account, and worked through onboarding. Clicking the square changed aria-checked from false to true. The hidden form input changed too. Clicking the label worked, and so did pressing Space.
Codex told me the local code was working and suggested I hard-refresh my tab.
The checkbox still looked empty on my screen. I copied its rendered HTML into the conversation. This is the relevant part, with the unrelated classes removed:
<button
role="checkbox"
aria-checked="true"
data-state="checked"
class="bg-paper-bright text-paper-bright data-checked:bg-brand"
>
<svg class="lucide-check">...</svg>
</button>
aria-checked tells assistive software that the control is checked. Bits UI, the component library I was using, also sets data-state so CSS can style the control according to its state.
My styling expected a different attribute: data-checked. That attribute wasn't there, so the green background never applied. The checkmark was present, in nearly white, against a nearly white background.
Before the CSS fix
The form value is true, but the checkmark blends into the background.
After the CSS fix
Same checked state. Visible checkmark.
Once it saw the HTML, the agent found the selector mismatch. After fixing the styles, it checked the rendered colors and the icon as well as the form state.
I wanted the label and keyboard checks too. But I'd reported a control that looked uncheckable, and the agent never checked whether I could see the selection. It had turned my bug report into a question about form data.
Adding toBeVisible() wouldn't necessarily catch this either. Playwright defines visibility in terms of layout and CSS visibility. An element needs a nonempty bounding box and must not have visibility: hidden; even an element with opacity: 0 counts as visible. It doesn't establish that you can distinguish a checkmark from its background. Playwright's visibility rules are narrower than the everyday meaning of the word.
The agent needed to inspect the appearance. Here, the computed colors explained why the checkmark disappeared into the background. A screenshot comparison could catch the bug if it came back, provided someone had reviewed the reference image and confirmed it was correct. Saving a screenshot of the broken checkbox as the reference would preserve the bug.
A browser has several versions of the same page
When I say "the page," I usually mean what I can see on my screen. A browser tool might return something quite different.
The DOM is the browser's document structure: elements, attributes, and text, including things hidden from view. The accessibility tree describes controls in terms such as "checkbox, checked" or "button, Submit." A screenshot shows what the browser rendered at a particular size and moment.
My checkbox could be checked in the accessibility tree and look empty on screen. There was no contradiction in the observations. I was asking about something the agent hadn't inspected.
On a client project, an agent made the opposite mistake and reported a bug that wasn't there.
It was verifying a shopping-cart fix. It added an item and checked that it survived a reload, then reported an additional problem: the loading indicator seemed to remain visible for twenty or thirty seconds.
I asked what was causing that.
The application used Vue's v-show, which hides an element by changing its CSS display value. The element stays in the document. The agent's check used allTextContents(), a Playwright method that reads the nodes' text content. It kept finding ..., the text inside the loader, even after the loader had been hidden.
This small reconstruction does the same thing. Hide the loader and watch what happens to the text read from the document.
Rendered page
Read from this element
- textContent
- "..." still in the DOM
- Computed display
- inline-flex
- Loader on screen
- Shown
display: none, as Vue's v-show does. The readings
come from the live element above. No cart request runs here.When the agent checked visibility, it found display: none. The server timings showed cart fetches taking roughly 0.74 to 0.93 seconds. It withdrew the claim about the loader. There was nothing to fix there.
Giving the agent more state wouldn't necessarily have helped either investigation. For the checkbox, it needed to inspect the appearance. For the cart, it needed to account for hidden elements when interpreting the text it read. The DOM contained the loader, just as the tool reported. The agent had assumed that meant I could see it.
A screenshot would have helped in both cases, but I still need the other checks. An image of a checked box can't tell me what value the form will submit. What I want the agent to inspect depends on what I'm asking it to verify.
Who decides what counts as working?
This is part of an old problem in software testing: the test oracle problem. An oracle is whatever a test uses to decide whether the result is correct. For addition, that can be the expected sum. For onboarding, someone has to define what completing the flow means. Barr and colleagues' survey describes how this limits test automation.
With a coding agent, all of this happens in one conversation. The same agent interprets my request, changes the code, chooses the test, and reports the result. By the time I read "tested in the browser," the decision about what to test may be buried several tool calls back.
The checkbox test amounted to something like this:
await checkbox.click();
await expect(checkbox).toHaveAttribute('aria-checked', 'true');
That's a reasonable assertion for a state transition. It doesn't contain the part of my request that mattered: I should be able to see that I've selected the option.
For that bug, I want the agent to state the expected behavior before editing the code. After I click, I should be able to see that the option is selected. Assistive software should receive the checked state, and the form value should update. Each needs a check; passing one doesn't complete the others.
I'd also want it to reproduce the original symptom. If the proposed test already passes on the version I reported as broken, it can't establish that the reported defect has been fixed. That's a cheap way to catch a test aimed at the wrong behavior.
Reading the source helps the agent find the control and understand how it works. But the code might be wrong about what should happen. If the agent gets its entire definition of success from that code, its test can repeat the same mistake.
I ran into a related problem with a search alert on WhatIsThatMovie. The agent found successful HTTP responses and treated them as evidence that searches were healthy. We needed to examine what a completed search actually meant in that application. The status codes were correct; the conclusion went further than they supported.
Some answers live outside the browser
In another app, a test checkout produced an active Stripe subscription, but the account page still offered "Get Premium."
The agent investigated the backend. The payment webhook, the message Stripe sends to the application after a payment event, had failed while syncing the account's email. Processing stopped before the app created the database record that granted premium access.
Stripe had recorded the subscription. The application had a separate job to do: give the right account access to the paid features. That part had failed.
The agent got this investigation right. It found the symptom in the browser and the explanation in the webhook history and database.
A browser tool could try opening a premium feature with the paying account to check access. But a "Get Premium" button alone doesn't tell it why access is missing. The page could be stale, the browser could be signed into a different account, or the backend could still be processing the payment.
I'd also need to know when it checked. Access might be unavailable immediately after checkout and work a few seconds later. Waiting an arbitrary amount of time doesn't tell us whether that's acceptable. The test needs an allowed delay, and the agent needs to report how long it watched.
I'd accept a report saying the payment was confirmed but access hadn't been established and the cause was still unknown. That gives me somewhere to continue. Calling the flow successful would stop the investigation too early. Blaming a webhook without looking at it would give me a cause the agent hadn't checked.
A CLI and a map of the application
Lauren Tan describes a useful approach in her talk about building trust in agents. Around 8:38, she introduces Control Glass, a verification skill she built while working on performance in Cursor's agents window. It gives agents a CLI for running the application and collecting traces through the Chrome DevTools Protocol, the interface external tools use to inspect Chromium.
With that CLI, agents can repeat the same inspection across sessions. Someone maintains the tool, so each agent doesn't have to write its own version. I'd want that consistency in my browser investigations too.
At 10:16, she describes a problem I recognize: someone posts a small screenshot with a few question marks, and the agent has no idea which part of the app they're talking about. It can run the application, but it has to guess what the report means.
Her team added a feature map that documents what features do and how to reach them, down to keyboard shortcuts and DOM elements. It lives alongside the skill, with automation to maintain it. She says agents could then make sense of requests from users as well as operate the app.
My checkbox agent also knew how to reach the control and change its state. What it missed was part of the behavior I wanted fixed. A feature map could help describe that behavior. The agent would still have to choose a test that catches the problem.
Around 15:46, she talks about making recurring corrections part of the codebase and static checks, with rules and skills on top. For my checkbox, I'd capture the correct appearance in a regression test. I shouldn't have to explain the same failure in the next session.
Her team owns the application and spends time maintaining this knowledge. That's a substantial advantage over a tool inspecting an unfamiliar website. I want to find out how much of a useful feature map a browser tool could build by observing the app, where it would need context from the team, and what it would have to leave unknown.
What I'd want from a browser runtime
I started looking at this because I'm considering building a runtime layer for coding agents. By runtime, I mean the application while it's executing: the state changes, rendered output, and requests produced when someone uses it.
My first instinct was to collect more information about the running app and give it to the model. After reading these sessions, I'm less sure how much that would help on its own.
Existing tools already expose a lot. Playwright MCP provides accessibility snapshots, and its traces can record DOM snapshots, screenshots, requests, and console messages. Chrome DevTools MCP gives agents browser debugging tools. The checkbox agent had a working browser connection. It stopped before using it to answer the visual question.
I'd start with a report after each action that compares what changed with what the agent expected. I want to see which expectations it checked and which it hasn't verified yet.
This is the sort of report I'd want for the checkbox. I haven't built the tool yet:
Expected: selecting the option visibly marks it as checked
Action: click the checkbox on the onboarding page
Observed:
aria-checked changed from false to true
hidden form input changed from false to true
checkmark exists; foreground matches its background
Assessment:
form-state check passed
checked appearance failed
Evidence:
before/after attributes, computed colors, checkbox crop
The report needs to distinguish what the tool read from what the agent concluded. I can inspect a recorded attribute or color. To judge "checked appearance failed," I also need the expected appearance and the observations the agent used to reach that conclusion.
I'd use the Chrome DevTools Protocol to capture the relevant state before an action and the changes afterward. The report could stay small while linking to the full evidence. It would also record the URL, browser session, viewport, and how long it observed. When an agent tests a fresh local account, I need that scope to survive into its answer about whether my app works.
Pages keep changing in the background, which makes the comparison harder. A request might start after a click without having anything to do with it. Two browser snapshots might capture different stages of an operation. I'd need to preserve their timing and distinguish relationships the tool observed from ones the model inferred.
Framework adapters could connect a control to component state where the framework exposes it. On my own apps, I could also inspect server traces or query the database. I'd have much less access on an arbitrary website, and I want the report to show those gaps. The model shouldn't fill them with an explanation it can't verify.
I'd still need to tell the runtime what should happen. A bug report or an existing test might provide enough to start. Inspecting the program alone won't tell it every product requirement that program is meant to satisfy.
I'd test the idea against better instructions first
I have cases I can reproduce. I still need to find out whether building this runtime would help an agent handle them.
I'd first try today's browser tools with better instructions: reproduce the symptom, state what should happen, test that behavior, and report anything still unchecked. If that works as well as the runtime I have in mind, I'd like to know before building the whole thing.
I'd run the same agent on the same tasks in the same environment with ordinary tool use, then with those instructions, then with the instructions and runtime report together. I'd start with the checkbox and cart because the agent needs to find a bug in one and dismiss a false alarm in the other. Declaring everything broken would only get one of them right.
The test set needs working and broken controls, operations that take time and ones that fail, and cases I hadn't used to design the tool. Each would get multiple runs. I'd count missed bugs separately from false alarms and read the "unknown" results to see whether the agent lacked access or had just stopped looking. I'd measure time and token cost too.
I'd consider the report useful if it helped the agent get the answer right, explain why, and need fewer corrections from me. I can also imagine giving it a larger dump of browser state and getting a longer, equally wrong explanation back.
For now, when an agent says it tested a fix, I want to know whether that test would have caught the problem I reported. The checkbox test would have passed while the box still looked empty. I'd rather find that out before being told to refresh my browser.