At a high level, the agent generates and maintains userscripts and CSS that are re-applied on page load. Rather than just editing DOM via JS in console the agent is treating the page, and the DOM as a file.
The models are often trained in RL sandboxes with full access to the filesystem and bash, so they are really good at using it. So to make the agent behave well, I've simulated this environment.
The whole state of a page and scripts is implemented as a virtual filesystem hacked on top of browser.local storage. URL is mapped to directories, and the agent starts inside this directory. It has the tools to read/edit files, grep around and a fake bash command that is just used for running scripts and executing JS code.
I've tested only with Opus 4.5 so far, and it works pretty reliably. The state of the file system can be synced to the real filesystem, although because Firefox doesn't support Filesystem API, you need to manually import the fs contents first.
This agent is really useful for extracting things to CSV, but it's also can be used for fun.
It's funny what details about our designs are similar through accident. And what other things are completely different. I can show you my design potentially.
Representing websites in a virtual filesystem is creative and definitely makes it easier for the agent to collect information about the page. But I'm confused between the `Bash` and the `Edit` tools. It seems like one uses the chrome executeScript API, and the other updates the file system. But if it's just doing file writes, are those edits visible in the browser, and persistent across sessions?
So Bash and Edit tools are a bit weird, Bash tool is essentially JS execution, and Edit tool automatically generates a script that performs the edits on the page. These tools are needed for the model to explore the page, whatever it does at the end it creates a separate script that will be applied on the page load.
That’s one of my biggest headaches writing user scripts currently: I write the script in an IDE with Claude then copy it to the browser / manually test it in the browser, then copy the results back to Claude or tell it what went wrong.
Looking forward to trying this.