What Happens When You Put an AI Agent Loop Inside the Browser?
What happens when an AI agent loop runs sandboxed inside your browser tab—with no app server to host and no direct access to your local files? Discover how far browser agents can go today and the future of on-demand UI.
Over the last few posts, I explored running AI models directly inside the browser—from neural text-to-speech and image background removal to full LLMs running locally on WebGPU.
What if we combined that with an agent loop? How far can we push an agent running inside a plain web page?
Not just a chatbot replying in a text box—an assistant living inside the tab that can decide what steps to take, pull live data, and build interactive screens on the fly:
“A wild future where if you want a custom UI experience, you just ask an agent to build it for you on the spot. On the webpage, nothing installed.”
That experiment became @mograblog/web-agent-loop—an open-source TypeScript library that runs an autonomous agent loop right inside the browser.
Try It Out Live in Your Browser
Here is @mograblog/web-agent-loop running autonomously inside a browser tab:
1. What Is an Agent Loop? (The 101 Version)
If you strip away the AI buzzwords, a regular LLM chat is one-shot: you send a prompt, the model predicts text, and it stops.
An agent loop (often called the ReAct pattern—Reason + Act) turns that single reply into a while loop:
while (!taskComplete && steps < maxSteps) {
const nextStep = await model.decide(history);
const result = await executeTool(nextStep);
history.push(result);
}
Instead of a human manually copy-pasting outputs and errors back and forth in a chat box, the loop orchestrates the User, the Agent Runtime (plain TypeScript in your browser), the LLM, and your Tools:
sequenceDiagram
autonumber
participant U as User
participant A as Agent Loop (Browser JS)
participant M as LLM (WebGPU / Ollama / API)
participant T as Tools (DOM / Canvas / MCP)
A->>T: Collect registered tools + JSON schemas
U->>A: Send goal ("Build a playable Pong game")
A->>M: Prompt (System + Tool List + Goal + History)
loop Until final answer or max steps
M-->>A: Structured JSON tool call { name, arguments }
alt Tool call succeeds
A->>T: Execute tool(arguments)
T-->>A: Observation (data / DOM / Canvas updated)
else Invalid JSON or tool throws error
A-->>A: Catch error & format recovery message
end
A->>M: Append tool result / error to history & ask again
end
M-->>A: Plain text reply (no tool calls)
A-->>U: Present final answer & rendered result
(Note: This is the simplified, 101-level version of an agent loop. Production agent frameworks layer on much more complexity—reusable skills, lifecycle hooks, context compaction, human-in-the-loop approvals, and subagents—all out of scope for this post, though the core loop underneath stays the same.)
Four Mechanics That Make the Loop Work
- Tool Discovery & Prompt Construction: Before the first call, the loop gathers every registered tool’s name, description, and JSON parameter schema (whether built-in like
render_viewor discovered from an MCP server) and injects them into the prompt alongside your goal. - Structured JSON Instructions: On each turn, the LLM doesn’t run code itself—it emits a JSON instruction specifying which tool to invoke and with what arguments.
- Detecting the “Final Result”: How does the loop know when to stop?
- No more tool calls: When the LLM responds with normal text instead of a JSON tool call (
finish_reason: "stop"), the loop treats that as the final answer and exits. - Step limit guardrail: A hard
maxIterationscap prevents a confused model from spinning forever.
- No more tool calls: When the LLM responds with normal text instead of a JSON tool call (
- Handling Failures (Self-Correction): Smaller browser models make mistakes—malformed JSON, hallucinated tool names, or runtime errors inside a canvas script. Instead of crashing, the loop catches the exception and feeds the error message back to the LLM as the tool’s observation. On the next iteration, the model sees exact feedback (e.g., “Error: unexpected token in canvas code”) and fixes its own mistake.
2. How Far Can a Browser Agent Actually Go?
Let’s be honest about where things stand today: pure, free in-browser WebGPU models are still too weak for real-world production usage.
As you can see in the video above (where I sped up the thinking pauses), small browser models are slow and still fumble complex multi-step tool calls. Right now, the pure-WebGPU mode is a proof-of-concept that shows what’s coming—not something you’d ship to end users as-is.
Where the architecture already gets limitless is how easily the browser loop can reach beyond the tab whenever you need real power:
flowchart TD
Agent["🧠 <b>WebAgentLoop</b><br/>(Inside Your Browser Tab)"]
Agent -->|"1. Inside Your Page"| Tab["<b>Direct Client Tools</b><br/>• Live DOM & Tailwind Dashboards<br/>• HTML5 2D Canvas & Games<br/>• localStorage State Persistence"]
Agent -->|"2. Over The Wire"| Cloud["<b>Cloud APIs & MCP Gateways</b><br/>• OpenAI / Grok / Claude / Gemini<br/>• Arcade.dev SaaS Connectors<br/>• Remote Databases & Web Feeds"]
Agent -->|"3. Down to Localhost"| Local["<b>Local Machine Bridges</b><br/>• Local Ollama<br/>• Local MCP Servers (Filesystem / Git)<br/>• Local Terminal Bridges"]
- Inside Your Page (Live DOM & Canvas): Unlike a backend script that can only stream text back, an in-browser agent can directly mutate the page you’re looking at:
render_view: Generates responsive HTML + Tailwind CSS and injects it straight into a container element.draw_on_canvas: Writes and executes 2D<canvas>animation loops (draw(t)) for interactive charts, physics simulations, and playable mini-games.web_storage: Reads and writes state inlocalStorage.
- Swap in a Real Brain (Cloud APIs or Local Ollama): When WebGPU isn’t strong enough, you can point the exact same browser loop at OpenAI, Grok, Anthropic, or Gemini—or connect straight to a locally installed Ollama.
- Infinite Tools via MCP: Any capability the browser lacks—whether cloud SaaS APIs (via gateways like Arcade.dev) or local filesystem/terminal access—can be plugged in over MCP (Model Context Protocol) via WebSocket or SSE, even on
localhost.
3. A Quick Look at the Code
If you want to experiment with it yourself, I published the runtime and built-in tools as @mograblog/web-agent-loop. Wiring up a model, browser DOM/Canvas tools, and an MCP server takes about 15 lines of TypeScript:
import {
WebAgentLoop,
WebLlmProvider,
createRenderViewTool,
createCanvasTool,
WebAgentMcp,
} from '@mograblog/web-agent-loop';
const loop = new WebAgentLoop({
provider: await WebLlmProvider.fromModel('Hermes-3-Llama-3.1-8B-q4f16_1-MLC'),
tools: [
createRenderViewTool('dashboard-container'),
createCanvasTool(document.getElementById('my-canvas')),
],
});
// Optional: plug in any local or cloud MCP tool server
const mcp = new WebAgentMcp({ name: 'local-bridge', serverUrl: 'ws://localhost:8765/mcp' });
await mcp.connect();
loop.registerMcp(mcp);
await loop.run('Search for recent news about Tesla and build an interactive dashboard.');
The Verdict
- Browser LLMs were disappointing: Honestly, I was hoping pure in-browser WebGPU models would be able to get a lot more done on their own by now, and they just aren’t there yet.
- Ollama was a pleasant surprise: Building this got me to try hooking the browser loop up to a locally installed Ollama, and it was way smoother and more capable than I expected.
- Cloud providers (like Grok) are still night and day: The moment you point the exact same browser agent loop at a strong cloud model like Grok, you immediately see the difference—tool calls snap into place and the UI/canvas output is clearly on another level.
The browser runtime itself is ready; now we’re just waiting for local models to catch up. Give it a spin in the Fullscreen Playground, check out @mograblog/web-agent-loop on npm, and let me know what you think in the comments!