Your AI Agent Broke in Production: Can Open-Source Tooling Tell You Why?

Last Updated: 14 November 2026

What a broken agent looks like when nothing throws an error

The failures that cost you money rarely arrive as exceptions. Google's own framing of the problem is unusually blunt. In a production agent, a bad run can return HTTP 200, stay inside its latency budget, and never raise a tool error, while the agent confirms seat 3A without checking availability or recommends steak to a vegan traveller. Nothing in your dashboard turns red. The customer finds out before you do.

That gap is why the numbers around agent projects are so poor. Gartner predicted in June 2025 that more than 40% of agentic AI projects will be cancelled by the end of 2027, citing weak return on investment and immature agency, and coined the term agent washing for repackaged RPA and chatbots. The MIT NANDA report, The GenAI Divide, found in August 2025 that 95% of organisations were getting zero return from GenAI pilots, based on 150 interviews, 350 employee surveys and 300 public deployments.

Capability benchmarks explain part of it. On TheAgentCompany, a suite of realistic workplace tasks published in December 2024, the best agent completed about 30% of tasks autonomously. On GAIA, humans score 92% against 15% for GPT-4 with plugins. An agent that is right seven times out of ten is not a broken product. It is a product that needs supervision, and supervision needs evidence.

So the useful question is not whether your agent works. It is whether you can tell, after the fact, which of yesterday's runs were wrong and why. That is the question two open-source releases in this news cycle are answering, from opposite ends of the stack.

Google's AQuA: a sidecar that reads your traces

Google open sourced the Ambient Quality Agent on 8 October 2026 as a reference implementation rather than a product. It lives inside the ADK recipes repository under an Apache 2.0 licence. AQuA never sits in the request path. It sweeps production trajectories from Cloud Trace, Cloud Logging or BigQuery on a schedule, after each deploy, or on demand, and it never writes back to the agent it is reviewing.

The loop has five stages. It samples sessions at random, capped at 1,000 per run. It reviews each session against a nine-point checklist covering procedure, tool selection and arguments, grounding and task completion. It clusters findings so that twenty variations of the same bug count as one issue. It verifies each cluster against up to three full transcripts using a separate model. Then it tracks each finding as NEW, RECURRING, or auto-RESOLVED after 14 days unseen.

The verification step is the design centrepiece, and it is where most home-grown quality tooling falls over. Clusters are treated as claims, not findings. On an internal benchmark of 87 traces, the verifier rejected 4 of 24 candidate clusters. There is no confidence score in the output, because an insight is its citations: session IDs in Cloud Trace plus path and line ranges validated against the immutable snapshot captured at deploy time. It proposes prompt edits but never applies them and never opens a pull request.

The cost is published, which is rare. A 96-session single-agent sweep cost $0.70, about $0.007 per session. A 32-session multi-agent sweep covering 1,583 spans cost $3.76, about $0.12 per session. On-demand root-cause analysis runs between $0.33 and $2.47 per insight.

The demo numbers show what a diagnosis loop is worth. On Google's travel-concierge agent, 32 sessions produced 27 with findings, 42 findings collapsed into 9 clusters, and 6 issues survived verification. After two one-line prompt fixes, seat-selection bypass fell from 15 sessions to 2, a drop of 87%, vegan-profile omissions fell from 7 to 0, and full-session pass rate rose from 5 of 32 to 13 of 32. Two sentences of prompt text, found by reading traces instead of guessing.

Cloudflare's Kitesurf: this one is not observability

Cloudflare open sourced Kitesurf on 13 November 2026 under Apache 2.0, after announcing it in August and shipping it as experimental. It is a browser engine that runs entirely on Cloudflare Workers with no browser container. It parses HTML, applies CSS, executes page JavaScript and renders screenshots, built on Rust to WebAssembly packages: Blitz for DOM, layout and painting, Stylo for CSS, and Boa as a JavaScript fallback.

The reason it exists is economic. Chromium was built for humans, with tabs, themes and 60 frames per second scrolling, and it is too memory-hungry to give every agent session its own instance. Kitesurf claims 3 to 7 times less memory and CPU than Chromium, at the cost of being about 1.7 times slower at HTML extraction. It speaks the Chrome DevTools Protocol, so Puppeteer and most agent harnesses connect to it without changes. Each page gets an isolated Dynamic Worker, and outbound network access is funnelled through a single sandbox worker on a least-privilege policy.

Put the two releases side by side and the distinction is clean. Kitesurf is infrastructure. It makes it affordable to give every agent session a disposable, isolated browser, and it shrinks what a hostile web page can reach, which matters because prompt injection arrives through page content. If your agent fails on timeouts, memory or hostile pages, Kitesurf helps. If your agent fails because it reasoned badly, Kitesurf tells you nothing about it.

Neither tool ships the person who reads the queue. AQuA assumes you already run Google Cloud telemetry, Cloud Run and Gemini billing, and even at $0.12 per session, a 50,000-session-a-day estate is real money. Kitesurf needs a Workers Paid plan and a Rust toolchain to self-host. Both are open source in the licence sense, not in the free-and-operated sense.

The stack you can actually run

If you want one honest sentence about agent observability in 2026, it is that the tooling has caught up with the instrumentation and not with the judgement. LangChain's State of Agent Engineering survey, published in June 2026 with more than 1,300 respondents, found that 57% of teams have agents in production, 32% name quality as the top barrier, 89% have observability, and only 52% have evals. Most teams can tell you what their agent did. Half can tell you whether it was right.

The open-source stack that works is boring and it is already standard. OpenTelemetry, Apache 2.0, carries spans. OpenLLMetry instruments LLM and agent calls into those spans. Langfuse, MIT licensed at the core, stores traces with token accounting, cost per task, prompt management and replay. Arize Phoenix covers tracing plus evaluation but ships under the Elastic Licence 2.0, which is source-available rather than OSI open source, so check that against your procurement rules before you standardise on it. LangSmith is commercial.

The span tree you want is three levels deep: an agent run, the LLM calls inside it, and the tool calls each LLM call triggers, each carrying token counts, latency and cost. That structure answers the questions that matter in an incident. Which tool call looped. Which step blew the context window. Which prompt change moved cost per task by 40%. Which session produced a confident answer with no supporting retrieval.

Then add the judgement layer. Sample a slice of production traffic daily, review it against a written checklist, cluster the findings, and verify each cluster against the full transcript before you act on it. That is AQuA's loop, and you can run it with any model you like, including one on your own hardware. The point is not Google's implementation. The point is that a finding without citations is a guess, and a guess will send you rewriting prompts that were never the problem.

What to do this week

Start with the cheapest thing that produces evidence. Turn on OpenTelemetry spans around your agent loop and your tool calls, and send them to a self-hosted Langfuse instance. If your agent touches the web, put its browser sessions behind an isolated worker rather than a shared Chromium. Then write down, in one page, what a correct run looks like, because that page is your eval rubric and you cannot outsource it to a vendor.

The order matters. Instrumentation without a rubric gives you a beautiful dashboard of runs you cannot grade. A rubric without instrumentation gives you opinions. Teams that get value from agents run both, and they accept that a model judging a model is a first pass, not a verdict. Google's own verifier rejected one in six candidate clusters. Assume yours will too, and budget a human hour per week to read the queue.

For Australian teams the deployment question is separate but related. Langfuse, OpenTelemetry and Phoenix all self-host cleanly, which means your traces, your prompts and your customer data can stay inside your own environment while the model doing the review runs wherever your obligations allow. That is a procurement decision before it is a technical one, and it is worth making deliberately rather than by default.

Frequently asked questions

What is Google's Ambient Quality Agent?

AQuA is an open-source reference implementation, released on 8 October 2026 under Apache 2.0 inside Google's ADK recipes repository. It samples production agent sessions, reviews them against a nine-point checklist, clusters findings, verifies each cluster against full transcripts, and tracks issues as new, recurring or resolved. It runs as a sidecar and never writes back to the agent.

Is Cloudflare Kitesurf an observability tool?

No. Kitesurf is a browser engine for AI agents that runs on Cloudflare Workers without a browser container. It reduces the memory and CPU cost of giving each agent session an isolated browser, and it limits what a hostile page can reach. It does not tell you whether your agent reasoned correctly.

How much does agent observability cost to run?

Instrumentation with OpenTelemetry and self-hosted Langfuse costs infrastructure and engineering time. Google publishes AQuA run costs of about $0.007 per session for single-agent sweeps and about $0.12 per session for multi-agent sweeps, plus $0.33 to $2.47 per root-cause insight on demand.

Why do AI agents fail silently?

Because most agent failures are quality failures, not availability failures. A wrong answer returns HTTP 200, stays inside its latency budget and raises no tool error, so standard monitoring sees a healthy service. Catching it requires reviewing the content of runs against a rubric, not watching error rates.

Do I need evals if I already have traces?

Yes. Traces record what happened. Evals decide whether it was right. LangChain's 2026 survey found 89% of teams have observability and 52% have evals, which means roughly half the market can replay a failure but cannot grade it.

About the author

AJ Awan is the founder of Flowtivity, an AI consultancy based on the Gold Coast. He is a former EY management consultant and a TOGAF certified enterprise architect, and he builds agent systems for Australian businesses that need them to run in production rather than in a demo.