How Do You Test An AI Agent Before It Touches A Live Business Workflow?

Last Updated: 31 October 2026

You test an agent by grading its behaviour on a fixed set of real business cases, in a sandbox, before it is allowed to act on live data. Not by reading its output once and deciding it looks good. The difference matters because agents are non-deterministic: the same input can produce a different output on a second run, which breaks the assertion-based testing most teams already know.

LangChain published a clear articulation of the workflow this week: offline evals use curated datasets to test changes before release, online evals use real production traces to monitor live traffic, and production failures should become regression tests so resolved issues do not return. Analytics Insight published a parallel guide on testing LLMs, agents and generative applications, and Hostinger's roundup of agentic frameworks shows how many options a team now faces. The tooling is plentiful. The discipline is not.

This post covers why agent testing is different, how to build an eval set, how to test tool calls safely, what shadow mode looks like, how to graduate autonomy, and what to do when production fails.

Why is testing an agent different from testing software?

Because the output is not a fixed value. Traditional software testing asserts that input X produces output Y. An agent given input X might produce Y, a reasonable variation of Y, or something confidently wrong, depending on sampling, context and the tools it chose to call. The test has to grade whether the behaviour is acceptable, not whether it matched a string.

There is a second difference: the agent's behaviour depends on its tools. A change to a CRM field, a new API version or a rate limit can alter outcomes without any change to the prompt. So the test surface includes the systems the agent touches, which means the test environment has to resemble production closely enough to be meaningful.

The third difference is compounding. A multi-step agent can make a small error at step two and a catastrophic one at step seven. Grading only the final output hides where the failure started, which is why trace-level checks matter as much as outcome checks.

How do you build an eval set?

From real work, not from imagination. Pull the last quarter of actual tasks: the invoices, tickets, enquiries or documents the workflow will handle. Pick 50 to 100 cases that cover the common path, the awkward variations and the known exceptions. Then write down what a competent human would do for each one, in enough detail that a grader can judge it.

Synthetic examples are useful for edge cases you cannot find in the archive, but they should be a minority. The failures that hurt in production usually come from the messy middle of real work: the supplier whose name is misspelled, the customer who replies in a different thread, the invoice with two tax lines. Synthetic data rarely reproduces that mess.

Grade with a rubric, not a vibe. For each case, define what correct looks like, what acceptable looks like and what failure looks like. Then have a person grade a sample, and compare their grades with an LLM judge. Where the judge and the human disagree, fix the rubric. That calibration step is what makes automated grading trustworthy enough to run on every change.

What is the difference between offline and online evals?

Offline evals run against your curated set before a change ships. They are fast, repeatable and cheap, and they answer the question "did this change break anything we already know about". Run them on every prompt change, model swap or tool update, and treat a regression as a blocker.

Online evals run against real traffic after release. They catch the cases your eval set did not imagine, which is most of them over time. Sample the traces, grade a subset, and watch for drift in the distribution of tasks as well as in the quality of answers.

The loop between them is the point. A production failure gets added to the offline set, so the same failure cannot return silently. Over a year, that set becomes the most valuable asset in the project, because it encodes what your business considers correct.

How do you test tool calls safely?

In layers. First, a sandbox with recorded responses: replay a set of realistic API responses so the agent can be tested without touching a live system. Second, a dry-run mode where write calls are logged rather than executed, so you can inspect what the agent would have done. Third, a test tenant with real but disposable data, where writes are allowed and then rolled back.

The sandbox catches logic errors. The dry run catches intent errors, which are the dangerous ones: the agent understood the task but decided to change the wrong record. The test tenant catches integration errors, such as a field that behaves differently from the documentation.

Do not test against production with a read-only key and call it safe. Read-only still exposes data, and the agent may behave differently when its write tools fail. The behaviour you need to observe is the behaviour it will have with full permissions.

What does shadow mode look like?

The agent runs alongside the human process without acting. It receives the same work, produces its recommendation, and the human proceeds as normal. You then compare the agent's output with what the human did, and record the disagreements.

Two weeks of shadow mode on a real queue tells you more than any benchmark. You learn the true distribution of tasks, the cases where the agent is confidently wrong, and the approval rate you can expect. It also gives the team something to react to, which reduces the resistance that appears when an agent arrives with a mandate instead of evidence.

Set a threshold before you start. For example: 95% agreement on the graded set, zero failures on the irreversible actions, and an exception rate the team can absorb. If the numbers are not met, keep shadowing or narrow the scope. Deciding the threshold after seeing the results is how pilots get approved on hope.

What does graduated autonomy look like?

Four stages, each with its own permission set. Suggest: the agent drafts, the human does everything. Approve: the agent drafts and the human approves with one click, so you can measure acceptance. Act with limits: the agent executes within a defined envelope, such as invoices under a value threshold, with everything outside escalated. Unattended: the agent runs the queue with sampling and alerts.

Most workflows should stop at stage three. Full autonomy is appropriate only where the action is reversible, the volume justifies it and the evidence from stages one to three is strong. Moving a stage should require a review of the numbers, not a feeling that it is going well.

Each stage also changes the human role, and that is where projects fail socially rather than technically. The person who used to process invoices now reviews exceptions, which is a different job with different satisfaction. Plan for that explicitly, or the agent will be quietly sabotaged.

What do you do when production fails?

Three things, in order. Capture the trace so you can see what the agent did and why. Add the case to the offline eval set with the correct answer. Then re-run the full set to confirm the fix did not break something else.

That third step is the one teams skip. A prompt change that fixes one failure often creates another, because instructions interact. Running the full set after every fix is the cheapest insurance available, and it takes minutes once the harness exists.

Keep a failure log with the date, the case, the impact and the fix. After six months it becomes the best documentation of the workflow's real behaviour, and it is far more persuasive to an auditor than a policy statement.

Why is this ROI protection rather than engineering purity?

Because the cost of a bad agent is not the licence fee, it is the rework. A workflow that processes 95 invoices correctly and five wrongly has not saved 95 units of work. It has saved 95 and created five investigations, plus the trust cost of someone having to check everything again.

Testing is how you find out whether the saving is real before you claim it. The eval set tells you the error rate, shadow mode tells you the exception rate, and graduated autonomy tells you what the team can absorb. Those three numbers are the business case, and they are more credible than any vendor demonstration.

If you are building the business case itself, our post on measuring agent ROI beyond vendor case studies covers the calculation. If you want the eval harness and rollout plan built for your workflow, talk to us.

Frequently asked questions

How many test cases does an AI agent need?

Fifty to a hundred cases drawn from real work is enough to catch most failures before launch. Cover the common path, the awkward variations and the known exceptions. Synthetic cases should be a minority, because production failures usually come from messy real data.

Can I test an agent without touching production?

Yes, in layers: a sandbox with recorded API responses, a dry-run mode where writes are logged instead of executed, and a test tenant with disposable data where writes are allowed and rolled back. Read-only production keys are not a safe substitute.

What is shadow mode?

The agent runs alongside the human process without acting. It produces recommendations, the human proceeds as normal, and you compare the two. Two weeks on a real queue reveals the true task distribution, the confident failures and the approval rate you can expect.

When should an agent run unattended?

Only when the action is reversible, the volume justifies it, and the evidence from earlier stages is strong. Most workflows should stop at acting within a defined envelope, with anything outside escalated to a person.

What should happen after a production failure?

Capture the trace, add the case to the offline eval set with the correct answer, then re-run the full set to confirm the fix did not break anything else. Prompt changes interact, so a fix for one case can create another failure.