Should Your AI Agent Harness Be Open Source, Or Locked To One Vendor?

Last Updated: 30 October 2026

An open harness is worth the effort when you can move one workflow to a different model in a day without rewriting it. Most teams cannot, and the reason is not the model. It is the memory store, the tool schema, the hosted sandbox and the evaluation data that were never designed to leave.

The market moved in both directions this week. On 28 October 2026, OpenAI and ServiceNow expanded a partnership that connects ServiceNow workflows to ChatGPT and Codex, and puts OpenAI models and agents inside ServiceNow products. The same day, Infini-AI-Lab published Hermit, an Apache 2.0 framework for training and deploying harness adapters that adapt an agent to new tasks without touching its model or its harness.

One announcement sells integration. The other sells portability. Both are reasonable, and the choice between them is the most consequential architecture decision an agent team makes in 2026. This post covers where lock-in actually hides, what an open harness buys you, where managed platforms genuinely win, and the five checks that tell you which side of the line you are on.

What actually creates vendor lock-in in an agent?

Four things, and none of them is the model. The first is the memory store: conversation history, embeddings and long-term context that live in a proprietary database you cannot export in a usable shape. The second is the tool schema: function definitions written against one vendor's SDK, with vendor-specific quirks baked into argument names and error handling. The third is the sandbox: the code execution environment, where your agent's actual work happens, usually with vendor-managed files and no export path. The fourth is the evaluation set: the prompts, the expected outputs and the scores you built to prove the thing works.

Of those four, the evaluation set is the one teams forget, and it is the one that hurts most. Your evals are the only objective record of whether a model change is an improvement or a regression. If they live inside a vendor's dashboard and cannot be replayed against a different model, then every future model decision is a leap of faith. You have outsourced the one measurement that keeps the vendor honest.

The model itself is the easy part. Base URLs are compatible, and most inference servers speak the OpenAI API shape. Swapping a model is a configuration change. Swapping a memory store, a tool layer and an eval harness is a project.

What is an agent harness, and why does it decide your portability?

The harness is everything around the model: the loop that decides what to do next, the tool definitions, the context assembly, the retry logic and the state handling. The model answers questions. The harness turns answers into actions. That is why the harness, not the model, is where portability is won or lost.

Hermit's framing is precise and worth quoting. "A harness adapter is a trainable external harness, like a plugin with built-in training support. Just as a model adapter adapts a model, a harness adapter adapts an existing agent to new scenarios without modifying its original harness or model." The framework, released at version 0.1.0 under Apache 2.0 on Python 3.12 or later, trains, evaluates and deploys those adapters, and it works even when the model and the harness are black boxes.

Two properties matter for a business. The first is that adapters are transferable across components, so an adapter trained for one workflow can be reused elsewhere. The second is that the harness stays frozen, which means upgrading the adapter does not force a rewrite of the thing you already tested. That is a meaningful change to how agent teams think about improvement: you version the adapter, not the whole stack.

The release also landed with a Hacker News discussion, which is worth reading for the scepticism rather than the enthusiasm. The recurring question is whether an adapter layer is a genuine abstraction or an extra moving part that hides the model's failures. That is a fair question, and the answer depends on whether you can measure the adapter's effect against a fixed eval set.

What does an open harness buy you in practice?

Three things you can actually test. The first is model choice: you can point the same workflow at DeepSeek, Qwen, Llama or a hosted frontier model and compare results on your own evals, rather than on a leaderboard. The second is cost control: you can route the cheap classification steps to a local model and the hard reasoning steps to a hosted one, without maintaining two versions of the workflow. The third is negotiating position: a vendor that knows you can leave prices differently from one that knows you cannot.

Our own stack is the worked example. The harness runs on a CPU, the model runs on a pair of DGX Sparks serving DeepSeek V4 Flash, and the tools sit behind a gateway. Because the model is addressed through an OpenAI-compatible endpoint, changing models is a base URL and a model name. Because the state lives in plain files rather than a vendor database, a session can be replayed on a different machine. Because the tools are described once in a protocol rather than in a vendor SDK, the same tool definitions work on both sides.

The protocol part is where most of the practical gain sits. The Model Context Protocol gives tools and data sources a standard description, so an agent can discover what a system can do without a bespoke integration per vendor. It does not make every tool portable, but it removes the most common reason teams cannot move: the tool layer being written against one platform's SDK.

There is a cost, and it is real. An open harness means you own upgrades, compatibility testing and the failure modes that a vendor would otherwise absorb. That is a fixed operating cost, and it only pays back if you actually use the portability you paid for.

Where does a managed platform genuinely win?

Operationally, almost everywhere at the start. A managed platform gives you a working agent, an audit log, an approval flow and a support contract in a week. It handles the patching, the scaling and the incident response. For a team without a platform engineer, that is not a compromise. It is the only realistic option.

The ServiceNow and OpenAI expansion shows the shape of that value. The new integration connects ChatGPT to a customer's own ServiceNow instance, and ServiceNow's Action Fabric applies the customer's existing business rules and permissions, so the actions people can take reflect the organisation's configuration and access. ServiceNow remains the system where actions execute and are governed. That is a genuinely good design for a large enterprise, because governance stays with the system of record.

Greg Brockman, OpenAI's president and co-founder, framed the intent: "AI becomes more useful when people can put it to work in the systems their businesses depend on. By bringing our models together with ServiceNow's workflows and business context, we're helping teams build software, address security risks, and put AI to work across their organizations."

The scale is not small. The January 2026 agreement between the two companies described enterprises running more than 80 billion workflows a year on the platform, which is roughly 2,537 workflows every second by our arithmetic. When an agent plugs into that, it inherits reach that no self-hosted stack can replicate, and it inherits the governance that goes with it.

The honest counterpoint is that this convenience is the lock-in. The workflow logic, the permissions and the audit trail live in ServiceNow. If the commercial terms change, moving the workflow means rebuilding it, because the agent was never separate from the platform it ran on.

What is the portability test?

Five checks, each answerable in a day. First, can you export the agent's state in a format another system can read? Second, can you re-point the model at a different base URL without editing code? Third, can you replay your evaluation set against the new model and get comparable scores? Fourth, can you swap one tool for an equivalent without touching the workflow? Fifth, can you name every person who can log in to the environment where the agent runs?

A stack that fails two or more of those is locked in, whatever the contract says. The fifth check is the one procurement teams forget, and it is the one that matters when a client asks who can see their data.

Run the test before you sign, not at renewal. Ask the vendor for a state export in a documented format, an eval replay against a model of your choosing, and a written list of roles with access to your environment. A vendor that cannot provide the first two is selling you a workflow you will have to rebuild if you ever leave.

What does this mean for an Australian business?

For an Australian business, the lock-in question usually arrives at contract renewal, when the price moves and the data cannot follow. That is a bad time to discover that the memory store exports as a proprietary blob and the evals live in a dashboard.

Two practical points. First, data residency is a portability issue: if prompts, completions and tool results sit in a vendor environment, you need a documented export and deletion path, and you need to know which jurisdictions the data transits. Under the Privacy Act 1988, APP 8 makes you accountable for what an overseas recipient does with personal information, so the export path is also a compliance artefact. Second, procurement should treat the eval set as an asset. If you cannot take your measurements with you, you cannot compare a replacement vendor on anything except their own marketing.

The middle path is usually the right one for a mid-sized Australian business. Keep the workflow and the state in your own repository, use an open harness, point it at a local model for the sensitive steps, and use a managed platform where the governance it provides is worth more than the exit cost. That is not purity. It is portfolio management.

How do you start?

Start by writing down the four lock-in points for your current stack and scoring each one: memory, tools, sandbox, evals. Whichever scores worst gets fixed first, and the fix is usually smaller than expected. Exporting state to plain files is a week. Rewriting tool definitions against a standard protocol is a fortnight. Building an eval set of fifty real cases is a day, and it is the highest-leverage day in the whole project.

Then run the swap. Take one workflow, point it at a second model, and replay the evals. If the results are comparable, you have portability and a negotiating position. If they collapse, you have learned something about your workflow that no vendor demo would have told you.

If you want a concrete example of an open harness doing real work, our post on building an agent around a content pipeline walks through the same decisions at the workflow level. If you are weighing a managed platform against a self-hosted stack, talk to us before you sign the contract.

Frequently asked questions

Is vendor lock-in really about the model?

Rarely. Base URLs are compatible and models are swappable. Lock-in lives in the memory store, the tool schema, the hosted sandbox and the evaluation data. If any of those cannot be exported in a usable format, changing models will not free you.

What is an agent harness?

The harness is everything around the model: the decision loop, tool definitions, context assembly, retries and state handling. The model answers, the harness acts. Portability is decided at the harness layer, not at the model layer.

What does Hermit do?

Hermit is an Apache 2.0 framework from Infini-AI-Lab for training, evaluating and deploying harness adapters. An adapter adapts an existing agent to new scenarios without modifying its original harness or model, and works even when both are black boxes.

When is a managed agent platform the better choice?

When you lack the platform engineering to run inference, isolation and audit yourself, and when the governance a platform provides is worth more than the exit cost. For most teams the right answer is a hybrid: own the workflow and the evals, rent the parts that are genuinely commodity.

How do I test whether my agent stack is portable?

Five checks: can you export state, re-point the model without code changes, replay your evals against a different model, swap one tool, and name everyone with access to the environment. Failing two or more means you are locked in, whatever the contract says.