Can You Run AI Agents Fully On-Premises Without Handing Your Data To A Cloud Vendor?

Last Updated: 30 October 2026

Yes, you can run AI agents fully on-premises today, and the pattern that makes it work became public on 29 October 2026, when Oxide Computer published references for running self-hosted agents in virtual machines. The design is not exotic. A long-lived orchestrator virtual machine accepts work orders, creates an ephemeral virtual machine for each agent session, and deletes it when the agent process exits.

Two releases this month moved the local stack from demo to practical. Ollama shipped speech models on 24 October 2026, and on a MacBook Pro with M5 Max its Phonon implementation ran 35% faster than Gradium's own runtime. Oxide's references answer the question that stopped most teams: how do you provision the machine an agent runs on, and how do you clean it up afterwards.

Practical is not the same as free or frictionless. A working stack needs five parts, three of them fail in ways that are easy to miss, and the cost comparison against per-token APIs depends on duty cycle rather than sticker price. This post covers what the stack needs, where it breaks, what it costs, and what an ephemeral VM per session means for an Australian business under the Privacy Act 1988.

What does a working on-premises agent stack actually need?

A working on-premises agent stack has five parts. An orchestrator accepts work orders and provisions sessions. An isolation layer gives each session its own virtual machine. A tool gateway brokers every outbound call the agent makes. A model serving layer holds the weights and answers inference requests. An audit trail records what each session did, with which credentials, and for how long. The model is the least interesting component. Most failed self-hosted pilots have a good model and no gateway, no isolation, or no log, which means nobody can say what the agent touched last Tuesday.

The orchestrator is the only long-lived piece. On Oxide it runs as a virtual machine that authenticates to the self-hosted environment and waits for work orders, which are requests to create an ephemeral runner for a user's session. When no work orders arrive, the number of runners scales down to zero. That is the property that makes the economics work. Idle capacity costs nothing because it does not exist.

The isolation layer is where most teams cut corners. Running agents as containers on a shared host is cheaper and faster, and it is the wrong default for anything that can write. Oxide's guidance is direct on this point: virtual machines give stronger isolation than containers because each one runs a separate guest kernel rather than sharing the host kernel.

The tool gateway is the part nobody budgets for. An agent is only useful when it can act, and every action is a credential. A gateway that holds the credentials, exposes a fixed list of tools, and logs each call keeps the agent's blast radius small. Without it, the agent inherits whatever the harness process can reach, which on a typical server is everything.

Model serving is the part people expect. The harness runs as a process on a CPU and talks to a model over the network. When the model asks for a tool call, the harness executes it on the CPU and sends the result back. Oxide puts the same idea more sharply: AI becomes agentic when GPU-based inference needs CPU-based tool execution. That loop is why an agent box needs CPU capacity as well as GPU capacity, and why a GPU-only plan tends to stall at the tool-calling stage.

The audit trail is what turns the stack into something you can defend. Per-session logs tied to a virtual machine identity give you a record of what ran, what it read, and when it stopped. That record is also the artefact a privacy officer will ask for, and the one most self-hosted pilots cannot produce.

Why does each agent session need its own virtual machine?

Because an agent executes tool calls with real credentials, and a shared container lets one bad call reach another tenant's data or delete a shared volume. A virtual machine gives each session a separate guest kernel, so a kernel-level fault in one session does not touch the host or its neighbours. The trade is startup time: seconds rather than milliseconds, and gigabytes rather than megabytes. That trade is worth paying when the workload can delete production data, and it is the difference between a demo and a system you can leave running overnight.

Matthew Sanabria, Solutions Software Engineer at Oxide, framed the starting point plainly. "The initial answer is simple. You run AI agents the same way you run other workloads on Oxide, in virtual machines. Customers already run long-lived services in virtual machines, so running AI agents feels familiar," he said.

The nuance is that agents are not quite long-lived services. They sit between that and an ephemeral CI/CD job. Run a long-lived agent that provisions sub-agents and it behaves like a service. Run an agent to complete a bounded task and it behaves like a build job. Either way the workflow is the same: provision a machine, run the agent, clean up the machine.

Cleanup is the part that usually gets hand-waved, and Oxide's example handles it in a way worth copying. The ephemeral virtual machine is configured to power off once the agent process finishes. A stopped machine is the signal. The orchestrator has no cleanup hook of its own, so a separate reaper process polls the API for stopped machines matching a name prefix, then deletes each one along with its boot disk.

That detail matters more than it looks. A machine that powers off and is then deleted with its disk has a bounded retention window. You can tell a client, or an auditor, that the session's disk no longer exists. A container that is stopped but whose volume persists cannot make that claim, and neither can a hosted agent where the vendor holds the transcript.

What changed this month that moved local agents from demo to practical?

Two things. Oxide published the orchestration pattern on 29 October 2026, which removes the guesswork about provisioning and cleanup. Ollama shipped speech models on 24 October 2026, and its Phonon implementation ran 35% faster than Gradium's own runtime on a MacBook Pro with M5 Max, measured over 45 seconds of speech as the median of 10 runs. Together they mean the serving layer and the lifecycle layer both have a reference implementation you can copy rather than invent.

Phonon is a family of small text-to-speech models from Gradium, and it is the first speech model in Ollama. It arrived on macOS first, with Windows and Linux support listed as coming. Ollama reports that its implementation matches the word error rate of the reference implementation in Hugging Face Transformers, tested on 200 sentences from the Seed-TTS English test set transcribed with Whisper large-v3. That accuracy claim is the vendor's own, and it is the kind of claim worth re-testing on your own audio before you put it in front of customers.

The licence is where local deployments get caught. Phonon itself is Apache 2.0, which is workable for commercial use. The 40M voice variants are released under CC BY-NC 4.0, which limits them to noncommercial use. A business that pipes customer-facing audio through those variants has a licensing problem, not a technical one, and it will not show up in any benchmark.

The practical value is broader than speech. Ollama exposes Phonon through the OpenAI-compatible /v1/audio/speech endpoint, so existing code that uses the OpenAI SDK needs a new base URL and nothing else. Five languages ship with it. That pattern, a local runtime that speaks the same API as the hosted one, is what lets a team move a workload on-premises without rewriting the application around it.

Where does the local stack still fall short?

Three places. Vision is the weakest link, because open-weight vision models still lag on document layout, small text and chart reading, and most real agent work touches a PDF or a screenshot. Tool-calling quality varies by model, so a model that writes excellent prose can still emit malformed arguments and stall a loop. GPU memory sets a hard ceiling, because 128 GB of unified memory has to hold weights, KV cache and context at the same time, and a large model at long context leaves little headroom for concurrency.

The memory maths is worth doing before you buy anything. A DGX Spark carries up to 128 GB of unified memory shared between the GPU, the CPU and the operating system, with 273 GB/s of memory bandwidth. A 70B-class model at 4-bit precision needs roughly 35 GB just for weights, before any context is loaded. Add a long context window and four concurrent sessions and the headroom disappears faster than most people expect.

Bandwidth explains the throughput more than raw compute does. Token generation is memory-bound: every token requires reading the active weights, so the ceiling is bandwidth divided by bytes read per token. At 273 GB/s and 35 tokens per second, each token is reading roughly 7.8 GB of weights. That is what you would expect from a mixture-of-experts model that activates a fraction of its parameters per token, and it is why a bigger GPU with more bandwidth helps more than a faster one with less.

Our own measurement on two DGX Sparks running DeepSeek V4 Flash came out at about 35 tokens per second on a single stream and 62 across four concurrent streams. That is our testing on our hardware, not a benchmark, and it is the number we plan capacity against. The jump from 35 to 62 is the useful part: batching four sessions buys most of the throughput back without doubling the hardware.

Vision is the gap that decides which tasks you can move. Text extraction from a clean PDF is usually fine. Reading a scanned invoice with a handwritten annotation, or interpreting a dashboard screenshot, is where local models still need a fallback. The honest design is to route those calls to a named exception rather than pretend the local model handles everything.

Is owned hardware cheaper than per-token cloud APIs?

It depends on duty cycle, not on price per token. A per-token API charges nothing when idle and everything at volume. Owned hardware charges the same whether it is busy or idle. At 62 tokens per second across four streams, a pair of Sparks can produce about 160 million tokens a month if it runs flat out. Below a few million tokens a month the API wins on cost. Above that, and wherever data cannot leave, the hardware wins.

There are two separate cost lines and they are usually mixed together in vendor comparisons. The cloud line is metered: a price per million tokens, multiplied by volume, plus whatever the vendor charges for the orchestration layer. The local line is capital plus power plus the time of whoever operates it. Neither line is free, and the local line is fixed, which is the whole point. A fixed cost is a budget you can plan against. A metered cost is a budget that moves when your usage does.

The capacity arithmetic is simple enough to do on a napkin. At 62 tokens per second, a pair of Sparks produces 223,200 tokens an hour, which is 5.36 million a day, which is about 160 million in a 30-day month at 100% duty cycle. Real duty cycle is lower, often much lower, because agent work arrives in bursts. Even at a quarter of that, 40 million tokens a month is more than most mid-sized businesses will consume.

Power is the smaller line. Each DGX Spark has a 140 W GB10 TDP, so a pair draws about 280 W at the chip level, which is roughly 204 kWh a month if both run continuously. Add the rest of the machine and a switch and you are still in the range of a small office appliance rather than a server room. We are not quoting a hardware price here because it moves with region and generation. The arithmetic you need is capex divided by months of service, plus power, plus the hours someone spends keeping it patched.

We will not pretend the hardware pays for itself at single-user volume. It does not. One person using an agent a few hours a day is better served by a subscription or a per-token API, and saying otherwise would be dishonest. The case for owned hardware is four conditions: several staff sharing one agent, API volume where per-token billing hurts, data that cannot leave your control, and work a hosted plan will not permit.

Cost line	Per-token cloud API	Owned on-premises pair
What you pay for	Tokens consumed, plus orchestration fees	Hardware, power, and operator time
Cost when idle	Zero	Fixed, whether busy or not
Marginal cost per token	Set by the vendor and can change	Effectively zero once paid for
Practical ceiling	Rate limits and quota	About 160 million tokens a month at full duty cycle
Where the data goes	To the vendor's infrastructure	Stays on hardware you control
Failure mode	Price change, outage, policy change	Hardware fault, memory ceiling, patching debt

What does this mean for an Australian business?

Under the Privacy Act 1988, APP 8.1 requires an APP entity to take reasonable steps before disclosing personal information to an overseas recipient to ensure the recipient does not breach the Australian Privacy Principles. Section 16C makes the entity accountable for the recipient's acts and practices. An ephemeral virtual machine per session turns that obligation into something you can evidence: the data stayed on hardware you control, in a session you can name, for a period you can bound.

That is the custody story, and it is stronger than the hygiene story. Hygiene says the agent cannot reach production data. Custody says you can prove where the data went and when it stopped existing. The second claim is the one a board, a client or a regulator will test, and it is the one a hosted agent cannot make on your behalf, because the transcript sits with the vendor.

The engineering consequence is that session identity has to be a first-class record. Each session needs an identifier that ties to a virtual machine, a start time, an end time, and a deletion event. If your orchestrator creates sessions but cannot tell you when a session's disk was destroyed, you have hygiene without custody.

The procurement consequence is a short list of questions worth asking any vendor, including us. Where does inference actually run? Do prompts and completions leave the country? What is the retention period, and can it be shortened? Can you produce a per-session log on request? Who inside the vendor can log in to the environment? A vendor that cannot answer the fourth question is selling you a black box with a privacy policy attached.

One caution. APP 8 has exceptions, and there are legitimate arrangements where a cross-border disclosure is permitted. Whether a specific arrangement fits an exception is a legal question, not an engineering one. The point here is narrower: if the data never leaves, the question does not arise, and that is a simpler position to defend than an exception you have to keep justifying.

How do you start without rebuilding everything?

Start with one workflow, one model and one machine. Pick a task with a bounded output and a clear failure mode, such as drafting a reply or summarising a document. Run the orchestrator as a virtual machine, give each session its own ephemeral machine, put every tool call behind a gateway with an allowlist, and log the session from start to deletion. Measure tokens per second and task completion for two weeks before you buy a second machine.

The order matters. Teams that start with the model spend months tuning prompts and then discover the agent cannot be trusted with credentials. Teams that start with the gateway and the isolation layer get a boring, safe system that works with a mid-sized model, then improve the model later. The second path ships. The first path produces a demo that never leaves the laptop.

We took the second path. Our own stack runs on two DGX Sparks with DeepSeek V4 Flash, and the design work that mattered was not the model choice. It was deciding which tools the agent could call, what each call was allowed to touch, and how a session ended. That work is the same whether you run one machine or twenty, and it is the part that makes an on-premises agent defensible rather than merely local.

If you are mapping a workflow before you commit to hardware, our write-up on designing an agent around a real content pipeline walks through the same decisions at the workflow level. If you want a second opinion on whether your workload belongs on-premises at all, talk to us before you buy the hardware.

Frequently asked questions

Can AI agents run entirely on-premises without a cloud vendor?

Yes. The orchestrator, the model serving layer, the tool gateway and the audit trail can all run on hardware you own. The one caveat is capability, not architecture: open-weight vision models still lag on scanned documents and screenshots, so some workflows need a named exception for those calls rather than a blanket on-premises rule.

What hardware do you need to self-host AI agents?

You need GPU capacity for inference and CPU capacity for the harness and tool calls. A DGX Spark carries up to 128 GB of unified memory with 273 GB/s of bandwidth, and up to four can be networked together. Our pair of Sparks runs DeepSeek V4 Flash at about 35 tokens per second on one stream and 62 across four.

Is self-hosting AI agents cheaper than using an API?

Only above a certain duty cycle. A per-token API costs nothing when idle, while owned hardware costs the same either way. At 62 tokens per second, a pair of Sparks can produce about 160 million tokens a month at full duty cycle. Below a few million tokens a month, the API is cheaper. Above that, and wherever data cannot leave, the hardware wins.

What is an ephemeral VM per agent session?

It is a virtual machine created for one agent session and destroyed when that session ends. Oxide's reference configures the machine to power off when the agent process finishes, then a reaper process deletes it along with its boot disk. The result is a bounded retention window you can describe to a client or an auditor.

Does running agents on-premises satisfy Australian privacy obligations?

It removes one obligation rather than satisfying all of them. APP 8.1 under the Privacy Act 1988 applies to disclosing personal information to an overseas recipient, and section 16C makes you accountable for that recipient. If the data never leaves your hardware, that disclosure question does not arise. The rest of the APPs still apply.