What Actually Makes A Local LLM Fast, And When Does On-Prem Beat The Cloud?

Last Updated: 31 October 2026

Memory bandwidth makes a local LLM fast, and utilisation decides whether on-prem beats the cloud. Those two sentences carry most of the argument. Everything else is arithmetic.

Two explainers landed this week that suggest buyers are confused about the mechanics. Tom's Hardware published a primer on KV cache, weights, quantization and how inference actually works, and a separate piece on inference hardware argued that local models are being pushed forward by better memory and quantized formats. When primers start appearing in mainstream tech press, it usually means the buying conversation has moved past "should we" into "why is this one slower".

This post covers what actually determines speed, why quantization matters more than parameter count, why concurrency changes the numbers, how to compute cost per million tokens, and where the break-even sits for a business rather than a benchmark.

What actually makes a local LLM fast?

Two different phases with two different bottlenecks. Prefill reads the prompt and builds the KV cache, which is compute-heavy and parallel, so it rewards raw FLOPs. Decode generates one token at a time, and each token requires reading the model's weights from memory, so it rewards memory bandwidth.

That is why a machine with enormous compute and modest bandwidth can feel slow in chat. For a 70B model at four bits, roughly 35 to 40 GB has to move through memory for every token generated. At 273 GB/s, the theoretical ceiling is around seven to eight tokens per second, before any efficiency losses. Our measured 35 tokens per second on DeepSeek V4 Flash comes from a smaller, faster model and a stack tuned for it, not from ignoring the bandwidth limit.

The practical rule: for interactive work, bandwidth is the specification that matters. For batch summarisation of long documents, prefill compute matters more, because most of the work happens before the first output token.

Why does quantization matter more than parameter count?

Because memory is the binding constraint, and quantization is how you fit a capable model into it. Sixteen-bit weights use two bytes per parameter. Eight-bit uses one. Four-bit uses roughly half a byte, plus scales and overhead, which is why a 70B model at four bits lands near 35 to 40 GB rather than 140 GB.

That difference decides which model you can run at all. A 128 GB machine can hold a 70B model at four bits with room for cache and a working context. The same machine cannot hold it at sixteen bits. So the question is not "how many parameters" but "how many parameters at which precision, with how much context, on how much memory".

Quantization has a cost, and it is not uniform. Aggressive quantization degrades the tasks that need precision: arithmetic, structured extraction from messy documents, long-chain reasoning and code that has to compile. The honest approach is to test your own workload at two precisions and compare error rates, rather than trusting a leaderboard that measured a different task.

There is a second memory consumer that surprises people: the KV cache. It grows with context length, layer count and attention width, and at long context it can rival the weights themselves. A model that fits comfortably at 8K tokens may not fit at 128K. If your workflow feeds whole contracts into the prompt, size for the cache, not just the weights.

Why does concurrency change the numbers?

Because decode is bandwidth-bound and batching amortises the read. When four requests run together, the server reads the weights once and applies them to four sequences, so aggregate throughput rises without a proportional rise in memory traffic. Per-stream latency usually rises a little, because each stream now waits its turn.

On our pair of DGX Sparks, a single stream runs at 35 tokens per second. Four concurrent streams together produce 62 tokens per second, so each stream averages around 15.5. That is the shape of every local deployment: one user gets the best experience, four users get acceptable aggregate throughput, and twenty users get a queue.

This is the number that should drive hardware decisions. If your workload is one analyst drafting documents, size for single-stream speed. If it is a back-office queue processing invoices overnight, size for aggregate throughput, because nobody is watching the latency.

What is the real cost per million tokens?

Fixed cost divided by volume. If a pair of machines costs A$15,000 and lasts three years, that is about A$417 a month before power. Add power: two units drawing around 280 W together, running continuously, is roughly 204 kWh a month, which is a small line item next to the capital. Add a few hours a month of oversight for patching and evaluation.

Then divide by volume. At 160 million tokens a month, which is what 62 tokens per second at full duty cycle produces, the fixed line works out to about A$2.60 per million tokens. At 50 million tokens a month, the same hardware costs about A$8.34 per million tokens. At 10 million tokens a month, it costs about A$41.70 per million tokens, which is usually worse than paying per token.

The arithmetic is simple, and the conclusion is uncomfortable: the hardware is only cheap if you use it. Idle capacity is the most expensive way to run a local model, because you paid for the bandwidth and nobody is reading the weights.

Those prices are illustrative, not a quote. Substitute your own invoice and your own volume. The formula is what matters, because it makes the trade-off visible: cost per million tokens equals monthly fixed cost divided by monthly volume in millions.

When does on-prem beat the cloud?

When three conditions hold at once. Volume is steady and high enough to keep the hardware busy. The data is sensitive enough that custody matters, or the workflow touches records you would rather not send to a third party. And the task sits within the capability of a model you can run locally, at a precision that passes your own evaluation.

If all three hold, the local stack usually wins on cost and always wins on custody. If one fails, the calculation changes. Bursty volume makes the fixed cost per token brutal. Frontier-capability tasks push you back to hosted models. And if your evaluation shows the local model failing on the task, no amount of cost advantage matters, because the output is not usable.

The practical pattern for most mid-market businesses is hybrid: local for the high-volume, well-understood tasks such as drafting, extraction, classification and summarisation, and hosted for the hard cases. That is not a compromise, it is the design that keeps unit costs down without pretending a small model can do everything.

When does on-prem not beat the cloud?

Low volume, bursty demand, frontier reasoning, vision-heavy work and anything that needs a model released this month. Hosted providers amortise hardware across thousands of customers and refresh models constantly. A single pair of machines cannot match that on capability or on elasticity, and pretending otherwise produces disappointing pilots.

There is also an operational cost that rarely appears in the spreadsheet: the stack is yours to patch, monitor and re-evaluate. Model updates, driver updates, quantization changes and prompt drift all need someone's attention. That is real work, and it should be priced into the comparison rather than discovered later.

What does this mean for an Australian business?

Three local factors shift the arithmetic in favour of on-prem. Data residency questions are easier to answer when the hardware is in your office. Currency exposure disappears when there is no per-token bill. And egress costs, both financial and contractual, drop when the data does not cross a border.

Against that, electricity in Australia is not cheap and hardware arrives with GST and freight. The honest comparison includes those lines. It also includes the cost of the person who runs it, which is the line most business cases omit.

If you want the measurement method we use for throughput and concurrency, our post on running agents fully on-premises covers the stack and the numbers. If you want a cost model built on your own volumes, talk to us.

Frequently asked questions

Why is a local LLM slow on a powerful machine?

Because token generation is bandwidth-bound. Each token requires reading the model's weights from memory, so a machine with high compute but modest memory bandwidth will feel slow in chat. Memory bandwidth, not FLOPs, sets the single-stream ceiling.

Does quantization make a model worse?

It can, and not uniformly. Four-bit quantization is usually acceptable for drafting, classification and summarisation, and noticeably weaker on arithmetic, precise extraction and long-chain reasoning. Test your own workload at two precisions rather than trusting a generic benchmark.

What is the KV cache and why does it matter?

It is the stored attention state for the tokens already processed. It grows with context length and can rival the weights at long context, so a model that fits at 8K tokens may not fit at 128K. Size for the cache, not just the weights.

How do I calculate cost per million tokens for on-prem?

Divide the monthly fixed cost by the monthly token volume in millions. A A$15,000 pair over three years is about A$417 a month, which is A$2.60 per million tokens at 160 million tokens a month and A$41.70 at 10 million.

When should a business stay on hosted models?

When volume is low or bursty, when the task needs frontier capability or vision, or when nobody is available to patch and re-evaluate the stack. Hosted providers amortise hardware across many customers, which is hard to beat at low utilisation.