The alpha in served intelligence: what 155 trillion tokens a day really tells you

Last Updated: 25 October 2026

On 20 October 2026 DeepSeek open sourced DeepInfer, the inference framework behind its production service, and put two numbers in the same paragraph: a single cluster serving 155 trillion tokens a day, at US$38 of revenue per GPU-hour. Those two numbers are more useful than any benchmark score, because they let you check the business model instead of arguing about it.

Run them against DeepSeek's own published price sheet and they close. At a 96% KV cache hit rate, 155 trillion tokens a day works out to about US$2.22 million of revenue a day, which is US$38.61 per GPU-hour across the roughly 2,400 GPUs the cluster implies. The disclosed revenue figure and the public list prices agree to within a rounding error.

That reconciliation is the alpha, and it is not the model. Served intelligence is not a compute business with a cache bolted on. It is a cache business that happens to need GPUs, and once you see it that way, the pricing, the open source strategy and the opportunity for anyone building on top of it all line up.

What did DeepSeek actually open source?

DeepInfer is an Apache 2.0 inference framework for one model, DeepSeek-V4.1-Flash, running on NVIDIA SM100 chips (B200 and GB200 class) and Huawei Ascend 950. It ships the optimised kernels, expert parallelism with load balancing, speculative decoding, SWA bounded replay, asynchronous scheduling, batch invariant inference and structured output handling. According to the GitHub repository, it had 22,138 stars and 2,544 forks nine days after it was created on 16 October 2026.

The codebase is 398 files of mostly Python and CUDA, with TileLang kernels for both NVIDIA and Ascend. Its own README credits vLLM as the original base, with ideas borrowed from SGLang and TensorRT-LLM. It supports exactly one model and two hardware families. If you were hoping to run it on a DGX Spark or a workstation GPU, this is not that release.

The important part is the section headed "Required for a complete inference service (not included in this release)". DeepSeek names three missing pieces: a KV cache system, vision encoder, prefill and decode disaggregation, and load balancing across data parallel replicas. In plain terms, the open source covers the parts that make a single engine fast, and withholds the parts that make a fleet cheap.

Component	In the Apache 2.0 release	Needed to serve at scale
Optimised kernels (FlashMLA, DeepGEMM, DeepEP, DeepSelect, TileLang)	Yes	Yes
Expert parallelism with EPLB	Yes	Yes
DSpark speculative decoding with confidence scheduling	Yes	Yes
SWA bounded replay	Yes	Yes
Async scheduler, structured outputs, deterministic sampling	Yes	Yes
KV cache system (3FS store, 72 hour retention, swap in)	No	Yes
Vision encoder, prefill and decode disaggregation	Partly	Yes
Prefill and decode load balancers across replicas	No	Yes

DeepSeek also open sourced deepseek-recipe, an MIT licensed Rust component that converts between the Chat Completions, Responses and Messages API formats and the model's token sequences. According to the Q&A document, its processing logic is "fully aligned with the official DeepSeek API", so a third party deployment behaves like DeepSeek's own service.

How does one cluster serve 155 trillion tokens a day?

Three design choices do the work. Vision encoder, prefill and decode run as separate pools, so each stage is sized independently. Decode uses wide expert parallelism of 64 to 128 GPUs per instance, which cuts the memory traffic each decode step needs. And a KV cache cluster of about 100 PiB of SSD holds computed context for more than 72 hours, so 96% of input tokens never need to be recomputed.

According to the DeepInfer deployment Q&A, prefill runs at expert parallelism of 2 to 8 on ordinary 8 GPU machines, using the smallest setting that HBM allows. Decode runs at expert parallelism of 64 to 128 on 64 to 128 GPU supernodes, using the largest setting the hardware allows. The prefill to decode GPU ratio is roughly 1:1, which makes the smallest efficient deployment 64 GPUs of prefill plus 64 GPUs of decode. A single prefill plus a single decode engine is deployable, but it is not the efficient unit.

The cache layer is the part worth understanding. Block KV cache lives on a dedicated 3FS storage cluster with 10+2 erasure coding, roughly 100 PiB of SSD, and a time to live of 3 days for agent traffic and 7 days for chatbot traffic. DeepSeek measures about 14.4 TB of logical cache per online GPU. Prefill load balancing buckets requests into four context tiers, under 16K, 16 to 64K, 64 to 256K and 256K plus, and reallocates engines as the demand mix shifts. Decode load balancing spreads requests across GPUs by both request count and total KV length.

Two smaller optimisations compound. DSpark speculative decoding drafts five tokens and uses a confidence head to decide how many to verify, which at the production operating point of about 60 concurrent sequences per rank cuts verified draft tokens from 5.00 to 3.58 and lifts throughput 5.4%. SWA bounded replay recomputes only the last 128 tokens instead of a 2,560 token decoder suffix, which DeepSeek says nearly halves total prefill computation.

The model itself is built for this deployment style. According to the DeepSeek-V4.1-Flash paper (arXiv:2609.19969), it has a 552B parameter backbone, activates 16B parameters per token during decode but only 8B during prefill, and compresses its KV cache to 890 bytes per token, about a quarter of the previous model. SWA bounded replay cuts the persistent cache footprint to roughly one eighth.

Does the US$38 per GPU-hour claim hold up?

Yes, on the public numbers. Kydo's back solve from the disclosed throughput assumptions implies about 2,400 GPUs: 60,000 fresh input tokens per second per prefill GPU, 14,000 output tokens per second per decode GPU, a 1:1 ratio, 77K average input length and a 96% cache hit rate. At DeepSeek's off-peak list prices, 155 trillion tokens a day implies about US$2.22 million of revenue a day, which is US$38.61 per GPU-hour over 2,400 GPUs. DeepSeek states US$38.

The tweet itself, posted by Kydo on 24 October 2026, had 33,940 views and 498 likes when we captured it. The original DeepSeek announcement on 20 October had 6.66 million views and 43,001 likes, so this is not a niche claim.

Here is the arithmetic, so you can check it yourself. Output tokens per day are 14,000 multiplied by 1,185 decode GPUs multiplied by 86,400 seconds, which is 1.433 trillion, or 0.92% of the total. That leaves 153.57 trillion input tokens. At a 96% hit rate, 147.4 trillion are cache hits and 6.14 trillion are misses. At US$0.003, US$0.15 and US$0.60 per million tokens off-peak, the three streams produce US$0.44 million, US$0.92 million and US$0.86 million a day respectively.

Two caveats. Both headline numbers, 155 trillion tokens and US$38 per GPU-hour, are self-reported by DeepSeek. The GPU count is a derived estimate, not a disclosure. But the fact that an independent back solve lands on the same revenue per GPU-hour as the price sheet is strong evidence that the disclosed figures are internally consistent, which is more than can be said for most infrastructure claims.

Where does the profit actually come from?

From the cache. A cache hit is billed at US$0.003 per million tokens against US$0.15 for a miss, a 50x difference, and a hit is close to free to serve because the KV cache is fetched from SSD and swapped into HBM instead of being recomputed. 96% of input tokens take that path. The expensive part of the business is the 6% that miss and the 0.9% of tokens that are actually generated.

That mix explains why revenue per token looks so low. Across the three streams the cluster earns about US$0.0143 per million tokens blended, which annualises to roughly US$0.8 billion from a single cluster. It also explains why DeepSeek can charge US$0.003 for a cached token: the marginal cost of serving it is a storage read, not a matrix multiply.

Token stream	Off-peak price per 1M	Share of tokens	Share of revenue
Input, cache hit	US$0.003	95.1%	20%
Input, cache miss	US$0.15	4.0%	41%
Output	US$0.60	0.9%	39%

Now the cost side, which DeepSeek did not disclose. Commenters on the Hacker News thread (2,347 points, 861 comments) put the all-in cost of a Blackwell GPU at roughly US$2 to US$3.50 per hour including depreciation, against cloud rental prices of US$6 to US$14. One commenter, kube-system, worked out that US$38 per card per hour is about US$2.6 million per 8 card server per year, and that a high end DGX pays for itself in just over a year at US$7 per card per hour.

Read that carefully. US$38 is revenue, not profit. The gap between revenue and cost is what funds training and the next cluster, and it is very large at the margin, but the number on its own does not tell you the margin. Utilisation, the free tier and subscription users all sit between the two.

What is the alpha in served intelligence?

Five plays. Design products so the prefix is stable and the cache hits. Sell memory as a feature, because 72 hour retention makes it cheap. Do not compete on price below the 64 plus 64 GPU floor. Compete on custody, where bans and privacy law create demand. And price outcomes, because at US$0.0143 blended per million tokens there is no money left in reselling tokens.

1. Design for the cache, not for the prompt

A 96% hit rate is not a property of DeepSeek's cluster. It is a property of how its traffic is shaped, and it comes from agent workloads that reuse the same long prefix over and over. If you build an agent that puts a stable system prompt, tool schemas and repository context at the front of every request, and appends new turns at the end, you inherit the same economics. If you inject a timestamp, a random session id or a reshuffled tool list into the prefix, you pay US$0.15 per million tokens instead of US$0.003. That is a product design decision worth 50x, and most teams make it by accident.

2. Sell memory, not tokens

The cache retains context for 3 days on agent traffic and 7 days on chatbot traffic, across a 100 PiB store. That turns memory into a feature you can put on a pricing page: an agent that already knows your repository, your CRM records and last week's decisions, without re-reading them on every turn. The infrastructure cost of that memory is a storage read. The customer value is continuity. Very few AI products charge for the second thing.

3. Do not compete on price below the floor

The smallest efficient deployment is 64 GPUs of prefill plus 64 GPUs of decode, and the two hardest components, the KV cache system and the cross replica load balancers, are not in the open source release. Anyone who tells you they will undercut DeepSeek on tokens with a handful of GPUs has not read the deployment guide. The floor is real, and it is the reason the price of tokens is set by a small number of operators rather than by the market.

4. Compete on custody

On the same day DeepInfer was published, the US federal register carried a notice excluding DeepSeek products from federal civilian executive branch systems. Demand for inference that runs under your own control went up, not down. In Australia, Privacy Act 1988 APP 8 requires reasonable steps before personal information is disclosed to an overseas recipient, which makes "where does the model run" a compliance question rather than an ideological one. Custody is the one axis where a small operator can win against a 2,400 GPU cluster.

5. Price outcomes, not tokens

At US$0.0143 of revenue per million tokens, the token itself has stopped being the unit of value. The margin has moved to the workflow around it: the review step, the audit trail, the correction that becomes a permanent check. If your business model is a markup on tokens, you are arbitraging a price that is already collapsing. If it is a price per completed task, you are selling the part that still has scarcity.

In our own rack, two DGX Sparks run DeepSeek-V4.1-Flash at about 35 tokens per second on a single stream and 62 at four concurrent requests. That is nowhere near DeepSeek scale and it is not meant to be. It is the custody tier, for work that cannot leave the building. And the agent runtime we use here, Hermes Agent, is the single largest application by token volume on OpenRouter for this exact model, at 2.45 trillion tokens according to the OpenRouter model page. The demand is real, and it is overwhelmingly agentic.

What are the counterarguments worth taking seriously?

Four. The US$38 is revenue, not profit, and DeepSeek has not published cost. Community estimates put all-in cost at US$2 to US$3.50 per GPU-hour including depreciation, which implies a very high gross margin, but utilisation, the free tier and subscription users all eat into it. Rival providers already claim better throughput on the same model. And the US federal exclusion order published the same day closes the largest single buyer market.

Start with the margin question. The Hacker News thread is unusually well informed on this, and the sceptics are right that a revenue figure is not a profit figure. One commenter, cloudbonsai, points out that the gap could come from utilisation well below 100%, from renting third party capacity at peak, or from free and subscription users, and cites a reported company wide gross margin of 44.6%. Another, HarHarVeryFunny, notes that a leaked investor figure said each DeepSeek project must break even within 10 months, which does not square with US$38 per hour, 24 hours a day, on cards that cost around US$20,000. The honest reading is that US$38 is a peak window figure, not a 24 hour average.

Then the competitive question. In the same thread, draxen argues that Together.ai and Baseten already achieve higher throughput and lower latency than DeepSeek's own service for V4.1-Flash-1017. The OpenRouter model page does list 24 or more providers for this model, with a weighted average price actually paid of US$0.01397 per million input tokens and US$0.7061 per million output tokens. So the serving advantage is not a durable moat. It is a lead measured in months.

And the direction of travel. As sandworm101 put it in the thread, when new supply comes online and demand levels off, the US$38 will shrink, and open weights destroy the monopoly that protects it. That is the whole point of open sourcing the framework: DeepSeek is commoditising the layer below the money while keeping the layer that earns it.

What should an Australian business do about it?

Treat it as a per task decision, not a platform war. Cloud for work where a frontier model and someone else's uptime matter. Local for work where data cannot leave your control, where volume makes per token billing painful, or where a hosted plan will not permit the workflow. The DeepInfer release does not change that rule. It sharpens the price of the cloud side and makes the local side more credible.

Be honest about the arithmetic. Two DGX Sparks at about US$6,950 each do not beat a US$200 per month subscription on subscription maths alone. Local wins on four conditions, not on price: several staff sharing one agent, API volume where per token billing hurts, data that cannot leave your control, and work a hosted plan will not permit. If none of those four apply, the cloud is cheaper and you should say so.

The strategic point is that served intelligence has become a utility with a visible cost curve. DeepSeek has shown that a cache shaped workload can be served for a blended US$0.0143 per million tokens of revenue, on hardware that is 0.1% of the world's Blackwell fleet by one estimate. That is the alpha: not a better model, but a better understanding of where the cost actually sits, and a product designed around it.

Frequently asked questions

What is DeepInfer?

DeepInfer is DeepSeek's production inference framework, open sourced on 20 October 2026 under Apache 2.0. It serves DeepSeek-V4.1-Flash on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs, and includes the kernels, parallelism, speculative decoding and scheduling that power DeepSeek's own service.

Can I run DeepInfer on my own GPUs?

Only on NVIDIA SM100 hardware (B200 and GB200 class) or Huawei Ascend 950. DGX Spark and consumer GPUs are not supported. You also need to build the KV cache system, the disaggregation layer and the load balancers yourself, because those are explicitly excluded from the release.

Is 155 trillion tokens a day a lot?

Yes. OpenRouter's entire marketplace was running at roughly 164 trillion tokens per week in mid October 2026, across every model and every provider. One DeepSeek cluster serving 155 trillion tokens per day is producing about a week of the whole marketplace's volume every 24 hours.

Why is the KV cache the important part?

Because it sets the price. A cache hit is billed at US$0.003 per million tokens and costs almost nothing to serve, while a miss costs US$0.15 per million and requires real compute. At a 96% hit rate, most of the traffic is nearly free to serve and most of the revenue comes from the small share that is not.

Does open sourcing DeepInfer make inference cheaper for everyone?

It makes single engine performance reproducible, which puts pressure on inference providers that were charging for inefficiency. It does not make fleets cheap, because the KV cache store, the load balancers and the orchestration stay closed, and the minimum efficient deployment is 64 GPUs of prefill plus 64 GPUs of decode.

Sources

• DeepSeek, DeepInfer repository and deployment Q&A, github.com/deepseek-ai/DeepInfer, published 20 October 2026.
• DeepSeek, DeepSeek V4.1 Flash Inference Deployment Q&A, doc/inference_deployment_qa.md, 155 trillion tokens per day and US$38 per GPU-hour at valley pricing.
• DeepSeek, Models and Pricing, api-docs.deepseek.com, captured 25 October 2026.
• DeepSeek-AI, DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression, arXiv:2609.19969.
• Kydo, X post on DeepInfer cluster economics, 24 October 2026, 33,940 views at capture.
• Hacker News discussion, DeepInfer, 2,347 points and 861 comments, 20 October 2026.
• Federal Register, Notice of Order Regarding Exclusion and Removal of DeepSeek Products, 20 October 2026, document 2026-21447.
• OpenRouter, DeepSeek V4.1 Flash model page, weighted average prices and app token volumes, captured 25 October 2026.