DeepSeek V4.1 Flash on dual DGX Spark: the best local setup for an AI agent

Last Updated: 21 October 2026

Key takeaways

• We run Nous Research’s Hermes Agent on DeepSeek V4.1 Flash across two NVIDIA DGX Sparks, and it is now the default model for every session we open.
• On our endpoint the model sustains about 35 tokens per second on a single stream, and 61.7 tokens per second with four requests in flight.
• A 131,072-token context costs roughly 111 MB of KV cache, because DeepSeek V4.1 Flash stores 890 bytes per token.
• The pairing works because the model activates 8B parameters during prefill and 16B during decode, which matches the prefill-heavy traffic an agent produces.
• Two Sparks pool 256 GB of unified memory and move about 111 GB/s over the link, enough to hold a 552B-parameter model with room for context.
• It is not cheaper than the cloud API on raw token price. It wins on data control, rate limits and burst capacity, and those are the reasons we keep it.

If you want to run a capable AI agent on hardware you own, the best pairing we have found is DeepSeek V4.1 Flash on two NVIDIA DGX Sparks. We moved our Hermes Agent onto exactly that stack, and after a day of measured production traffic it is now the default. The model is a 552B-parameter open-weight mixture of experts that activates only 8B parameters during prefill and 16B during decode, which is why a pair of 128 GB boxes can serve it at all. In our testing it generated about 35 tokens per second on a single stream and 61.7 tokens per second across four concurrent requests, while a full 131k-token context consumed roughly 111 MB of KV cache. That combination of small active compute, tiny KV footprint and strong agentic benchmark scores is what makes this the model to put on two Sparks rather than something bigger or smaller.

What is the stack we actually run?

The stack is two DGX Sparks linked over ConnectX-7, serving DeepSeek-V4.1-Flash-1017 through an OpenAI-compatible endpoint, with Hermes Agent as the client. Hermes is the open-source agent from Nous Research: MIT licensed, with persistent memory, self-written skills, more than 60 tools and a messaging gateway that puts it on Telegram, Discord, Slack or WhatsApp. Because Hermes is provider-agnostic, pointing it at a local endpoint is a configuration change, not a fork. Our provider entry sets the base URL, the model name, a 131,072-token context cap and a 32,768-token output cap. The agent, its cron jobs, its subagents and its Telegram gateway all run against that one endpoint, which means every token we spend stays inside the two boxes.

• Hardware: two DGX Spark units, GB10 Grace Blackwell superchips, 128 GB of unified LPDDR5X each, 273 GB/s per unit, linked at 200 Gbps over ConnectX-7.
• Serving: tensor parallelism with TP=2, so each layer is split across both nodes and the pair behaves like one larger accelerator.
• Model: DeepSeek V4.1 Flash, 552B backbone parameters, 8B active in prefill, 16B active in decode, multimodal, MIT licence.
• Agent: Hermes Agent, MIT licensed, with memory, skills, cron scheduling, subagent delegation and a messaging gateway.
• Context: 131,072 tokens configured, well below the model’s 1M-token ceiling, to keep the KV cache and prefill latency predictable.

Why is DeepSeek V4.1 Flash a better fit for an agent than a chat model?

An agent is not a chatbot with tools bolted on. It reads long system prompts, tool schemas and conversation history on every turn, then emits a short tool call and repeats. That traffic is overwhelmingly prefill, not decode, and DeepSeek V4.1 Flash is built around exactly that asymmetry. According to DeepSeek-AI’s model card, the model uses a Causal Encoder-Decoder architecture in which a 20-layer causal encoder feeds a 20-layer decoder, and the decoder’s KV cache is projected from the encoder’s final hidden states instead of being rebuilt in every decoder layer. The result is 8B active parameters per token during prefill and 16B during decode, from a 552B-parameter backbone.

The second half of the fit is memory. DeepSeek’s card states that Compressed Sparse Attention 2, combined with FP4 main KV caching, brings the global KV cache to 890 bytes per token, about a quarter of DeepSeek V4 Flash and 437 times smaller than DeepSeek V1. At 131,072 tokens that is roughly 111 MB, which is small enough that long agent sessions do not crowd the weights out of memory. The model also ships with DSpark speculative decoding, a draft-and-verify scheme that raises effective throughput without changing the output distribution.

Quality matters too, because an agent that cannot finish a task is expensive regardless of tokens per second. On DeepSeek’s own comparison table, V4.1 Flash scores 90.6 on Terminal-Bench 2.1, ahead of Claude Opus 5.0 at 89.1, GPT-5.6 Sol at 88.8 and GLM-5.3 at 88.2. It scores 74.2 on DeepSWE v1.1 against Opus 5.0’s 74.0, and it leads the table on AutomationBench at 54.8, Agent’s Last Exam at 31.8 and CyberGym at 88.1. Those are vendor-run numbers, so treat them as directional, but the direction is consistent: this is an agentic model first and a chat model second.

“The decoder’s global KV cache is projected from the final encoder hidden states rather than derived from each decoder layer’s own hidden states,” DeepSeek-AI explains in the model card. That single design choice is the reason a 552B model can serve long agent contexts on two deskside boxes.

Why does the model need two DGX Sparks?

Because 552B parameters do not fit in 128 GB. According to NVIDIA’s DGX Spark product page, each unit pairs the GB10 superchip with 128 GB of unified LPDDR5X memory at 273 GB/s, about 1 petaFLOP of FP4 compute, a ConnectX-7 200 Gbps NIC and a 240 W power envelope, and up to four units can be linked. Two units give you 256 GB of pooled memory. Our serving stack reports about 113 GB of device memory in use, which is roughly 88% of one node’s capacity and leaves headroom for KV cache, activations and the operating system.

Tensor parallelism is what turns the pair into one machine. Every layer is cut in half, each node computes its half, and the two exchange partial results over the link before the next layer runs. MindStudio’s October 2026 clustering guide measured the 200 Gbps RoCE link at about 111 GB/s in practice, roughly half its rated speed but still fast enough that the interconnect does not dominate decode. The same guide measured DeepSeek at about 38 tokens per second on a dual-Spark cluster, with a 32,000-token prompt processed in about 17 seconds and a 128,000-token prompt in about 72 seconds. Our own numbers land in the same range, which is a useful independent cross-check.

There is a hardware argument beyond capacity. Decode is memory-bandwidth bound, and each Spark has 273 GB/s, far below the 1.2 TB/s of a high-end Mac Studio. Prefill is compute bound, and the Sparks win there. For an agent, which spends most of its tokens on prefill, that trade is the right one. It is also why a sparse mixture of experts is the correct model class for this box: only 8B or 16B parameters are touched per token, so the 273 GB/s ceiling hurts far less than it would with a dense model of the same total size.

What did we measure on our own endpoint?

We benchmarked the endpoint directly rather than trusting a spec sheet. Over three single-stream runs of 450 output tokens each, the mean time to first token was 0.88 seconds and the mean end-to-end rate was 35.1 tokens per second, with decode alone at 37.7 tokens per second. With four requests in flight, aggregate throughput rose to 61.7 tokens per second while individual streams fell to between 15.5 and 24.7 tokens per second. A 10,992-token prompt was prefilled in 6.42 seconds, about 1,713 tokens per second of prefill. The endpoint was also serving the live agent during the test, so these figures are conservative.

The most useful number is not any single rate. It is the ratio. Across the 137 requests the endpoint had served when we sampled its metrics, it processed 10,035,900 prompt tokens against 59,939 generated tokens. That is about 167 input tokens for every output token. Agent workloads are prefill machines, and any model you put on two Sparks should be judged on prefill cost, KV footprint and task completion, not on headline decode speed alone. V4.1 Flash’s 8B prefill activation and 890-byte KV cache are aimed squarely at that profile.

How does a Hermes turn run on two Sparks?

A single Hermes turn is a loop, not a request. The agent assembles a prompt from its system instructions, memory, loaded skills, tool schemas and conversation history, sends it to the endpoint, receives either a tool call or a final answer, executes any tools, appends the results and sends the larger prompt again. Each iteration is a fresh prefill of a growing context, followed by a short decode. On our stack, the prefill is split across both Sparks, the KV cache is stored in FP4 at 890 bytes per token, decode uses DSpark speculative decoding, and the loop repeats until the agent stops calling tools.

That shape explains the numbers. Time to first token stays under a second on short prompts and rises with context, because prefill scales with prompt length. Aggregate throughput scales with concurrency, because four streams share the same weights and fill the batch. Decode per stream falls as concurrency rises, because bandwidth is finite. If you run cron jobs, subagents and a live chat at once, as we do, the four-stream figure of 61.7 tokens per second is closer to your real experience than the single-stream 35.

Where does this setup fall short?

It is not a frontier-killer on every task, and it is not free. On DeepSeek’s own table, Claude Opus 5.0 still leads Terminal-Bench 3.0 at 43.3 against 30.0, Terminal-Bench 4.0 at 51.8 against 31.2, and NL2Repo-Bench at 75.3 against 64.0. Decode is bandwidth bound, so single-stream speed sits near 35 tokens per second rather than the hundreds a hosted provider can report. The 131k context cap we configured is a deliberate trade, not a limit of the model, and lifting it toward 1M would raise prefill latency and KV pressure. Two boxes also mean two power supplies, two fans and a 480 W combined envelope, plus the operational work of keeping a cluster healthy.

The honest framing is that this is the best model for this hardware, not the best model in the world. If your work is dominated by the hardest long-horizon coding and terminal tasks, a frontier cloud model will still finish more of them. If your work is an always-on agent that reads your files, runs your tools, remembers your context and must not ship that context to a third party, the local pair is the better trade.

Is a local agent cheaper than the cloud API?

Not on raw token price, and it is worth saying so plainly. OpenRouter lists DeepSeek V4.1 Flash at roughly $0.05 to $0.10 per million input tokens and $0.50 to $0.72 per million output tokens, with cache reads near $0.008 per million. Two Sparks cost about $9,398 at the September 2026 price of $4,699 each, according to OpenClaw DC’s price check. At our measured 167:1 input-to-output ratio, you would need hundreds of millions of output tokens before the hardware paid for itself against list prices.

So the case for local is not the token bill. It is that a 24/7 agent with cron jobs, subagents and a chat gateway produces bursty, unpredictable load that hits rate limits and per-token anxiety, and local capacity has zero marginal cost once bought. It is that client documents, code and conversation history never leave the building. And it is that time to first token is stable and local, with no provider queue between your agent and its next tool call. If none of those matter to you, the cloud API is the cheaper and simpler answer.

How do you set it up?

Four steps get you from two boxes to a working agent endpoint. Link the Sparks over ConnectX-7 with RoCE, run an inference server in tensor-parallel mode with TP=2, expose it as an OpenAI-compatible endpoint, then point Hermes at it with a provider entry. The only non-obvious part is the context cap: set it deliberately, because the model supports 1M tokens but your KV cache, prefill latency and memory headroom all scale with it.

1. Link the nodes. Connect the two Sparks with the 200 Gbps ConnectX-7 link and configure RoCE so each node can write into the other’s memory directly. MindStudio measured about 111 GB/s in practice.
2. Serve with tensor parallelism. Start the inference server with TP=2 so every layer is split across both nodes. Confirm the model loads at roughly 113 GB per node, leaving room for KV cache.
3. Expose an OpenAI-compatible API. Bind the server to your private network and verify it with a models request and a short chat completion before involving the agent.
4. Configure Hermes. Add a provider entry with the base URL, model name, API key, context length and output cap. Set it as the default model, then restart the gateway so new sessions pick it up.
5. Benchmark before you trust it. Measure time to first token, single-stream decode, aggregate throughput at your real concurrency, and prefill on a long prompt. Keep the numbers so you can tell when a driver or server update changes them.

What is the verdict?

DeepSeek V4.1 Flash on two DGX Sparks is the best local agent setup we have run, and it is now the default for our Hermes Agent. The model’s 8B prefill and 16B decode activation, its 890-byte KV cache, its agentic benchmark profile and its MIT licence line up with the hardware’s 256 GB of pooled memory and its compute-heavy, bandwidth-light character. Measured at 35 tokens per second on a single stream, 61.7 across four, and 1,713 tokens per second of prefill, it is fast enough for real agent loops and cheap enough to leave running all day. If you are building an on-prem agent and want the shortest path to something you can trust, start here.

Frequently asked questions

Can a single DGX Spark run DeepSeek V4.1 Flash?

No. A 552B-parameter model does not fit in 128 GB of unified memory alongside its KV cache and activations. You need at least two linked Sparks for tensor-parallel serving, and NVIDIA supports linking up to four units if you want more headroom or a larger model.

How fast is DeepSeek V4.1 Flash on dual DGX Spark?

On our endpoint it sustained about 35 tokens per second on a single stream, 61.7 tokens per second across four concurrent streams, and roughly 1,713 tokens per second of prefill. Independent testing by MindStudio measured about 38 tokens per second on a dual-Spark cluster, which matches our results closely.

Does Hermes Agent work with a local OpenAI-compatible endpoint?

Yes. Hermes Agent is provider-agnostic and speaks the OpenAI chat completions format, so a local server looks the same to it as a hosted provider. You add a provider entry with the base URL, model name and key, set it as the default, and every session, cron job and subagent uses it.

Is DeepSeek V4.1 Flash better than GLM-5.3 for agent work?

On most agentic benchmarks, yes. DeepSeek’s comparison table puts V4.1 Flash ahead of GLM-5.3 on Terminal-Bench 2.1 (90.6 against 88.2) and DeepSWE v1.1 (74.2 against 66.9), though GLM-5.3 still leads on Terminal-Bench 4.0 (37.9 against 31.2). The stronger reason to pick V4.1 Flash on two Sparks is its much smaller KV cache, which keeps long agent sessions affordable.

What context length should you configure?

We run 131,072 tokens, a deliberate cap below the model’s 1M-token ceiling. At 890 bytes per token that costs about 111 MB of KV cache, which leaves most of the 128 GB per node for weights and activations. Raising the cap increases prefill latency and memory pressure, so set it to the largest context your agent actually needs.