LocalRouter on One DGX Spark: A Full-Duplex Local AI Assistant, Measured

Last Updated: 23 October 2026

Key takeaways

• LocalRouter puts a full duplex voice assistant, a 27B chat model, a speech model, an image model and a video model on a single NVIDIA DGX Spark, with nothing leaving the box.
• The tweet claims about 0.5 seconds from the end of speech to the first word. The project's own instrumented runs, dated 21 and 22 October 2026, report a median of 961 milliseconds, down from 2,172 milliseconds before streaming was added.
• The genuinely new part is not the demo, it is the memory scheduler: a 14 MiB Zig daemon rationing 121.7 GiB of unified memory across five models, one job at a time, with bit exact verification gates and qualification markers bound to the host.
• The router is Apache 2.0, but the payload is not. The image model is licensed for non-commercial research only, the video model excludes the US, EU, UK and South Korea, and the speech model caps commercial use at US$500,000 annual revenue with no hosted service.
• It works, it installs in about half an hour, and it is a single operator tool. It is not yet a product you can sell.

On 22 October 2026, JJ Eaton posted a short video of something he called LocalRouter and wrote that it "actually feels like Jarvis in a box". At the time of writing the post had 20,493 views and 314 bookmarks, which for a local inference demo is a lot. The interesting sentence was not the Jarvis line. It was this one: "about 0.5s after I stop talking, it starts with its first word. Interrupt it anytime."

That is the number every local voice assistant has failed to hit. So I read the repository, all of it: the README, the architecture notes, the voice design document, the chat benchmarks, the bring-up log and the qualification results. Everything below comes from the project's own instrumented runs on a machine it calls DGX-01, dated 21 and 22 October 2026, plus the upstream repositories it builds on. Where the tweet and the code disagree, I give you both and tell you which one I trust.

What LocalRouter actually is

LocalRouter is not an inference engine. It is a router, a scheduler and a voice loop wrapped around one. The engine underneath is TensorFold, an Apache 2.0 inference engine for Metal and CUDA by ashhart, sitting at 1,454 stars when I looked. LocalRouter itself is a Zig project by jayleaton, created on 8 October 2026, Apache 2.0, 59 stars and 5 forks at the time of writing, described in its own words as "fast and opinionated local AI routing: one MCP server that loads local image and video models on demand on your own GPU box".

Five models share the box:

• Qwen3.8-27B, a 27.8B dense multimodal model with 64 layers, a hybrid attention stack that mixes Gated DeltaNet linear layers with full attention, 262K context and an Apache 2.0 licence. It runs here at MLX 4-bit with a DFlash2 speculative drafter.
• Fish Audio Drama 3 for text to speech, streamed rather than generated whole. Its backbone is a 40-layer Gated DeltaNet and attention hybrid, with a 13-layer temporal codebook tower writing ten codec codebooks per frame through a modified DAC codec at 44.1 kHz and roughly 21.5 frames per second.
• Whisper large-v3-turbo for speech to text, with a Silero voice activity gate.
• Qwen-Image 2.1 Turbo for images, 1024 by 1024 in 6.3 seconds warm.
• MiniMax H3 for short video with native audio, 768 by 448.

The design is deliberately thin. One MCP server by default, plus a simple web UI at /voice that is a single HTML file with no build step and no libraries, embedded in the binary. One job at a time per lane. No orchestration layer, no agent framework, no plugin marketplace. The whole thing is a daemon that decides which model gets the GPU right now.

The 0.5 second claim, against the repo's own numbers

Here is the uncomfortable part. The repository documents its own latency in a table titled "First spoken word under 1 s". Two commits, before and after the streaming work:

• Before (4011b7e0): 2,172 ms median from the end of the user's speech to the first non-silent reply audio, p90 2,389 ms, and 31 of 33 checks passed.
• After (369808d8): 961 ms median, p90 981 ms, and 33 of 33 checks passed.

That is a 2.3x improvement in one round of work, and it is a real one. It is also not half a second. So where does the difference come from? The project publishes the stage medians, in milliseconds, which is unusually generous of it:

• Voice activity detection ends the turn at 349 ms. The draft's sentence end closes the turn after 300 ms of quiet, down from 500 ms. This is the single biggest line item and it is a setting, not a computation.
• Speech to text takes 252 ms, but it starts 263 ms before the turn ends. In other words the transcript is already finished when the turn closes. This is the trick that makes the loop feel fast: it transcribes the trailing silence as if it were speech.
• The chat model's first token arrives 184 ms after the turn ends, thanks to prefix caching and a warm model.
• 183 ms to reach the first text the speech model can start on, because the lead hold releases one token past the first clause.
• 131 ms for Drama 3 to emit its first audio samples, plus 2 ms of delivery and 9 ms of leading silence.

Add those up and you land at roughly 860 ms, against a measured median of 961 ms. The difference is real-world jitter across six turns. Note what is not in the table: no cloud round trip, no upload, no queue, no rate limit. The entire loop is a local socket and a GPU.

So the tweet's 0.5 seconds is either a newer build than the documents describe, or it counts from a different moment than "I stop talking". The most likely reading is the second one: if you start the clock when the voice activity detector commits rather than when the last voiced sample ends, and you shorten the quiet window to its 300 ms floor, you can plausibly get near 0.6 seconds. What the repository can prove today is 961 ms, and it says so in its own table. I would rather publish the number the code produces than the number the tweet remembers.

The part worth copying: a scheduler that refuses to guess

Forget the demo for a moment. The engineering that makes this work is memory management, and it is the part I would steal.

A DGX Spark's GB10 Grace Blackwell gives you 121.7 GiB of coherent unified memory in this configuration, on a 256-bit interface at 273 GB/s, with up to 1 PFLOP of FP4 tensor performance. That is one pool shared by CPU and GPU, which is why five models can live on it at all, and also why a careless allocation takes the whole box down.

LocalRouter's budget, from the bring-up log:

• Chat, Qwen3.8-27B at 4-bit with the drafter: 24 GiB, 19.2 GiB resident at rest and 21.1 GiB at a 30k token prompt.
• Drama 3: 42 GiB, with a peak of 39.84 GiB during qualification.
• Whisper: 3.6 GiB.
• Qwen-Image 2.1 Turbo: 35.2 GiB.
• Total resident: about 105 GiB of 121.7, leaving 13.7 GiB available.

Three rules keep that inside the box. First, image and video never sit beside each other while the voice loop is up, because each wants about 39 GiB. Second, a video job unloads only the image model; chat, Drama 3 and Whisper are never evicted for generation. Third, idle models are dropped on a timer, 60 seconds for image and 30 for video, and because each model is its own process, unloading returns every byte rather than leaving a fragmented heap behind.

The daemon itself sits at 14 MiB of RSS with no GPU memory at rest. That is the whole scheduler: fourteen megabytes of Zig deciding what gets to exist.

Two design decisions deserve more attention than they are getting.

Speculative decoding is verified, not hoped for. TensorFold's DFlash2 drafter accepts a draft token only if it is exactly the token the same engine would produce with draft: false. That is not an approximation or a tolerance; it is bit exactness. You get the speed of speculation without a quality question, because a rejected draft simply falls back to the real distribution. In this configuration it is worth roughly 3x on decode: 38.6 to 41.2 tokens per second on chat prose and 74.7 to 77.8 on code, against 13.3 tokens per second without the drafter.

Qualification is bound to the host. Every model ships with a qualification marker that says it has been measured on this machine and passed. Move the install to a different box and the model reports "qualified": false and refuses to serve with a 503 until qualify.sh passes there. I have not seen that pattern in an open source inference stack before, and it is the right instinct: it refuses to pretend a number is portable when it is not.

What it feels like when it works

The chat lane is genuinely quick for a 27B model on a desktop-class box. Live service on 22 October 2026 measured 34 to 39 tokens per second on prose, 70 to 87 on code, with a time to first token of 0.14 to 0.16 seconds once the prefix cache is warm. That is the difference between a local model you tolerate and one you use.

Images take 6.3 seconds warm at 1024 by 1024. A spoken request, "generate a picture of a dog", is done 8.1 to 10.2 seconds after you stop talking, with the confirmation spoken 8.8 to 11.4 seconds after. Video is the slow lane: a one second clip takes about 24 seconds and a five second clip 54 seconds warm at 768 by 448 with audio. That is not interactive, and the project does not pretend it is.

Barge-in is the detail that sells the whole thing. When you start speaking over the assistant, the old answer is cancelled 41 ms after the speech detector fires, with no audio from the interrupted reply afterwards. There is no model call in that path, no GPU work, just a flag on the reader thread. Interruption is a scheduling problem, not an intelligence problem, and the project treats it that way.

Where it breaks

The repository is unusually honest about its own failures, which makes this section easy to write and hard to argue with.

Contention ruins the feel exactly when you want it most. While Drama 3 is synthesising speech, chat drops to 14 to 19 tokens per second and Drama 3's real time factor rises from 0.66 to 1.47 or 1.52, meaning it is generating slower than playback. TensorFold exposes no priority between processes, so the scheduler cannot arbitrate. Talking to it while it reads you a long answer is the one thing this box does badly.

There is no authentication at all. The project's own warning is blunt: loopback only, or a tailnet with tailscale serve, and "never all on an untrusted network". A voice assistant with no auth on an open port is not a product, it is an incident waiting for a port scan.

Some numerical gates fail and were accepted anyway. Drama 3's all-row cosine similarity is 0.99860 against a 0.999 bar, and its relative L2 error is 0.0528 against a 0.05 bar. The bring-up accepted this explicitly with --accept-native-numerics. Qwen-Image 2.1 Turbo's bf16 twin fails the project's own cos >= 0.9999 gate at sigma 0.2, landing at 0.99980, even though it sits closer to an fp32 reference than the diffusers implementation does. These are small deviations and the reasoning is documented, but they are deviations. If you need bit-exact audio reproduction, this is not it.

Generation is one-way. No image edits and no image to video, because, in the project's words, "edit inputs are refused by the current engines". You can make a picture, you cannot change one.

The install is not a laptop project. Downloads and builds take 31.5 minutes and pull 175.7 GB, chat qualification adds 113 seconds and Drama 3 qualification another 20.5 minutes. Budget about 250 GiB of free disk. This is a dedicated machine, not a side quest.

It says early preview and means it. The README carries a warning block: "LocalRouter is new and under active development: expect bugs and breaking changes." The author's own question at the end of the tweet, "what about TP=2 or Mac Studio versions", has no answer in the repository. No other profile is tested yet.

The licence trap: the router is open, the models are not

This is the part I would put in front of any client before they build on it, because the stack looks open and mostly is not.

• LocalRouter: Apache 2.0. Take it, fork it, sell a service around it.
• Qwen3.8-27B: Apache 2.0. Commercially usable.
• Qwen-Image 2.1 and 2.1 Turbo: Qwen Research License, non-commercial research or evaluation only. The image lane cannot ship in a paid product as configured.
• MiniMax H3: MiniMax community licence, which excludes the United States, the European Union, the United Kingdom and South Korea, plus downstream use conditions. Australia is not on that exclusion list, which is useful locally, but "not excluded" is not the same as "licensed for you".
• Fish Audio Drama 3: Fish Audio Community License v1.0. Commercial use only below US$500,000 annual revenue, and no hosted service, meaning letting third parties generate speech through an API, without a separate licence from Fish Audio.

That last clause explains the architecture. The assistant listens on loopback or a tailnet because the speech model's licence does not permit offering synthesis as a service. It is a single operator tool by design and by contract, not by accident.

The practical rule: you can build a private assistant for your own team on this stack today. You cannot turn it into a product that other people use without replacing the speech model, replacing or licensing the image model, and checking your territory for the video model.

What an Australian business should take from it

Three things, in order of usefulness.

One: the architecture is the lesson, not the demo. Keep the small, always-on models resident. Give the big generation models a timer and a lane. Never let a 39 GiB image model evict the 24 GiB chat model that makes the assistant feel alive. Verify speculative decoding rather than trusting it. Bind qualification to the machine. Any team building a private AI station should copy those four rules whether or not they copy the code.

Two: the hardware maths has changed. A DGX Spark 128GB launched at US$3,999, was raised to US$4,699, and has been listed at US$6,950 by some retailers in early October 2026; a new 64GB tier starts at US$4,999 and began shipping on 23 October 2026 through Acer, ASUS, Dell, Gigabyte, HP and MSI. For a single operator, one box at that price replaces a cloud voice stack plus a per-seat subscription, and no audio, transcript or image ever leaves the premises. For an Australian practice with client confidentiality obligations, that is the argument, and it is a stronger one than the benchmark numbers.

Three: be precise about what you would deploy. I would put this in front of a team as an internal research assistant, an intake drafting station, or a private "talk to our documents" terminal, behind a tailnet, with one named operator. I would not put it on a customer-facing phone line, and I would not sell image generation from it, until authentication exists and the image and speech licences are resolved. The gap between "works on my desk" and "safe to hand to a client" is exactly the gap the project documents in its own warnings.

JJ Eaton built something worth studying, and the honesty of the documentation is the best part of it. A demo that publishes its failed gates, its contention numbers and its licence limits is worth more than a demo that publishes a launch video. The 0.5 second claim will get the views. The 961 millisecond table is what you should read.

Frequently asked questions

How fast is LocalRouter's voice loop, really?

The project's own measured median is 961 milliseconds from the end of speech to the first non-silent reply audio, with a p90 of 981 ms, down from 2,172 ms before streaming was added. The announcement post claims about 0.5 seconds. The difference is most likely a shorter silence window and a different start point for the clock.

Can LocalRouter run on a Mac Studio or two DGX Sparks?

Not yet. The repository ships one tested profile, a single GB10 machine with 121.7 GiB of unified memory. The author asks about tensor parallel across two Sparks and a Mac Studio version in the announcement, but the project states that no other profile is tested. Qualification markers are also bound to the host, so a copy on another machine refuses to serve until it is qualified there.

Is LocalRouter open source?

The router is Apache 2.0 and the chat model is Apache 2.0, so the code and the language model are genuinely open. The rest of the stack is not: the image model is research-only, the video model excludes several territories, and the speech model caps commercial use at US$500,000 revenue and forbids hosted services without a separate licence.

Do I need a DGX Spark to run it?

For the tested configuration, yes. It targets a GB10 Grace Blackwell box with 121.7 GiB of coherent unified memory, CUDA 13 and driver 580.178.04 on Ubuntu 24.04 for aarch64. The install pulls 175.7 GB and needs about 250 GiB of free disk, plus roughly an hour of build and qualification time.

Can I use it commercially?

For internal use by your own team, the stack is workable, provided you stay inside the speech model's revenue cap and do not offer synthesis as a service. For a product you sell to others, you would need to replace the image model, licence or replace the speech model, and check the video model's territorial conditions.

Sources

• JJ Eaton, LocalRouter announcement, X, 22 October 2026: x.com/jayleaton/status/2113261675654881637
• LocalRouter repository, README and docs/dev/VOICE.md, docs/dev/CHAT.md, docs/dev/RESULTS.md, docs/dev/JARVIS-BRINGUP.md: github.com/jayleaton/localrouter
• TensorFold inference engine: github.com/ashhart/TensorFold
• NVIDIA DGX Spark specifications: nvidia.com/products/workstations/dgx-spark
• VideoCardz, DGX Spark pricing changes, 2 October 2026
• HWBusters and SantageAI, DGX Spark 128GB retail pricing, 2 to 3 October 2026
• Qwen3.8-27B model card, released 14 August 2026
• Fish Audio Drama 3 model card and Fish Audio Community License v1.0
• MiniMax H3 open-weight release and community licence