TensorFold: The Inference Engine That Makes Speculative Decoding Exact

Last Updated: September 27, 2026

Key takeaways

• TensorFold is an open-source MIT-licensed inference engine that serves local LLMs on Apple Silicon and NVIDIA GPUs through an OpenAI-compatible API.
• Its core innovation is exact speculative decoding: drafted output is byte-identical to serial decoding, so speed improves without changing a single sampled byte.
• According to the project's own benchmarks, TensorFold decodes 1.6x to 3.1x faster than vLLM with MTP=3 on DGX Spark across Qwen3.8 and GLM-5.3-Flash models.
• On a Mac M5 Max with 128 GB, it serves Nemotron 3.5 Lightning 30B at 188 to 206 tok/s, versus 138 tok/s for mlx_lm on the same model.
• Model support is narrow by design: hand-written kernel packages exist for only four model families, and formats like GPTQ and AWQ are refused.
• The benchmarks are self-reported and workload-dependent, so treat TensorFold as a promising young project rather than a vLLM replacement today.

What is TensorFold?

TensorFold is a local LLM inference engine published on GitHub by ashhart under an MIT license in 2026. You name a model on Hugging Face, choose a context window and sampling settings, and TensorFold downloads it, loads it with Metal or CUDA kernels written specifically for that model family, and serves an OpenAI-compatible endpoint at /v1/chat/completions. Any OpenAI client, including coding agents, SDKs and curl, can talk to it. The project had roughly 328 GitHub stars as of late September 2026, with commits landing hours before this review, so it is young but moving fast.

Why exact speculative decoding matters

Speculative decoding is the standard trick for faster LLM inference: a small draft model guesses several tokens, the big model checks them in one pass, and accepted guesses save whole decoding rounds. The catch is that conventional implementations accept drafts with a random test, so the same prompt and seed can produce different text depending on how drafting went. For coding agents and any pipeline that depends on reproducible output, that nondeterminism is a real operational risk. TensorFold's answer is to make every acceptance decision exactly match what one-token-at-a-time decoding would sample: each token is an exact draw from the top-k/top-p distribution with Gumbel noise derived only from the seed, position and token id, and each row of a multi-row verify pass receives the same arithmetic bits as a single-row step. A draft is accepted only when it equals the serial answer, so drafts change speed and nothing else. You can audit the claim yourself by sending the same request with "draft": false and comparing bytes.

How fast is TensorFold in practice?

According to the TensorFold project's own measurements, which used its bench_openai single-stream client, the speedups are substantial. On a Mac M5 Max with 128 GB, NVIDIA Nemotron 3.5 Lightning 30B-A3B in 4-bit decodes at 188 to 206 tok/s where mlx_lm serves the same model at 138 tok/s, and Qwen3.8-27B with the DFlash2 drafter reaches 189 tok/s on code against 26 tok/s without drafting. On DGX Spark hardware, TensorFold reports decode 1.6x to 3.1x faster than vLLM with MTP=3 across Qwen3.8-27B, Qwen3.8 Flash Next and GLM-5.3-Flash, with two Sparks splitting a model over their direct link reaching 103.8 tok/s on Flash Next code workloads. Two caveats apply: these numbers are self-reported rather than independently benchmarked, and speculative speed depends heavily on content, since repetitive text like file edits and code drafts far better than fresh prose.

Hardware requirements and setup

On macOS, TensorFold needs an Apple Silicon Mac and Python 3.11 or newer. According to the project documentation, the supported checkpoints need 32 GB or more of unified memory for the 30B-class models and 192 GB or more for the 113 GB Qwen3.8 Flash Next. All Apple Silicon chips from M1 to M5 support lane batching, though the fastest kernels need M5-generation tensor units. On Linux, TensorFold runs its CUDA engines inside NVIDIA's PyTorch container, and two DGX Sparks can split one model tensor-parallel over their direct link, with rank 0 serving HTTP. Version sensitivity is worth noting: on an M5, MLX 0.32.2 fails TensorFold's load-time exactness check for Nemotron, which then runs without drafts, same output but slower, and the project recommends pinning mlx 0.31.2.

Who should use TensorFold?

TensorFold fits teams running local AI coding agents where output reproducibility matters as much as speed. The prefix-cache design is explicitly built for agent traffic: follow-up turns prefill only their new suffix, system blocks are cached to disk once, and conversations survive server restarts. At Flowtivity we see growing businesses increasingly want local or sovereign inference for privacy-sensitive workflows, and an engine that guarantees a coding agent sees identical output for identical input removes a whole class of debugging pain. For broad production serving across many architectures, vLLM and llama.cpp remain the safer choice today; for the specific Mac and DGX Spark setups TensorFold targets, it is worth a serious look.

What are the limitations?

TensorFold's narrowness is both its strength and its risk. It supports four model families with hand-written kernel packages, and refuses unsupported weight formats such as GPTQ, AWQ and NVFP4 before downloading anything. Benchmarks are self-reported with no independent verification yet. The project depends on specific MLX versions for full drafting speed on some chips, and a known edge case means Flash Next and Nemotron prompt caches can differ in their last bits depending on which prefix was already cached. With a small community around a single primary maintainer, adoption should be weighed accordingly.

TensorFold is on GitHub under the MIT license. Written by AJ Awan: former EY management consultant, TOGAF certified enterprise architect, and founder of Flowtivity.