On this page
Last Updated: September 26, 2026
Short answer: the Recurrent Looped Transformer (RLT) is a real and interesting architecture, the "infinite reasoning depth" phrasing from the viral tweet is structurally true but was never a capability claim, and the first independent controlled evaluation found that per-token recurrence does not pay for language modeling at the scales tested. Where it did pay, in Zhang's own experiments, is length generalization on small algorithmic state-tracking tasks. Both findings matter, and this post separates them.
Key Takeaways
- RLT feeds each token's final decoder output into the next token's input through a gated merge, so the recurrent path grows with sequence length at a fixed per-token cost. That is the entire "infinite depth" claim.
- Zhang's experiments show parity at 100% accuracy at 256 bits, trained only up to 40 bits, against about 50% for a matched Transformer, and swaps-based tracking at 55.7% versus 0.85% at twice the training length.
- Empero AI trained RLT on 500M tokens with an exact alpha-equals-zero ablation: it tied the Transformer at 50M parameters, was the worst of three models at 140M, and needed about 20 times the GPU-hours.
- Most of RLT's compute penalty comes from the sequential recurrent decoder, which cannot batch tokens the way standard training does.
- Today's practical takeaway: RLT is a research direction for explicit state-tracking workloads, not a drop-in replacement for Transformers, and there is no checkpoint or API to run.
What is the Recurrent Looped Transformer?
A standard Transformer language model reads every token through the same fixed stack of layers, and nothing of that token's computation is carried into the next token's earlier layers. RLT closes that loop. According to Zhang's technical report, a causal encoder first produces a representation for each token and projects them into a global key-value memory. The decoder then processes one token at a time: a gated merge combines the current token's encoder representation with the previous token's final decoder output, then decoder blocks attend over recent decoder activations with sliding-window attention and over the encoder memory.
The two loops are different things, and it is worth keeping them separate. In looped or recurrent-depth models like Huginn-3.5B, and reportedly OpenAI's Astra, one block-stack is applied several times within a single token's forward pass, then resets for the next token. In RLT the depth per token is ordinary, but the final state persists across every token, including across the prompt-response boundary. Nothing resets when the model switches from reading your prompt to writing its answer. After t tokens the state has traversed t multiplied by decoder depth in blocks, which is the sense in which depth is "unbounded."
Why did the RLT tweet blow up?
Because the framing outran the paper. Zhang announced RLT on September 12, 2026 with "We are at the dawn of Superintelligence" and "Transformers with Infinite Reasoning depth." According to the tweet's public metrics, it drew 1.36 million views and 7,300 likes in a day, and the repository passed 900 stars within two weeks. The tweet is from the paper's own first author, a Princeton PhD researcher with prior stints at NVIDIA and ByteDance Seed, so this was not third-party exaggeration. It was the author's own maximalist packaging of a cautious technical report.
The paper itself is more restrained. Its abstract claims exactly three things: recurrence extends the computation path with sequence length at fixed per-token cost, several RLT configurations learn parity earlier and generalize it to 256 bits, and gains depend on the task and depth split. The word "superintelligence" does not appear anywhere in the 48-page report. The gap between those two documents is the whole story.
What did Zhang's own experiments show?
The September 17 revision added real results: five eight-layer encoder-decoder splits (4+4 through 8+0) against a decoder-only Transformer, on six algorithmic tasks, three seeds each, all in CPU FP32. These are small models, roughly 26 to 29 million parameters against a 25.3 million parameter baseline, trained for 2,000 steps.
The headline result is parity. According to the report's validation table, the 5+3 and 7+1 splits reach 100% accuracy at 256-bit strings in all three seeds, trained only on strings up to 40 bits. The Transformer baseline reaches 94.84% in-distribution and 50.07% at 256 bits, which is chance. Learning speed diverges too: at step 500 the 6+2 split is at 99.44% parity while the Transformer sits at 48.48%.
On swaps-based permutation tracking at length 512, double the training length, the 4+4 split reaches 55.70% final-state accuracy against 0.85% for the Transformer. The paper reports this honestly with a huge standard deviation: 55.70 plus or minus 25.78, meaning one seed carried the mean. Independent community experiments from a roughly 79K-parameter community implementation support the pattern on parity but show five-state tracking falling to chance at four times the training length.
What did the independent evaluation find?
On September 14, Empero AI, an independent lab in Germany, published "Does Recurrence Pay?", the experiment the original report was missing. They implemented RLT exactly as specified, wrote exact CUDA-graph and Triton kernels for the recurrence, and pretrained it on 500M tokens of the ULTRAX-PREVIEW mixture at roughly 50M and 140M parameters against parameter-matched and block-matched Transformers, plus the decisive control: an identical RLT with the feedback scale set to zero, all on the same tokens in the same order.
The result is not ambiguous. At 50M parameters, RLT ties the parameter-matched Transformer at 3.8845 versus 3.8841 nats of validation loss. At 140M, RLT is the worst of the three models at 3.688, against 3.666 without feedback and 3.642 for the Transformer, so the feedback connection actively hurt at the larger scale. The deficit grows toward the end of documents, from about 0.006 nats in the first position bucket to 0.025 nats past position 256, which is exactly where a carried state should have been helping. Downstream, RLT scored 0.147 on LAMBADA accuracy versus 0.163 and 0.166 for the other two, and showed no context-extrapolation advantage at 1024, 2048 or 4096 tokens.
How slow is per-token recurrence to train?
Very. This is the part most takes skipped. A standard Transformer trains on all tokens of a sequence in parallel. RLT's decoder must update sequentially at every position, in both training and prompt prefill, and batching across sequences does not create parallelism within a sequence. Empero's accelerated kernels cut RLT decoder time substantially, and the model still trained at 8.4K tokens per second per RTX 5090 against 124K for the Transformer, roughly 15 times slower. Processing 500M tokens took about 22 GPU-hours for RLT versus 1.14 for the Transformer, about 19 to 20 times the compute for a worse model, and scaling from four to eight GPUs yielded only 1.38 times the aggregate throughput.
RLT versus a standard Transformer
| Dimension | Recurrent Looped Transformer | Standard Transformer |
|---|---|---|
| State carrying | Final decoder output and layerwise sliding-window KV carried across every token, including the prompt-response boundary | Forward pass per token, no persistent decoder state beyond the attention KV cache |
| Parity at 256 bits, 8 layers | 100% (5+3 and 7+1 splits, all seeds), per Zhang et al. | 50.07%, chance level, per Zhang et al. |
| Language modeling, 140M params | 3.688 nats, worst of three models, per Empero AI | 3.642 nats, best of the three |
| Training throughput on RTX 5090 | 8.4K tokens/s with exact custom kernels, per Empero AI | 124K tokens/s |
| GPU-hours per 500M tokens | About 22, per Empero AI | About 1.14 |
| Best known use today | Algorithmic state-tracking with long test horizons, RL replay consistency research | Everything currently in production |
Where does RLT actually go from here?
Three honest readings. First, the architecture idea lives. Per-token recurrence with a persisted state is a legitimate and old research thread, and the parity result shows the inductive bias can beat a matched baseline on tasks that are provably hard for shallow parallel computation. If a lab later shows state-of-the-art results on explicit state-tracking workloads, RLT's recurrence will be part of that story.
Second, the RL co-design idea may outlive the architecture. RLT's proposal that pretraining, SFT, sampling and current-policy RL replay share one exact state transition, with caches rebuilt under current parameters, is a clean answer to the prefill-decode kernel mismatch problem in RL training. That contribution stands on its own and is watchable regardless of whether the recurrence beats Transformers.
Third, the social lesson is the most transferable. The cycle that played out here, maximalist announcement, 1.3 million views in a day, an independent lab shipping a controlled evaluation within 48 hours that includes the missing ablation, is now the standard pace of architecture claims. The evaluation that tested exactly the claimed mechanism rather than the vibes is the work that mattered, and it is a healthy pattern for the field. Announcements recruit believers; matched ablations recruit pages like this one.
The Flowtivity verdict on RLT
In our own dual DGX Spark setup running DeepSeek V4 Flash at roughly 60 tokens per second, throughput is the binding constraint on everything we ship, which is why the 15x training slowdown registers with us more than the parity win. An architecture that costs 20 times the GPU-hours for worse language modeling is not a threat to anything in production, and the honest framing survived its first adversarial contact with less damage than most viral claims do. Watch the state-tracking niche and the RL replay idea. Ignore "infinite reasoning depth" as a capability claim. The most interesting thing RLT produced was not a new architecture, but the fastest clean rebuttal cycle we have seen, which is itself a signal about how open frontier research now polices its own hype.
One email a month, no noise
Practical AI notes for Australian businesses. Unsubscribe anytime.
