Last Updated: September 13, 2026
The Recurrent Looped Transformer (RLT) is a new transformer architecture that separates memory from computation: a causal encoder builds global key-value memory once, and a recurrent decoder generates tokens by re-running the same blocks while carrying complete state forward. According to the RLT technical report on GitHub, released September 12, 2026, the computation path after t tokens has traversed t × L_D decoder blocks while blocks executed per token stay fixed, which the authors frame as infinite reasoning depth. Yifan Zhang announced the work on X at 9:27 PM that day, and the post drew 386,100 views in under 24 hours. The released configuration runs 48 encoder plus 48 decoder layers with compatible weights shared, and no benchmarks ship.
"We are at the dawn of Superintelligence. Introducing the Recurrent Looped Transformer (RLT), We now have Transformers with Infinite Reasoning depth. From now on, we should pace progress at the Open Frontier of Superintelligence, Until Safe Superintelligence is achieved,"
wrote Yifan Zhang in the announcement on X. The framing is loud, so this article sticks to what the report claims.
What is the Recurrent Looped Transformer?
The Recurrent Looped Transformer is a two-part architecture introduced by Yifan Zhang in a September 12, 2026 technical report. A causal encoder processes input tokens in parallel and builds a global key-value memory covering every past token. A recurrent decoder then runs the same 48-layer stack for every new token, reading that memory through cross-attention, attending to recent context via a sliding window, and feeding back its previous final hidden state. The complete decoder state H_t = (s_t, C_t^D) bundles the recurrent hidden state with the decoder cache, and next-token probabilities come from a softmax readout on the RMSNorm of s_t.
According to the RLT technical report, weights across the 48 encoder and 48 decoder layers are compatible, and the decoder runs extra cross-attention each step, so its FLOPs exceed the encoder's. Code, English and Chinese PDFs, and the bibtex entry zhang2026recurrentlooped are on the GitHub repository and project page.
What does infinite reasoning depth actually mean?
Infinite depth in RLT is a property of the computation path across tokens, not unlimited computation inside one token. According to the RLT technical report, after t tokens the path has traversed t × L_D decoder blocks: with the released 48-layer decoder, token 40 ends a 1,920-block path that keeps extending. The blocks executed per token stay fixed at 48, so per-token cost is designed to stay flat. The report calls this an extensible temporal computation path, not a measured reasoning gain.
"Reasoning improvements, hardware speedups, and RL scaling are research goals rather than measured results in this report,"
the RLT technical report states. That line matters: this is an architecture and a training recipe, not measured wins.
How does RLT differ from a standard transformer?
A standard transformer runs every token through a fixed stack once and carries an append-only KV cache. RLT re-runs one decoder stack recurrently, so effective depth accumulates. Its complete state H_t = (s_t, C_t^D) does not reset at the prompt-response boundary: neither the recurrent output nor the sliding-window cache resets when answering starts. Attention memory is global encoder key-value memory plus a sliding window holding the current token and W minus 1 historical entries, while per-token compute stays fixed.
| Aspect | Standard Transformer | Recurrent Looped Transformer |
|---|---|---|
| Compute path | Fixed: L blocks per token | Grows to t × L_D blocks after t tokens |
| Blocks executed per token | Full stack every step | Fixed 48 decoder blocks, re-run recurrently |
| State carried | Append-only KV cache | Complete state H_t = (s_t, C_t^D), recurrent hidden state plus cache |
| Attention memory | Full attention over the context window | Global encoder KV memory plus sliding window with W minus 1 past entries |
| Prompt boundary | Prefill and decode run as separate phases | No reset: one transition spans prompt and response |
| Evidence status | Mature architecture, heavily benchmarked | No published benchmarks, goals stated as research directions |
How can one transition serve prefill, training, and RL?
RLT uses the same complete-state transition everywhere. Prefill runs the recurrence to build state, generation continues it, pretraining trains it with full backpropagation through time, SFT supervises assistant tokens inside it, and current-policy RL replay rebuilds rollouts under current weights. According to the RLT technical report, detaching any state during replay counts as a gradient approximation, not a change of dynamics. Because nothing resets at the prompt boundary, the answer is produced with the same machinery and state lineage that processed the question.
Why does memory bandwidth decide whether RLT ships?
A sequential recurrent decoder cannot hide behind parallel prefill: every token depends on the previous final hidden state, so decode throughput is bound by state and memory movement, not raw FLOPs. The report treats hardware co-design as part of the contribution, listing parallel encoder work, cross-sequence batching, memory reuse, and activation checkpointing, and links a companion note on the prefill-decode kernel mismatch. The deployment question is whether sliding window plus global memory fit real bandwidth budgets.
We see the same constraint in our own stack. In our dual DGX Spark setup running DeepSeek V4 Flash, decode speed rather than model quality sets agent throughput at roughly 60 tokens per second, and the ceiling is memory bandwidth on the sequential decode path. RLT leans into that wall: per-token blocks stay fixed while the path grows, so it lives or dies on streaming state through memory.
What does growing depth mean for AI agent strategies?
For teams betting on AI agents, growing-depth architectures are worth tracking for three reasons. Agent workloads are long and stateful, and RLT is designed so state never resets, which matches the shape of the problem. Cost per token is designed to stay flat as effective depth grows, because blocks per token stay fixed at 48 even while the path reaches thousands of blocks. And evals must shift toward stateful long-horizon tasks, because quiz benchmarks say little when depth grows with context. None of this is measured yet, so treat RLT as a direction to monitor, not a capability to buy.
The practical stance: keep production agents on proven serving stacks, benchmark your longest workflows rather than generic leaderboards, and watch memory bandwidth roadmaps, since they set the ceiling for any recurrent decoder. If RLT-style models ship measured results later, pilot on your most stateful workflow first.
Frequently asked questions
What is the Recurrent Looped Transformer?
An architecture that splits a transformer into a causal encoder building global key-value memory and a recurrent decoder re-running the same 48-layer stack with cross-attention, sliding-window attention, and state feedback. Announced September 12, 2026.
What does infinite reasoning depth mean?
The compute path grows to t × L_D blocks after t tokens while per-token blocks stay fixed at 48: an extensible path, not unlimited computation in one token.
How does RLT differ from a standard transformer?
A standard transformer gives each token one pass through a fixed stack plus an append-only KV cache. RLT re-runs one decoder stack, carrying state that never resets at the prompt boundary.
Has RLT been benchmarked?
No. The report states that reasoning improvements, hardware speedups, and RL scaling are research goals rather than measured results.
