Beyond attention · Part 2
What makes hybrid AI different from transformer AI
September 25, 2026
Almost every chatbot you have used since 2022 runs on the same basic design, the transformer. It was introduced in the 2017 paper Attention Is All You Need, whose authors built a model “based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.”
Beyond attention · Part 2 · At a glance
Hybrids keep a few attention layers for exact recall and make the rest state layers
Share of layers that use full attention
From published papers and model configs. Samba uses only window attention.
82%
Of the quality gap to attention that Zoology traces to recall
Fixed
Memory a state layer keeps, however long the input
12 of 12
Standard tasks where NVIDIA's 8B hybrid beat its 8B transformer
From the outside a hybrid works like any language model. Inside, most layers carry a fixed summary and a few look back exactly.
A newer family of models breaks with that. They swap most of the attention layers for a different kind of layer, one that keeps a running summary of the past instead of the past itself. These are called hybrid models. NVIDIA, IBM, AI21, Microsoft, Alibaba’s Qwen team, and Moonshot AI have all published or shipped one.
This post explains what actually changes inside the model, what does not, and why the mix exists at all. Part 1 covered the energy picture, and Part 3 covers serving cost. Here the question is simpler. How does each kind of layer remember?
First, a few terms
A language model reads and writes text as tokens, chunks of a word or a short word. It generates one token at a time, and each new token has to take into account everything that came before it in the conversation or document. That running text is the context.
A model is a stack of layers. Each layer takes in a representation of every token so far and passes an updated one to the next layer. Two kinds of layer matter for this post. One kind mixes information across tokens (this is where attention or a state layer sits). The other kind, the MLP or feed forward block, works on each token by itself and holds much of what the model learned in training. Every design in this post keeps the MLP blocks. What differs is the token mixing layer.
What an attention layer does
When a transformer produces a new token, each attention layer compares that token against every earlier token and decides how much weight to give each one. If you ask about the name of a character introduced forty pages ago, attention can look straight back at those exact tokens.
To make this fast, the model does not recompute the earlier tokens every time. It stores two short vectors per token per attention layer, a key and a value, and reuses them. This store is the KV cache (key value cache). It is the model’s working memory for the conversation, and it is exact.
The catch is that it grows with every token. The vLLM paper from 2023 describes the cache as “huge” and says it “grows and shrinks dynamically.” There is also a compute cost. Longformer notes that standard self attention “scales quadratically with the sequence length,” because every token is compared against every other one when the model first reads a long input.
To make the size concrete, here is our arithmetic from the published Llama 3 70B configuration (80 layers, 8 key value heads, head size 128, 16 bit numbers). At 128K tokens of context, one conversation needs about 40 GiB of KV cache. That is more than half of an 80 GB H100 for a single user, before the model’s weights. Engineers have found ways to shrink this, such as grouped query attention, which Llama 3 already uses, and sliding window attention, which only keeps the last few thousand tokens. But in full attention the cache still grows with the context.
What a state space layer does
A state space model, or SSM, takes the older idea of a recurrent network and makes it trainable at scale. The best known version today is Mamba, from Albert Gu and Tri Dao in December 2023, which built on earlier work such as S4.
Instead of storing every earlier token, a Mamba layer keeps a state, a fixed size block of numbers that summarizes what it has read. Each new token updates the state, and the layer produces its output from the updated state. The past tokens themselves are not kept.
Mamba’s key trick is that the update depends on the input. The authors describe it as letting the model “selectively propagate or forget information along the sequence length dimension depending on the current token.” In plain terms, the layer learns what is worth writing into its limited memory and what to let fade.
This has two big consequences. Memory per conversation stays the same whether the context is one thousand tokens or one million. And the work per new token is constant too. The paper says “autoregressive inference requires only constant time per step since it does not require a cache of previous elements,” and the total cost grows linearly with length rather than quadratically. The Mamba authors claim “5× higher throughput than Transformers” at inference. Later versions, Mamba-2 in 2024 and Mamba-3 in 2026, refine the same idea.
How layers remember
Attention keeps every token, a state layer keeps one summary, and a hybrid keeps both kinds
Schematic of what each layer type reads when it produces one new token. Not to scale, no measured values.
Attention
Looks back at every earlier token and must keep all of them.
Sliding window attention
Looks back only at the last 3 tokens. Memory is capped.
State layer (Mamba)
Reads one fixed size summary, then updates it. Memory never grows.
Hybrid
Stacks both. The state carries the long past, attention handles exact recent detail.
The weakness of a fixed size memory
A summary is lossy by design. If a model has to squeeze a whole book into a fixed number of values, some exact details will not survive. For many tasks that is fine. For some it is not.
Researchers have measured this directly. The Zoology study (December 2023) compared attention models with efficient alternatives and found that “82% of the gap is explained by each model’s ability to recall information that is previously mentioned in-context.” The task is called associative recall. If a document says a key is paired with a value, can the model later retrieve the value when shown the key? In their tests a 70M parameter attention model beat a 1.4B parameter gated convolution model on it, a model twenty times larger.
The Repeat After Me paper (February 2024) looked at copying. The authors show that a small transformer can in principle copy very long strings, while state space models “are fundamentally limited by their fixed-size latent state.” On real pretrained models, they report that “transformer models dramatically outperform state space models at copying and retrieving information from context.”
The follow up paper on Based names the underlying rule, “a key tradeoff between a model’s state size and recall ability.” A bigger state remembers more exactly but costs more. There is no free lunch.
Why mixing works
The same Zoology paper hinted at the fix. It found that “hybrids with input-dependent sparse attention patterns can close 97.4% of the gap to attention, while maintaining sub-quadratic scaling.” In other words, you do not need attention everywhere. You need a little of it in the right places.
The most cited test at scale came from NVIDIA in June 2024, with Dao and Gu among the authors. In An Empirical Study of Mamba-based Language Models, pure Mamba models “lag behind Transformers on tasks which require strong copying or in-context learning abilities.” But an 8B hybrid built from “43% Mamba-2, 7% attention, and 50% MLP layers” “exceeds the 8B Transformer on all 12 standard tasks” the team evaluated, and the authors say it is “predicted to be up to 8x faster when generating tokens at inference time.” Note the word predicted. That speed figure is a projection in the paper, not a measured serving result.
The recall problem
Recall explains most of the gap, and a few attention layers close most of it
Figures the authors report from their own experiments. Measured on the tasks each paper chose, not reproduced here.
82 %
Share of the quality gap between attention and efficient models that Zoology attributes to recall
97.4 %
Share of that gap closed by hybrids with input dependent sparse attention, per Zoology
12 of 12
Standard tasks on which NVIDIA reports its 8B hybrid beat an 8B transformer
The intuition is a division of labor. The state layers carry the gist of a long context cheaply. The occasional attention layer can still look back at exact tokens when the task needs a name, a number, or a line to copy.
What real hybrids look like
Since 2024 several labs have shipped models built this way. The layouts differ, but the pattern is similar. Roughly one attention layer for every three to nine cheaper layers.
- Jamba from AI21 (March 2024) uses an attention to Mamba ratio of 1 to 7. Its 32 layers form four blocks of eight, with one attention layer in each block and mixture of experts layers (a way of splitting the MLP into many specialist parts) every other layer.
- Nemotron-H from NVIDIA (April 2025). The 8B model has 52 layers, of which 4 are attention, 24 are Mamba-2, and 24 are MLP. NVIDIA puts attention at “roughly 8%” of layers across the family and claims the models are “up to 3× faster at inference” than similar sized transformers.
- IBM Granite 4.0 (October 2025) combines “Mamba-2 blocks and conventional transformer blocks sequentially in a 9:1 ratio.” The H-Small model card lists 4 attention and 36 Mamba-2 layers.
- Samba from Microsoft (June 2024) goes a step further. It interleaves Mamba with sliding window attention that only looks back 2,048 tokens, so even its attention memory has a cap. Part 4 looks at Samba in detail.
A related branch replaces Mamba with linear attention, a reworked attention that, like a state layer, keeps a fixed size running state. Qwen3-Next from Alibaba stacks three Gated DeltaNet layers for every full attention layer. Kimi Linear from Moonshot AI uses the same 3 to 1 ratio with its own linear layer. Other designs, such as Falcon-H1 and NVIDIA’s Hymba, run attention and state heads side by side inside each layer rather than stacking them. The family resemblance is the same. Most layers keep a fixed state, and a few keep an exact memory.
Layer layouts
Hybrids replace most attention layers with state layers and keep the MLP blocks
Published layer order of five models, input on the left. Each colored cell is one layer. From model papers and configuration files, read September 2026.
Llama 3 70B (transformer)80 full attention, 80 mlp
80 layers, each with full attention. Every layer keeps a KV cache.
Jamba (AI21)4 full attention, 28 state (mamba), 32 mlp
32 layers in four blocks of eight, one attention layer per block.
Granite 4.0 H-Small (IBM)4 full attention, 36 state (mamba), 40 mlp
40 layers, attention at layers 6, 16, 26, and 36.
Samba 1.7B (Microsoft)12 sliding window attention, 12 state (mamba), 24 mlp
No full attention. The window attention looks back 2,048 tokens.
The memory effect of that layout follows from the arithmetic. Granite 4.0 H-Small keeps a KV cache in only 4 of its 40 layers. By our arithmetic from its config, that is about 2.0 GiB of KV cache at 128K tokens, plus a fixed Mamba state of roughly 77 MB that does not grow. IBM itself claims “over 70% reduction in RAM needed to handle long inputs and multiple concurrent batches.” Part 3 follows that saving through to cost per token.
What does not change
It is easy to overstate how different these models are. Much stays the same.
- Same tokens in and out. Hybrids read and write the same kind of tokens as transformers, and often use the same kinds of tokenizers.
- Same MLP blocks. The feed forward layers, and in many cases the mixture of experts designs, are carried over unchanged. In the NVIDIA 8B study, half of all layers are MLP.
- Same training recipe, broadly. Hybrids are pretrained the same way, by predicting the next token over very large text collections, then tuned for chat and instructions. The labs above describe familiar data, optimizers, and fine tuning stages rather than a new way to learn.
- Same use from the outside. A user or developer sends a prompt and gets text back. Nothing in the chat window reveals which kind of layer produced it.
What changes is the part that sits between tokens, how the model carries the past forward.
Side by side
A hybrid trades a little exact memory for a much smaller working memory
Qualitative summary of the three designs, drawn from the papers cited in this post. Not a benchmark.
| Transformer | State model (Mamba) | Hybrid | |
|---|---|---|---|
| How it reads the past | Looks at every earlier token | Reads one running summary | Mostly summaries, plus a few layers that look back |
| Working memory per conversation | Grows with every token | Fixed size | Grows slowly (a few attention layers), or fixed with a window |
| Work per new token | Grows with context length | Constant | Mostly constant |
| Exact recall and copying | Strong | Weak, limited by state size | Restored by the attention layers |
| MLP blocks and tokens | Standard | Same | Same |
| Tooling maturity | Mature | Younger | Younger, prefix caching is harder |
Honest counterpoints
Hybrids are not a settled win, and some of the strongest caveats come from labs that built them.
MiniMax shipped a hybrid with seven linear attention layers per full attention layer in MiniMax-01, then went back to full attention for its next model. In an October 2025 post, its pretraining lead Haohai Sun wrote that the hybrids showed “clear deficits in complex, multi-hop reasoning tasks” at larger scale, that “the infrastructure for linear and sparse attention is much less mature,” and that linear attention is “far more sensitive to numerical precision than full attention.” His conclusion was that “efficient attention still has some way to go before it can definitively beat full attention.”
The savings also depend on the workload. The Falcon-H1 team, who built a hybrid, note that “Transformers are slightly faster at shorter context lengths.” The hybrid advantage grows with long inputs and many users at once. For a short chat the difference can be small or even go the other way.
Serving software has to catch up too. A common trick is prefix caching, reusing the stored work for a shared opening such as a long system prompt. With attention you can reuse any matching prefix. The Marconi paper (MLSys 2025) points out that hybrids update their state in place, which “precludes rolling back cache entries for partial sequence overlaps, and instead mandates only exact-match cache hits.” Marconi proposes a caching policy to recover much of that, but it is extra engineering that transformers do not need.
Finally, most of the efficiency numbers in this post are claims from the teams that built the models, measured on the tasks and hardware they chose. They point in a consistent direction, which is meaningful. They are not independent benchmarks.
The short version
A transformer remembers everything exactly and pays for it in memory that grows with every token. A state model remembers a fixed size summary and pays for it in recall. A hybrid keeps a few attention layers so it can still look things up exactly, and makes the rest cheap. From the outside it looks and trains like any other language model. Inside, most of its layers have stopped keeping the whole past.
Sources
- Attention Is All You Need, Vaswani et al., June 2017.
- Longformer: The Long-Document Transformer, Beltagy, Peters, Cohan, April 2020.
- Efficiently Modeling Long Sequences with Structured State Spaces (S4), Gu, Goel, Ré, October 2021.
- GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints, Ainslie et al., Google, May 2023.
- Efficient Memory Management for Large Language Model Serving with PagedAttention, Kwon et al., September 2023.
- Mistral 7B, Mistral AI, October 2023.
- Mamba: Linear-Time Sequence Modeling with Selective State Spaces, Gu and Dao, December 2023.
- Zoology: Measuring and Improving Recall in Efficient Language Models, Arora et al., December 2023.
- Repeat After Me: Transformers are Better than State Space Models at Copying, Jelassi, Brandfonbrener, Kakade, Malach, February 2024.
- Simple linear attention language models balance the recall-throughput tradeoff (Based), Arora et al., February 2024.
- Jamba: A Hybrid Transformer-Mamba Language Model, AI21 Labs, March 2024.
- Jamba-v0.1 configuration file, AI21 Labs, Hugging Face, 2024.
- Transformers are SSMs (Mamba-2), Dao and Gu, May 2024.
- Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling, Ren et al., Microsoft, June 2024.
- An Empirical Study of Mamba-based Language Models, Waleffe et al., NVIDIA, June 2024.
- The Llama 3 Herd of Models, Meta, July 2024.
- Hymba: A Hybrid-head Architecture for Small Language Models, NVIDIA, November 2024.
- Marconi: Prefix Caching for the Era of Hybrid LLMs, Pan et al., MLSys 2025, November 2024.
- MiniMax-01: Scaling Foundation Models with Lightning Attention, MiniMax, January 2025.
- Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models, NVIDIA, April 2025.
- Nemotron-H-8B-Base-8K configuration file, NVIDIA, Hugging Face, 2025.
- Falcon-H1 blog, Technology Innovation Institute, May 2025.
- Falcon-H1: A Family of Hybrid-Head Language Models, Technology Innovation Institute, July 2025.
- Qwen3-Next-80B-A3B-Instruct model card, Alibaba Qwen, September 2025.
- IBM Granite 4.0 announcement, IBM, October 2025.
- granite-4.0-h-small model card and configuration file, IBM, Hugging Face, October 2025.
- Why Did M2 End Up as a Full Attention Model?, Haohai Sun, MiniMax, October 2025.
- Kimi Linear: An Expressive, Efficient Attention Architecture, Moonshot AI, October 2025.
- Mamba-3: Improved Sequence Modeling using State Space Principles, Lahoti et al., March 2026.