Beyond attention · Part 4
Samba, Mamba, and what hybrid models mean for chip design
September 25, 2026
This is the last part of a four part series. Part 1 looked at the energy and water footprint of AI. Part 2 explained how hybrid models differ from transformers, and Part 3 showed why they can cost less to serve. Here we compare two specific designs, Mamba and Samba, and then ask what they mean for the chips that run them.
Beyond attention · Part 4 · At a glance
Samba's memory per stream stops growing. A transformer's does not.
Memory one stream keeps at 128K tokens, matched 1.7B scale models
Our arithmetic from the Samba paper's configs, 16 bit.
2,048
Tokens in Samba's attention window, so its cache stops growing there
119×
More memory per stream for the transformer than Samba at 128K (our arithmetic)
2027
HBM shortage expected through at least the end of 2027 (IEA forecast)
When memory per stream is fixed, a chip can plan for it. Most production hybrids still keep a small attention cache that grows.
The short version. Mamba keeps a fixed size summary of the past. Samba adds a small window of exact recent memory on top. Because that window has a fixed length, Samba’s total memory per conversation stops growing once the window fills. That one property changes how a chip can hold, schedule, and feed the model.
Mamba in one page
A language model reads and writes text as tokens, short chunks of a word. A transformer’s attention layer looks back at every earlier token, so while it generates text it keeps a KV cache, the stored keys and values for every past token. That cache grows with every token and with every user.
Mamba (Albert Gu and Tri Dao, December 2023) takes a different path. It is a state space model. Instead of keeping every past token, each layer carries a fixed size state, a compact running summary that it updates as each new token arrives. Mamba’s twist is that the update is selective. The paper describes parameters that let the model “selectively propagate or forget information along the sequence length dimension depending on the current token.”
The payoff is in generation. The authors write that “autoregressive inference requires only constant time per step since it does not require a cache of previous elements.” They also claim “5× higher throughput than Transformers” and that Mamba-3B “matches Transformers twice its size.” Those are the authors’ results on their benchmarks.
Two follow ups matter. Mamba-2 (Dao and Gu, 2024) showed that state space models and attention are “actually quite closely related,” and its core layer is, per the authors, “2-8X faster” than Mamba’s. Mamba-3 (Lahoti et al., ICLR 2026) adds complex valued state updates and a multi input, multi output form. The authors report “comparable perplexity to Mamba-2 despite using half of its predecessor’s state size.”
The weakness is recall. A fixed state is a lossy summary. Repeat After Me (Jelassi et al., 2024) found that “transformer models dramatically outperform state space models at copying and retrieving information from context.” An NVIDIA study coauthored by Dao and Gu found pure Mamba models “lag behind Transformers on tasks which require strong copying or in-context learning abilities.”
Samba adds a window
Samba (Liliang Ren and colleagues at Microsoft, 2024) is the paper’s answer to that weakness. It “layer-wise combines Mamba, a selective State Space Model (SSM), with Sliding Window Attention (SWA),” plus MLP layers (the feed forward blocks every language model has). Sliding window attention is attention that only looks back a fixed number of tokens. Samba’s window is 2,048 tokens.
The division of labor is simple. The window gives exact recall of the last 2,048 tokens. The Mamba layers carry a compressed summary of everything older.
Architecture
Samba interleaves state layers with window attention, while most production hybrids keep a few full attention layers
Layer order from the Samba paper (Table 11, Figure 1, and the released config) and the Nemotron-H 8B config. Each small bar is one layer, input on the left.
Samba 1.7B12 sliding window attention, 12 state (mamba), 24 mlp
Repeats Mamba, MLP, window attention (2,048 tokens), MLP. Twelve of each mixer.
Nemotron-H 8B4 full attention, 24 state (mamba), 24 mlp
A full attention hybrid for contrast. Four full attention layers among 52.
The Samba authors make strong claims for this design. Pretrained on 4,000 token sequences, they report improved perplexity on contexts “up to 1M” tokens. After fine tuning on 4K sequences, they report it “efficiently extrapolates to a 256K context length with perfect memory recall on the Passkey Retrieval task.” On speed they claim “3.73× higher throughput compared to Transformers with grouped-query attention for user prompts of 128K length,” measured on a single A100 GPU against a 1.6B parameter Llama-3 style model they trained themselves, and a “3.64× speedup when generating 64K tokens.” These are the authors’ numbers against their chosen baseline.
The paper also explains why it avoids full attention. In its ablation, extrapolation perplexity “is exploding at a context length of 16K even if a single full attention layer is placed at the beginning of the model.” And it is candid about limits. The pretrained base model “has retrieval performance similar to that of the SWA-based model,” so the strong recall results come after instruction tuning. Passkey retrieval, finding one planted number in a long text, is also a fairly easy test.
The key property is fixed memory per stream
Call each running conversation a stream. Here is what each design has to keep in memory per stream while it generates.
A full attention transformer keeps a KV cache that grows by the same amount with every token. Mamba keeps only its fixed state. Samba keeps its fixed state plus a window cache that stops growing at 2,048 tokens. Hybrids with full attention layers, like Jamba, Nemotron-H, and Granite 4.0, sit in between. Their cache is small because only a few layers have one, but it still grows.
We can compute this from the Samba paper’s own matched 1.7B scale configs (Table 11), which share a model width of 2,048, 32 query heads, and 4 key value heads. The Llama-3 style baseline has 24 attention layers, which comes to 24 KiB of cache per token at 16 bit precision.
Per stream memory
Samba's memory stops growing at its window, while a transformer's keeps climbing
Memory one stream must keep while generating, at 16 bit precision, from the Samba paper's matched 1.7B scale configs. Our arithmetic, not a measurement. Weights, activations, and framework overhead are excluded.
4K tokensscaled to its largest bar
Transformer 1.6B needs 3.7× the memory of Samba 1.7B
32K tokensscaled to its largest bar
Transformer 1.6B needs 30× the memory of Samba 1.7B
128K tokensscaled to its largest bar
Transformer 1.6B needs 119× the memory of Samba 1.7B
1M tokensscaled to its largest bar
Transformer 1.6B needs 953× the memory of Samba 1.7B
- Transformer 1.6B. Llama-3 style baseline in the Samba paper. 24 full attention layers, 4 KV heads of size 64.
- Samba 1.7B. 12 window attention layers capped at 2,048 tokens, plus 12 Mamba layers of fixed state.
- Mamba 1.8B. 64 Mamba layers of fixed state. Nothing grows with context.
Show as table
| Context | Transformer 1.6B | Samba 1.7B | Mamba 1.8B |
|---|---|---|---|
| 4,096 tokens | 96 MiB | 25.8 MiB | 9.5 MiB |
| 32,768 tokens | 768 MiB | 25.8 MiB | 9.5 MiB |
| 131,072 tokens | 3 GiB | 25.8 MiB | 9.5 MiB |
| 1,048,576 tokens | 24 GiB | 25.8 MiB | 9.5 MiB |
By our arithmetic, Samba needs about 25.8 MiB per stream at any length past the window. The transformer needs 3 GiB at 128K tokens, about 119 times more. Pure Mamba needs 9.5 MiB. The window, not the state, is most of Samba’s footprint.
For contrast, Nemotron-H 8B’s four full attention layers add 16 KiB per token, about 2 GiB at 128K tokens (again our arithmetic, from its config). That is far smaller than a transformer of its size, but it is still a number that grows with the conversation.
Why this matters for chips
The rest of this post stays with published research and general principles. It does not describe any particular chip design.
Generating text is limited by memory bandwidth. To produce each new token, a model reads its weights and each stream’s cache from memory, then does fairly little arithmetic on them. The roofline survey by Yuan et al. explains “why LLMs are memory-bound.” An NVIDIA H100 lists 80GB of memory at 3.35TB/s of bandwidth. Weights are shared across a batch of streams, but each stream’s KV cache is private, so batching does not spread that cost.
The big memory is HBM, and HBM is scarce. HBM (high bandwidth memory) is the stacked DRAM packaged beside an AI chip. It is large and fast by DRAM standards, but it is still off the chip itself.
Memory supply
The memory that holds KV caches is in short supply
Supply facts from the International Energy Agency's April 2026 report and TrendForce. Measured and forecast values are labeled on each card.
2027
An HBM shortage is expected to last through at least the end of 2027 (IEA forecast).
3×
Wafer capacity HBM needs per gigabyte versus conventional memory (Micron figure cited by the IEA).
10×
Rough rise in memory prices from late 2024 to early 2026, "an order of magnitude" (IEA, measured).
The IEA reports that the shortage “is anticipated to persist through at least the end of 2027.” TrendForce reports that Micron “has already locked in pricing and volume agreements covering its entire calendar 2026 HBM supply.” If HBM is the scarce input, a model that needs less of it per user changes the economics directly.
On-chip SRAM is fast but small. SRAM is the memory built into the chip next to the compute. It is much faster than HBM but holds far less. Two companies built whole designs around it. Groq says its “on-chip SRAM has memory bandwidth upwards of 80 terabytes/second,” against about eight for GPU HBM. Cerebras says its wafer scale chip integrates “44GB of SRAM” with “21 petabytes/s of aggregate memory bandwidth.” Those are vendor claims. The trade is capacity for speed, and a large KV cache is exactly what does not fit.
A fixed size state can stay close to compute. Mamba’s own GPU kernel already works this way inside a single call. It loads the parameters “from slow HBM to fast SRAM,” does the recurrence there, and writes only the outputs back. A state that never grows can, in principle, be sized to fit fast memory for a known number of streams. A cache that grows with every token cannot.
Where data lives
A bounded model's per stream memory can sit near compute, while a growing cache is pushed out to HBM
A simplified conceptual picture of a generic accelerator, not a measured layout of any real chip. Block sizes are illustrative, not to scale.
Full attention transformer
Chip
On-chip SRAM small, fast
HBM large, off chip
Weights and a KV cache that grows with every token and every stream both live in HBM. On-chip SRAM mostly holds temporary working tiles.
Samba style hybrid
Chip
On-chip SRAM small, fast
HBM large, off chip
The fixed state and the bounded window cache have a known size per stream, so they could live on chip or in a fixed HBM slot. Weights are still streamed from HBM.
Attention still needs a KV path. Almost every production hybrid keeps some attention. The 56B Nemotron-H model keeps “roughly 8%” attention layers. Granite 4.0 mixes Mamba-2 and transformer blocks at 9 to 1. So hardware for hybrids needs two kinds of path. One is the matrix math and cache reads that attention needs. The other is the scan, the step by step state update that Mamba layers run. The mix of layers tells a designer how the work splits between them. Samba style models are the case where the attention side is bounded too.
Batching and scheduling get more predictable. Serving systems spend real effort managing caches that grow and shrink. The vLLM paper says KV cache memory “grows and shrinks dynamically” and that waste ends up “limiting the batch size.” When per stream memory is fixed, a server can reserve one slot per stream and know in advance how many streams fit.
Our arithmetic gives a sense of scale. Assume about 70 GiB of an 80GB card is free for per stream memory after the small model’s weights, ignoring activations and overhead. At 128K tokens, the 1.6B transformer fits 23 streams. Samba fits 2,780. That is a memory count, not a throughput claim. Compute, bandwidth, and latency limits would bind long before the larger number.
Fixed is not the same as free, though. Marconi (Pan et al., MLSys 2025) shows that hybrids break standard prefix caching, because in place state updates rule out “rolling back cache entries for partial sequence overlaps.” Its authors had to build a new caching policy for that reason.
Published Mamba accelerators
Researchers have already built dedicated hardware for Mamba. All figures below are the authors’ claims against baselines they chose.
- MARCA (Li et al., ICCAD 2024) proposes a reconfigurable accelerator for Mamba and reports “up to 463.22×/11.66× speedup” over an Intel Xeon 8358P CPU and an NVIDIA A100 GPU implementation, with larger energy efficiency gains.
- LightMamba (Wei et al., 2025) runs Mamba on an FPGA, a reprogrammable chip, with most computation cut to 4 bit. It reports “4.65x to 6.06x higher energy efficiency over the GPU baseline” and “93 tokens/s” on an Alveo U280 board, “1.43x that of the GPU baseline.”
- FastMamba (Wang et al., version 4, July 2025) targets Mamba-2 on a Xilinx VC709 FPGA. It reports “68.80× and 8.90× speedup” over an Intel Xeon 4210R CPU and an NVIDIA RTX 3090 GPU for prefill on Mamba2-130M. For decode on Mamba2-2.7B, it reports “6× higher energy efficiency than RTX 3090 GPU.” We cite version 4, the current one.
Read these with care. The models are small, from 130 million to under 3 billion parameters. The GPU baselines range from a consumer RTX 3090 to an older A100. All three target pure Mamba models, not hybrids with attention. Much of each paper’s effort goes into quantization, storing numbers in fewer bits, which fits MiniMax’s warning below about precision.
The honest counterpoints
GPUs run these models well today. Mamba’s fast scan was written for GPUs. Samba’s throughput numbers were measured on an A100, and Nemotron-H’s on H100s. Nothing here says hybrids need new chips to be useful.
Software is less mature. When MiniMax explained why its M2 model returned to full attention after shipping a linear attention hybrid, its pretraining lead wrote that “the infrastructure for linear and sparse attention is much less mature.” He added that “Linear attention is currently far more sensitive to numerical precision than full attention,” and that hybrids showed “clear deficits in complex, multi-hop reasoning tasks.” MiniMax used linear attention rather than Mamba, but it belongs to the same fixed state family.
Recall has limits. A fixed state cannot hold everything. Samba’s window gives exact recall only for its last 2,048 tokens, and beyond that it relies on the state. The paper’s own limitations section says its base model retrieves about as well as a window only model.
Short contexts gain little. Falcon-H1’s authors note that “Transformers are slightly faster at shorter context lengths.” The memory savings above grow with context length and the number of streams.
What to take away
Mamba and Samba point to the same idea. Much of what a model remembers can be a fixed size summary rather than a record that keeps growing. Samba goes further by capping its exact memory too, so its whole per stream footprint is fixed. For hardware, that turns memory from a growing unknown into a known budget per stream, at a time when HBM is scarce. Most production hybrids still keep full attention layers, though, so chips that serve them will need a good attention path and a good scan path, and the software to use both.
Sources
- Mamba: Linear-Time Sequence Modeling with Selective State Spaces, Gu and Dao, arXiv, December 2023.
- Transformers are SSMs (Mamba-2), Dao and Gu, arXiv, May 2024.
- Mamba-3: Improved Sequence Modeling using State Space Principles, Lahoti et al., arXiv, March 2026.
- Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling, Ren et al., Microsoft, arXiv, June 2024 (revised February 2025).
- Samba code and model configs, Microsoft, GitHub, fetched September 2026.
- Repeat After Me: Transformers are Better than State Space Models at Copying, Jelassi et al., arXiv, February 2024.
- An Empirical Study of Mamba-based Language Models, Waleffe et al., NVIDIA, arXiv, June 2024.
- Jamba: A Hybrid Transformer-Mamba Language Model, AI21 Labs, arXiv, March 2024.
- Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models, NVIDIA, arXiv, April 2025.
- Nemotron-H-8B-Base-8K model card and config, NVIDIA, Hugging Face, fetched September 2026.
- IBM Granite 4.0 announcement, IBM, October 2025.
- granite-4.0-h-small model card, IBM, Hugging Face, October 2025.
- LLM Inference Unveiled: Survey and Roofline Model Insights, Yuan et al., arXiv, February 2024.
- NVIDIA H100 Tensor Core GPU, NVIDIA, fetched September 2026.
- Key Questions on Energy and AI, International Energy Agency, April 2026.
- Micron hikes capex, 2026 HBM supply fully booked, TrendForce, December 2025.
- What is a Language Processing Unit?, Groq, fetched September 2026.
- Introducing Cerebras Inference: AI at Instant Speed, Cerebras, August 2024.
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, Dao et al., arXiv, May 2022.
- Efficient Memory Management for Large Language Model Serving with PagedAttention, Kwon et al., arXiv, September 2023.
- Marconi: Prefix Caching for the Era of Hybrid LLMs, Pan et al., arXiv, November 2024.
- MARCA: Mamba Accelerator with ReConfigurable Architecture, Li et al., arXiv, September 2024.
- LightMamba: Efficient Mamba Acceleration on FPGA with Quantization and Hardware Co-design, Wei et al., arXiv, February 2025.
- FastMamba: A High-Speed and Efficient Mamba Accelerator on FPGA with Accurate Quantization (v4), Wang et al., arXiv, July 2025.
- Why Did M2 End Up as a Full Attention Model?, Haohai Sun, MiniMax, October 2025.
- Falcon-H1 blog, Technology Innovation Institute, May 2025.