Transformers fake memory and reset constantly. State space models like Mamba carry evolving internal states – tracking what matters instead of every token. Hybrids mix both approaches for better generalization and lower cost. The shift is from attention-heavy text bursts to stateful systems that persist through time.
The story
What I just learned: the “post-LLM” shift probably isn’t just bigger transformer stacks. It’s a different *shape* of brain.
Transformers have a weird therapy problem: they don’t really carry an inner state. They reset, then try to fake memory with attention, and the bill climbs fast as sequences get longer or more continuous. They’re great for bursts of text, but kind of clumsy over time.
State space models are built around an evolving hidden state – more like a running mental note that updates every step. S4, Hyena, and Mamba treat sequences as dynamics, not just a pile of tokens to stare at. That lines up with how people think: we track the situation, not every word we’ve ever heard.
Mamba goes further with “selective” state updates, where the state transition depends on the input. In plain terms: it chooses what to remember instead of giving everything the same emotional weight. Faster inference, similar quality, less drama.
Hybrids feel like the bridge species. IBM’s Granite mixes transformer layers with state tracking for tasks like counting and event prediction, and it often generalizes better than pure state models. Jamba and Codestral Mamba bring these blocks into production-scale systems without paying the “attention everywhere” tax.
Then things get interesting: neuro-symbolic and brain-inspired hybrids pair temporal state with causal “what-if” simulation for world models, robotics, and real-time 3D planning. Neuromorphic chips co-designed for this could push energy use toward brain-like levels.
So yeah, LLMs were the spark. By 2026, stateful, modular systems look more like the fire. What would you build if your model could stay itself while the world keeps moving?
Recent Comments