State Space Model (SSM)

A State Space Model (SSM) is an architectural family for processing sequences that compresses everything seen so far into a fixed-size hidden state and updates that state one token at a time using a linear recurrence, rather than attending back over every previous token the way a Transformer does. Because the update per step doesn’t depend on how long the sequence has grown, SSMs process a sequence of length L in O(L) time and constant memory per step, versus the O(L²) time and growing key-value cache that self-attention requires. The idea traces back to classical control theory (the same state-space equations used to model circuits and mechanical systems), was adapted for deep learning through a line of work called S4, and became widely known through Mamba, the specific model that made SSMs competitive with Transformers on language modeling for the first time. This entry covers the general SSM mechanism, the math underneath it, and how the family as a whole trades off against attention; Mamba itself, as one particular instantiation, has its own dedicated entry.

The Continuous-Time State Equation

Every SSM starts from a pair of equations borrowed directly from control theory, describing how a hidden state x(t) evolves in continuous time in response to an input u(t):

x'(t) = A x(t) + B u(t)
y(t)  = C x(t) + D u(t)

Here x(t) is the hidden state (a vector), u(t) is the input signal at time t, and y(t) is the output. A is the state transition matrix, controlling how the existing state decays or persists; B controls how much the new input is written into the state; C reads the state back out into an output; and D is a direct pass-through term (often dropped or folded into a residual connection in practice). Nothing here is unique to deep learning: this is the same formulation used to describe an RC circuit or a spring-damper system, and it’s exactly why the SSM literature borrows so much language (“state,” “transition,” “observation”) from classical signal processing rather than from NLP.

The hard part is choosing A, B, and C well. A poorly initialized A either forgets everything within a few steps (the state decays to zero) or blows up numerically. The HiPPO framework (Gu, Dao, Ermon, Rudra & Ré, 2020) solved this by deriving a specific structured form for A, the HiPPO matrix, that provably gives the state an optimal running compression of the entire input history under a chosen notion of “how much older information matters.” This is what let S4 and its successors, including Mamba, actually train stably on long sequences instead of relying on a randomly initialized A that either forgets or explodes.

Discretization: From Continuous Math to Discrete Tokens

Language models don’t operate on continuous time; they operate on a discrete sequence of tokens. So before any of this can run on token 1, token 2, token 3, the continuous equations above need to be converted into a discrete recurrence. This step is called discretization, and it introduces one more parameter: Δ (delta), a step size that represents how much continuous time passes between one token and the next.

The most common discretization method is the zero-order hold (ZOH), which assumes the input stays constant between sample points and integrates the continuous equation exactly under that assumption. ZOH turns the continuous (A, B) into discrete versions:

A_bar = exp(Δ * A)
B_bar = (Δ * A)^-1 * (exp(Δ * A) - I) * Δ * B

The resulting discrete-time recurrence, the form an SSM actually runs at inference time, is:

# A naive discretized SSM recurrence step, one token at a time.
# Scenario: streaming sensor readings from an industrial monitoring
# system where tokens arrive continuously and memory must stay flat.
def ssm_step(x_prev, u_t, A_bar, B_bar, C):
    x_t = A_bar @ x_prev + B_bar * u_t   # update hidden state
    y_t = C @ x_t                        # read output from state
    return x_t, y_t

x = initial_state
outputs = []
for u_t in token_stream:                # O(1) work per token, O(L) total
    x, y_t = ssm_step(x, u_t, A_bar, B_bar, C)
    outputs.append(y_t)

Δ is not just a technical footnote, it is the single most important tunable quantity in an SSM: a large Δ lets the state change quickly and forget the past faster, effectively narrowing the model’s “attention span” backward in time, while a small Δ makes the state change slowly and retain information over many more steps. In the original S4 formulation, Δ was a fixed, learned scalar (or one per channel), the same for every position in every sequence, a limitation that mattered enormously once Mamba introduced input-dependent parameters (see below).

Two Equivalent Views: Recurrent and Convolutional

One of the more elegant properties of a linear time-invariant (LTI) SSM, meaning one where A, B, C, and Δ are fixed and don’t change per token, is that the same recurrence can be rewritten as a single global convolution. Unrolling the recurrence step by step shows that the output at time t is a weighted sum of all past inputs, where the weights form a convolution kernel built from powers of A_bar:

# Convolutional-form SSM: precompute a kernel, then run one FFT-based
# convolution instead of a token-by-token loop.
# Scenario: training on fixed-length batches of protein sequences,
# where full parallelism across the sequence dimension matters more
# than streaming, so the convolutional form is preferred at train time.
import numpy as np

def build_ssm_kernel(A_bar, B_bar, C, length):
    # K[i] = C @ (A_bar ** i) @ B_bar, the impulse response at lag i
    return np.stack([C @ np.linalg.matrix_power(A_bar, i) @ B_bar
                      for i in range(length)])

def ssm_convolve(u, kernel):
    # Full sequence processed at once via FFT convolution, not a loop
    L = len(u)
    padded_len = 2 * L
    u_fft = np.fft.rfft(u, n=padded_len)
    k_fft = np.fft.rfft(kernel, n=padded_len)
    return np.fft.irfft(u_fft * k_fft, n=padded_len)[:L]

This dual view is the reason SSMs are practical at all: the recurrent form is used at inference time (cheap, O(1) memory per step, ideal for streaming generation), while the convolutional form is used at training time (fully parallelizable across the sequence dimension on a GPU, the same way a Transformer’s attention matrix is computed all at once rather than token by token). S4 relied heavily on this duality; Mamba-2’s “structured state space duality” (SSD) result later showed formally that this same convolutional view is mathematically a special case of linear attention, drawing a direct theoretical bridge between the SSM and Transformer families rather than treating them as unrelated approaches.

Linear Time vs. Quadratic Attention

The entire practical case for SSMs rests on one asymptotic fact. Self-attention computes a similarity score between every pair of tokens in a sequence, an O(L²) operation in both compute and the memory needed to store the key-value cache; techniques like FlashAttention and sliding-window attention reduce the constant factor or cap the window, but the underlying pairwise comparison, or an approximation of it, is still there. An SSM, by contrast, never materializes a pairwise comparison at all: it only ever updates a fixed-size state and reads from it, so both compute and memory per step are O(1), and the whole sequence is O(L).

graph TB
    classDef default fill:#ffffff,stroke:#4338CA,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef data    fill:#EEF0F7,stroke:#0D9488,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef process fill:#F7F8FC,stroke:#6366F1,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef output  fill:#4338CA,stroke:#4338CA,stroke-width:2px,color:#ffffff,rx:8px,ry:8px;

    subgraph SSM["SSM Recurrent Scan (O(L))"]
        U1([u_t]):::data --> UPD["State update: x_t = A x_t-1 + B u_t"]:::process
        XPREV[("x_t-1, fixed size")]:::data --> UPD
        UPD --> XNEXT[("x_t, carried forward")]:::data
        UPD --> READ["Readout: y_t = C x_t"]:::process --> Y1([y_t]):::output
        XNEXT -.-> UPD
    end

    subgraph ATT["Self-Attention Block (O(L^2))"]
        Q([Query t]):::data --> SCORE["Score against every key 1..t"]:::process
        K[("Keys 1..t, growing cache")]:::data --> SCORE
        SCORE --> SM["Softmax over all t scores"]:::process
        V[("Values 1..t, growing cache")]:::data --> SM
        SM --> Y2([y_t]):::output
    end

The diagram’s left half is the entire SSM story in one loop: the state box carries forward, it never grows, and each new token only ever touches that one fixed-size box. The right half shows why attention costs more as sequences grow: the key and value caches on the left keep growing with t, so both the comparison step and the memory footprint scale with how much text has already been processed. Drag the slider below to see how far apart the two costs actually get as sequence length increases.

The SSM curve (teal) is scaled up visually so it stays visible near the axis; its true cost is L, while attention's true cost is L² (indigo), read out numerically below the chart.

At 2,000 tokens the gap is barely visible; by 20,000 tokens, attention’s raw pairwise-comparison count is a hundred times larger than an SSM’s, which is exactly why context windows in the hundreds of thousands of tokens are practical for SSM and hybrid models in a way they simply aren’t for pure quadratic attention without heavy approximation.

The Selective Innovation: Breaking Time-Invariance

Everything described so far describes a linear time-invariant (LTI) SSM: A, B, C, and Δ are fixed parameters, learned once during training and then applied identically at every position of every sequence. This is precisely what makes the convolutional form above possible, but it’s also a real limitation: an LTI system cannot decide, based on what it’s currently reading, to remember one token and immediately forget the next. It treats every position the same way structurally, which is a poor match for language, where whether a token matters depends heavily on the token itself (a name that will be referenced later, versus a filler word that won’t).

Mamba’s core contribution was to make B, C, and Δ functions of the input at each timestep, rather than fixed learned matrices shared across the whole sequence:

# LTI SSM: same A, B, C, Delta for every timestep in every sequence.
Ab, Bb, C, delta = fixed_A_bar, fixed_B_bar, fixed_C, fixed_delta
for u_t in sequence:
    x = Ab @ x + Bb * u_t
    y_t = C @ x

# Mamba's selective SSM: B, C, and Delta are computed FROM the current
# input, so the model can choose, token by token, what to keep or drop.
# Scenario: parsing a legal contract where a defined term ("the Buyer")
# must be remembered for hundreds of tokens, but boilerplate connective
# words should be forgotten almost immediately.
for u_t in sequence:
    B_t = input_dependent_B(u_t)         # learned projection of u_t
    C_t = input_dependent_C(u_t)
    delta_t = softplus(input_dependent_delta(u_t))
    Ab_t = discretize(A, delta_t)         # Delta now varies per token too
    x = Ab_t @ x + B_t * u_t
    y_t = C_t @ x

This breaks the pure linear time-invariance that made the convolutional form so cheap to compute, so Mamba pairs the idea with a hardware-aware parallel scan, an algorithm that recomputes the recurrence efficiently on GPU by exploiting the fact that the scan is still associative, even though the parameters change per step, and by keeping intermediate states in fast SRAM rather than repeatedly reading and writing to slower GPU memory (the same spirit of hardware-awareness behind FlashAttention). The result is a model that keeps the O(L) asymptotic cost of an SSM while gaining much more of the input-dependent, content-based behavior that made attention so effective in the first place. Mamba-2’s structured state space duality result later reframed this selective mechanism as a special, highly structured case of linear attention, formally connecting the two families rather than treating “selective SSM” as an unrelated trick.

Where SSMs Still Struggle

The selective mechanism closed much of the gap with attention, but not all of it. Attention has one structural advantage that a fixed-size state cannot fully replicate: it keeps an exact, addressable copy of every past token’s key and value, so it can retrieve any one of them with perfect precision regardless of how long ago it appeared. An SSM’s state, however large, is still a compressed summary; something has to be dropped or blended as new tokens arrive. This shows up concretely in tasks requiring exact copying or associative recall, such as reproducing a long verbatim quote, looking up a specific fact mentioned once far earlier in a document, or “needle in a haystack” retrieval benchmarks, where pure SSM models have historically underperformed Transformers of comparable size, even though they win decisively on raw throughput and long-context memory footprint.

Hybrid Architectures: Combining Both

The dominant response to this trade-off, as of 2025-2026, is not to pick a side but to interleave SSM and attention layers in the same model, letting each do what it’s structurally best at.

Jamba (AI21 Labs) was the first large-scale production example, interleaving blocks of Transformer and Mamba layers with a mixture-of-experts layer added periodically for capacity, reporting a 256K-token context window that needs a fraction of the KV-cache memory a comparable pure-Transformer model would require. Zamba and Hymba followed with their own interleaving ratios and, in Hymba’s case, parallel rather than sequential combination of attention and SSM heads within the same layer. Nemotron-H applied the same hybrid principle at larger scale, replacing most self-attention layers with Mamba-2 layers and keeping a small number of attention layers specifically to preserve recall quality. The common thread across all of them: use SSM layers for the bulk of the sequence to keep compute and memory near-linear, and sprinkle in just enough attention layers to cover the recall weaknesses a pure SSM stack would otherwise have.

# Illustrative hybrid layer routing, in the spirit of Jamba/Nemotron-H.
# Scenario: processing a 100k-token log file where quadratic attention
# over the whole file would be too slow, but a few exact-match lookups
# (error IDs referenced earlier in the file) still need real recall.
LAYER_PATTERN = ["mamba"] * 7 + ["attention"]  # 1 attention layer per 8

def forward(hidden_states, layer_idx):
    layer_type = LAYER_PATTERN[layer_idx % len(LAYER_PATTERN)]
    if layer_type == "mamba":
        return selective_ssm_layer(hidden_states)   # O(L), cheap per token
    return self_attention_layer(hidden_states)       # O(L^2), used sparingly

What’s New (2025-2026)

• Mamba-2 and structured state space duality. Dao & Gu’s SSD result reframed the selective scan as a special case of linear attention, unifying the theory behind SSMs and Transformers and enabling a 2-8x faster core layer than the original Mamba while remaining competitive on language modeling.
• Hybrid architectures becoming the default for long-context production models, rather than a research curiosity: Jamba, Zamba, Hymba, and Nemotron-H all ship mixed SSM/attention stacks specifically to get near-linear scaling without giving up recall quality.
• SSM layers inside otherwise-Transformer model families, as several 2025-2026 model releases quietly add a handful of Mamba-style layers into an existing Transformer stack rather than committing to a fully separate architecture, treating “SSM layer” as one more building block alongside attention and MoE rather than a rival paradigm.
• Distillation from attention to SSMs. Work on converting pretrained quadratic-attention Transformers into subquadratic SSM-like models via distillation has continued to mature, aiming to recover much of an existing model’s quality while inheriting SSM-style inference costs, rather than training a hybrid architecture from scratch.
• Growing use in non-language modalities, including genomics, audio, and time-series forecasting, where sequence lengths routinely exceed what quadratic attention can handle affordably, making the O(L) cost of an SSM a hard requirement rather than a nice-to-have.

Summary

A State Space Model processes a sequence by carrying a fixed-size hidden state forward through a linear update rule, discretized from continuous-time equations via a step size Δ and (in modern designs) initialized using the HiPPO framework for stable long-range memory. That single design choice, no pairwise comparison across the whole sequence, is what gives the family its O(L) compute and constant per-step memory, against attention’s O(L²) cost, and it’s why SSMs matter for genuinely long sequences (100k-token documents, genomic sequences, hours of audio) where quadratic attention becomes the bottleneck. Mamba’s selective SSM innovation, making B, C, and Δ input-dependent, closed much of the quality gap with attention on language tasks by letting the model choose what to remember token by token, at the cost of the pure linear time-invariance that made the original convolutional form so cheap. The remaining gap, mostly around exact recall and copying over very long contexts, is what hybrid SSM-attention architectures like Jamba and Nemotron-H are built to close, treating SSM layers and attention layers as complementary tools rather than competing philosophies.

# Scenario: summarizing 200-page compliance PDFs where the whole
# document must fit in context, and a quadratic-attention model
# would blow the latency and memory budget.
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "state-spaces/mamba-2.8b-hf"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

long_document = open("compliance_report.txt").read()  # ~150k tokens
inputs = tokenizer(long_document, return_tensors="pt")

# Memory stays roughly constant per generated token regardless of
# how long `long_document` is, because inference only ever touches
# a fixed-size hidden state, not a growing KV cache.
output = model.generate(**inputs, max_new_tokens=500)
print(tokenizer.decode(output[0], skip_special_tokens=True))