Sliding Window Attention (SWA)

Sliding Window Attention (SWA) caps how far back a token is allowed to look: instead of attending to every earlier token in the sequence, each position only attends within a fixed-size window of its nearest neighbors. That single change turns attention’s cost from quadratic in sequence length, O(n²), into linear, O(n·w), where w is the window size, since every token now does a constant amount of attention work regardless of how long the sequence gets. First proposed for long-document encoders in Longformer, SWA became a decoder-only LLM staple after Mistral 7B combined it with Grouped Query Attention (GQA) in 2023, and it now appears, usually interleaved with occasional full-attention layers, in Gemma, GPT-OSS, and most other context-length-conscious open-weight model families.

Why Full Attention Doesn’t Scale

In standard self-attention, every one of n tokens computes a score against every other token, producing an n × n attention matrix. Double the sequence length and the compute and memory for that matrix quadruple, not double. Flash Attention makes computing that full matrix dramatically more I/O-efficient, but it doesn’t change its size: the matrix is still O(n²), and at inference time the KV cache backing it still grows with every token generated. For a 128K-token context, a full attention matrix has over 16 billion entries per head, per layer. Something has to give if context windows are going to keep growing without a proportional blowup in compute and memory.

SWA gives up global, unrestricted lookback in exchange for that scaling. Rather than trying to make the full O(n²) computation cheaper, it changes the computation itself so most token pairs never need to be scored at all.

The Mechanism: A Banded Attention Mask

With window size w, token at position i attends only to tokens in the range [i - w + 1, i] under a causal mask (or [i - w/2, i + w/2] for a bidirectional encoder like Longformer). Every other position gets masked out, exactly as if its attention score were set to negative infinity before the softmax. The resulting attention matrix isn’t dense anymore, it’s a diagonal band of width w, and a real implementation never materializes the zeroed-out region at all.

# scenario: building the causal + sliding-window mask for one
# attention layer, before it's fed to a fused kernel like
# FlashAttention-2's window_size argument
import torch

def sliding_window_mask(seq_len: int, window: int) -> torch.Tensor:
    idx = torch.arange(seq_len)
    i, j = idx.unsqueeze(1), idx.unsqueeze(0)
    causal = j <= i                  # can't see the future
    windowed = j > (i - window)      # can't see further back than `window`
    return causal & windowed         # True = attend, False = masked out

# for i=5000, window=4096: token 5000 attends to tokens 905..5000 only,
# regardless of how many tokens came before position 905
mask = sliding_window_mask(seq_len=8192, window=4096)

In practice, nobody builds this mask densely for long sequences; FlashAttention-2 and vLLM’s paged attention both accept a window_size argument and skip the out-of-window key/value blocks inside the kernel itself, so the O(n·w) saving shows up as real wall-clock speedup and real memory reduction, not just a sparser matrix that still gets computed.

The Receptive Field Argument: Stacking Recovers Range

A single sliding-window layer can only see w tokens back, which sounds like a hard information ceiling. But attention layers stack, and information that reaches a token’s window in layer k becomes part of that token’s representation in layer k+1, whose own window can then reach it again, exactly the way a CNN’s receptive field grows with depth even though every convolution kernel is small. Mistral 7B’s paper makes this concrete: with window w = 4096 and 32 layers, a token’s theoretical attention span is roughly layers × window ≈ 131K tokens, even though no single layer ever looks back further than 4096 positions.

graph TB
    classDef default fill:#ffffff,stroke:#4338CA,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef data    fill:#EEF0F7,stroke:#0D9488,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef process fill:#F7F8FC,stroke:#6366F1,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef output  fill:#4338CA,stroke:#4338CA,stroke-width:2px,color:#ffffff,rx:8px,ry:8px;

    T0([Token at position i, layer 0]):::data --> W1["Layer 1 window:<br/>sees i-w..i"]:::process
    W1 --> W2["Layer 2 window:<br/>sees info already<br/>gathered from i-2w..i"]:::process
    W2 --> W3["Layer 3 window:<br/>sees info already<br/>gathered from i-3w..i"]:::process
    W3 --> OUT(["Effective range after L layers<br/>≈ L × w"]):::output

That’s a real, if theoretical, receptive field, not a guarantee that a specific fact 100K tokens back actually propagates cleanly through 32 rounds of compression. In practice this is why almost no production model relies on stacking alone: they mix in a small number of full-attention layers to give the model a direct, single-hop path to any position, rather than trusting every long-range dependency to survive dozens of local hops.

Interleaving Local and Global Layers

The dominant 2024–2026 pattern is to alternate: most layers use a cheap local window, a minority use full, unrestricted attention. The ratio and window size are both tunable per model, and they’ve been trending toward smaller windows and sparser global layers over time as labs push harder on KV cache reduction.

Gemma 3’s team reported that shrinking the local window from 4096 down to 1024 tokens cost only a minor perplexity penalty, while sharply cutting how much of the 128K-token context the majority of layers ever need to keep cached, which is the whole point: only the global layers, a small fraction of the network, have to hold keys and values for the full context length.

Try It: Attention Cost and Effective Range vs. Window Size

The chart below plots exactly what the banded mask actually costs and reaches. Attention FLOPs per layer scale with n × w instead of n², and the shaded region on the right shows how far back a token’s information can theoretically travel after stacking layers, layers × w, against the actual sequence length. Drag the window size down and watch the compute curve flatten while the “full context reached” marker moves.

Drag w down and n up together, the way real long-context serving does, and watch compute reduction climb into the hundreds while the average per-layer KV cache (weighted across local and global layers) stays far below the sequence length.

What Sliding Window Attention Costs

The trade is real, not free. A token genuinely cannot attend directly to something outside its layer’s window; whatever reaches it has to arrive indirectly, compressed through however many intermediate layers of local hops it takes to travel that far. For tasks that need precise retrieval of a specific fact from deep in a long context, “needle in a haystack”-style evaluations, pure stacking without any global layers tends to underperform full attention, which is exactly why the interleaving pattern above exists: a small number of full-attention layers give the model at least one direct path to anywhere in the context, while the local layers carry the bulk of the sequence-length cost.

Combining SWA with GQA

Sliding window attention and Grouped Query Attention target two independent axes of the same KV cache problem and stack cleanly: GQA shrinks the width of what gets cached per token (fewer distinct key/value heads), while SWA shrinks the number of tokens that need caching per layer (a bounded window instead of the whole sequence). Mistral 7B was the first widely-used model to ship both together, and it’s now the norm rather than the exception.

# scenario: estimating KV cache size for a model that combines
# GQA (fewer KV heads) and SWA (bounded window) simultaneously,
# vs. a hypothetical full-MHA, full-attention baseline
def kv_cache_bytes(layers, kv_heads, head_dim, cached_tokens, bytes_per_elem=2):
    return 2 * layers * kv_heads * head_dim * cached_tokens * bytes_per_elem

# Mistral-7B-shaped: 32 layers, 8 KV heads (GQA), window=4096 (SWA)
gqa_swa = kv_cache_bytes(layers=32, kv_heads=8, head_dim=128, cached_tokens=4096)

# same shape, but 32 query heads (no GQA) and full 32K attention (no SWA)
full_mha_full_attn = kv_cache_bytes(layers=32, kv_heads=32, head_dim=128, cached_tokens=32768)

print(full_mha_full_attn / gqa_swa)  # roughly 32x smaller cache, combined

What’s New (2025–2026)

• Shrinking windows, sparser global layers. The trend from Gemma 2 to Gemma 3 (4096 to 1024) and GPT-OSS’s 128-token window shows labs pushing the local window much smaller than Mistral 7B’s original 4096, betting that a thin band plus infrequent global layers loses less than it saves.
• Learned attention sinks. GPT-OSS pairs its 128-token sliding window with a learned per-head attention sink, a small set of always-visible positions that keep very-short local windows from destabilizing, a technique that traces back to StreamingLLM-style streaming attention research.
• Training-time window curricula. Rather than fixing the window at one size for the model’s whole life, some 2025-era work trains with a shrinking or scheduled window, aiming to get windowed attention’s efficiency without the quality gap paying for it all at once during training.
• SWA as a hybrid-architecture bridge. As state-space and linear-attention hybrids like Mamba mix recurrent layers with a handful of attention layers for precise recall, several of those designs use sliding window rather than full attention for the attention layers they do keep, treating SWA as the efficient default even outside pure transformer stacks.

from transformers import MistralConfig, MistralForCausalLM

# Mistral 7B's original recipe: 32 layers, window=4096.
# Stacking layers gives a theoretical receptive field of roughly
# num_hidden_layers * sliding_window =~ 131K tokens, even though
# any single layer only ever looks back 4096 positions.
config = MistralConfig(
    hidden_size=4096,
    num_hidden_layers=32,
    num_attention_heads=32,
    num_key_value_heads=8,        # GQA, paired with SWA in the same model
    sliding_window=4096,
    max_position_embeddings=32768,
)
model = MistralForCausalLM(config)