Mamba in vision, seen as linear attention (MLLA)
Demystify Mamba in Vision: A Linear Attention Perspective (the MLLA paper) is the “wait, what is Mamba actually doing?” paper. it rewrites the selective SSM and linear attention in one unified formulation, declares Mamba a linear-attention transformer with six distinctions, then ablates them. verdict: the forget gate and the block design are the core contributors; the other four barely matter. this note is my walkthrough of the paper + repo. the SSM side lives in my Mamba notes.
linear attention in one move
softmax attention’s quadratic cost is brutal on high-resolution images. linear attention attacks the bottleneck at the softmax: replace the non-linear softmax with linear normalization, apply a kernel $\phi$ to $Q$ and $K$:
\[\text{Attn}(Q, K, V) = \frac{\phi(Q)\left(\phi(K)^\top V\right)}{\phi(Q)\phi(K)^\top \mathbb{1}_N} \qquad \text{instead of } \text{softmax}(QK^\top)V\]
the trick is the computation order: $\left(\phi(Q)\phi(K)^\top\right)V$ is $O(N^2)$ materialization, but regrouping to $\phi(Q)\left(\phi(K)^\top V\right)$ is $O(N)$ in sequence length.

in an autoregressive setting the receptive field just gets restricted to $j \le i$ - causal linear attention, same regrouping.
the selective SSM, written the same way

for input $x \in \mathbb{R}^{N\times C}$, Mamba applies its recurrence per-channel:


one formulation, six distinctions

the equivalence, in one line:
selective SSM = linear attention + input gate + forget gate + shortcut, minus attention normalization, minus multi-head, plus a modified block design.
- input gate - the step size $\Delta_i$, computed as $\Delta_i = \text{softplus}(x_i W_1 W_2)$, gates how much of the current token $x_i$ gets into the hidden state. the model learns per-token “utility”: big $\Delta_i$ for useful tokens, small for filler.
- forget gate - $\tilde{A}$ (henceforth $A$), decays the previous hidden state as each new token arrives.
- shortcut - the $Dx$ term, skipping state entirely.
- no attention normalization - linear attention’s denominator is dropped; there’s nothing forcing the attention weights to sum to anything.
- single-head - linear attention usually runs many heads/subspaces; the SSM recurrence is one head per channel.
- modified block design - the Mamba-block shape itself (SSM notes has it).
the forget gate carries the baggage
the forget gate has two properties that do a lot of work:
- local bias: every element of $A$ is strictly in $(0,1)$, so the previous hidden state always decays when a token arrives. recency is structurally baked in.
- positional information: because decay compounds over steps, the model is sensitive to token order - that’s positional encoding for free.

the authors’ tension point: to use $A$ on images you first flatten the image into a 1D sequence and run a recurrence per token - the receptive field shrinks to preceding tokens only, and you pay extra latency. and images are not causal data. their speculation, which is the paper’s best idea: a positional encoding that provides local bias + order information can substitute for the forget gate. LePE and RoPE both qualify.

and it works: the forget gate can be replaced with RoPE. (the experiments swap $A$ for RoPE-style encoding and keep the results - the “must be a recurrence” intuition for SSMs takes a hit.)

reading this right after S4 notes reframes it nicely: S4’s whole identity was LTI (that’s what Mamba removed), and here the selection mechanism itself turns out to be optional plumbing over “linear attention + positional bias”. the SSM is becoming an implementation strategy, not an essence.
training + what MESA is (and isn’t, in this note)
the setup: AdamW from scratch, 300 epochs, cosine decay with 20-epoch warmup, weight decay 0.05, total batch 4096, lr $4\times10^{-3}$. RandAugment / Mixup / CutMix / random erasing. UPerNet for segmentation with Swin’s training settings.


plus one extra: MESA - an overfitting-prevention strategy the authors apply when training the MLLA models, because vision-Mamba training protocols aren’t as settled as ViT ones. the ablation says MESA adds 0.1-0.3 accuracy and MLLA beats vision-Mamba SOTA with or without it. what MESA is mechanically, my notes never say - see todos.
architecture + the takeaway


Mamba can be seen as a variant of linear attention transformer with specialized linear attention and a modified block design. once you hold that, Mamba stops being exotic: it’s one point in the linear-attention family tree - which also contains the FLatten transformer with its focused linear attention. same $O(N)$ goal, different fixes for linear attention’s known weaknesses (smooth attention distributions, low-rank).
papers & sources
todos
- MESA: pull the mechanism out of the paper’s reference [11] (overfitting prevention via ??? - my notes never say; i got it wrong twice while guessing arXiv IDs, so look it up properly).
- write out the paper’s unified-equation derivation under “one formulation, six distinctions” (the Eq. 9-10 step-by-step inside the screenshot above is the load-bearing math).
- put actual ImageNet/UPerNet score tables next to the screenshots with my own takeaways - right now the numbers live only in images.