Implement one decode step of Multi-Head Latent Attention (MLA), the attention variant used by
DeepSeek-V2/V3. Instead of caching a full key and value vector per head, MLA caches a single
low-rank latent vector c of size kv_lora_rank per position, shared by
every head, plus a small rotary key k_pe of size rope_dim that is also
shared. Keys and values are reconstructed on the fly from the latent through the per-head
up-projections W_UK and W_UV.
Materializing every key and value would defeat the purpose of the compressed cache, so production
kernels use weight absorption: fold W_UK into the query and run the whole
attention in latent space. For a single decode query with per-head query
q = [q_nope | q_pe] and cache row kv_cache[s] = [c[s] | k_pe[s]]:
\[ \tilde{q}_h = q\_nope_h W\_UK_h \qquad s_{h,t} = \frac{\tilde{q}_h \cdot c_t + q\_pe_h \cdot k\_pe_t}{\sqrt{head\_dim + rope\_dim}} \] \[ a_h = \text{softmax}(s_h) \qquad o_h = \left( \sum_{t=0}^{seq\_len-1} a_{h,t}\, c_t \right) W\_UV_h \]
Note that c plays the role of both key and value, and that the rotary term uses the
same k_pe for every head. All tensors are float32.
solve(q, kv_cache, W_UK, W_UV, output, num_heads, seq_len, kv_lora_rank, head_dim, rope_dim).q has shape (num_heads, head_dim + rope_dim): the first head_dim entries of each row are q_nope, the remaining rope_dim entries are q_pe.kv_cache has shape (seq_len, kv_lora_rank + rope_dim): the first kv_lora_rank entries of each row are the latent c, the remaining rope_dim entries are k_pe.W_UK has shape (num_heads, head_dim, kv_lora_rank) and W_UV has shape (num_heads, kv_lora_rank, head_dim).1 / sqrt(head_dim + rope_dim) and a softmax over the seq_len cached positions.output buffer of shape (num_heads, head_dim).
With num_heads = 2, seq_len = 2, kv_lora_rank = 2,
head_dim = 2, rope_dim = 1:
\(q\) (2×3, last column is q_pe):
\[
\begin{bmatrix}
1 & 0 & 1 \\
0 & 1 & 0
\end{bmatrix}
\qquad
kv\_cache\text{ (2}\times\text{3, last column is } k\_pe\text{):}
\begin{bmatrix}
1 & 0 & 0 \\
0 & 1 & 1
\end{bmatrix}
\]
\(W\_UK_0\) (2×2):
\[
\begin{bmatrix}
1 & 0 \\
0 & 1
\end{bmatrix}
\qquad
W\_UK_1\text{ (2}\times\text{2):}
\begin{bmatrix}
0 & 1 \\
1 & 0
\end{bmatrix}
\]
\(W\_UV_0\) (2×2):
\[
\begin{bmatrix}
1 & 2 \\
3 & 4
\end{bmatrix}
\qquad
W\_UV_1\text{ (2}\times\text{2):}
\begin{bmatrix}
1 & 0 \\
0 & 1
\end{bmatrix}
\]
Output (2×2, values rounded to 4 decimals): \[ \begin{bmatrix} 2.0000 & 3.0000 \\ 0.6405 & 0.3595 \end{bmatrix} \]
Head 0 scores are equal, so its attention is uniform and the latent result is
[0.5, 0.5]. Head 1 sees scores [0.5774, 0], giving weights
[0.6405, 0.3595], which pass through the identity W_UV unchanged.
num_heads ≤ 128seq_len ≤ 8,192kv_lora_rank ≤ 512head_dim ≤ 128rope_dim ≤ 64float32num_heads = 128, seq_len = 4,096, kv_lora_rank = 512, head_dim = 128, rope_dim = 64