โ† All problems
#92 Decaying Causal Attention Medium

Implement decaying causal attention. Given query matrix Q, key matrix K, and value matrix V, each of shape seq_len × d_model, and a scalar decay factor gamma ∈ (0, 1], compute the unnormalized causal attention output where position n attends to all past positions m ≤ n with weight gamman−m:

\[ \text{output}[n] = \sum_{m=0}^{n} \gamma^{n-m} \cdot \frac{Q[n] \cdot K[m]}{\sqrt{d_{\text{model}}}} \cdot V[m] \]

Unlike standard softmax attention, there is no normalization โ€” the weights decay geometrically from the current position backward. This is the parallel form of the Retention mechanism (RetNet), used as a recurrence-friendly alternative to attention in sequence models.

Causal Decay Mask D[n,m] = γ^(n−m) m=0 m=1 m=2 m=3 n=0 n=1 n=2 n=3 1 γ 1 γ² γ 1 γ³ γ² γ 1 Computation Q [S, D] query K [S, D] key V [S, D] value QKᵀ / √D attn scores [S,S] ⊙ decay mask weighted [S,S] lower triangular @ output [S, D] weighted @ V

Implementation Requirements

Example

Example 1 โ€” with seq_len = 2, d_model = 4, gamma = 0.5:

\[ Q = \begin{bmatrix} 1 & 1 & 0 & 0 \\ 1 & 1 & 0 & 0 \end{bmatrix}, \quad K = \begin{bmatrix} 1 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 \end{bmatrix}, \quad V = \begin{bmatrix} 4 & 8 & 12 & 16 \\ 4 & 8 & 12 & 16 \end{bmatrix} \]

Attention scores \(QK^\top / \sqrt{4}\): \[ A = \begin{bmatrix} 0.5 & 0.5 \\ 0.5 & 0.5 \end{bmatrix} \] Causal decay mask \(D_{nm} = 0.5^{n-m}\) for \(n \ge m\), else \(0\): \[ D = \begin{bmatrix} 1 & 0 \\ 0.5 & 1 \end{bmatrix} \] Weighted attention \(A \odot D\): \[ \begin{bmatrix} 0.5 & 0 \\ 0.25 & 0.5 \end{bmatrix} \] Output \((A \odot D)\,V\): \[ \text{output} = \begin{bmatrix} 2 & 4 & 6 & 8 \\ 3 & 6 & 9 & 12 \end{bmatrix} \]

Constraints

Guided walkthrough coming soon. Below is the official starter signature in each language; the clickable line-by-line explanation hasn't been authored yet.
Problem text & starters from the official AlphaGPU/leetgpu-challenges. Solve it on leetgpu.com.