← All problems
#80 Grouped Query Attention Medium

Implement Grouped Query Attention (GQA), the attention mechanism used in modern large language models such as LLaMA-3, Mistral, and Gemma. GQA reduces the KV-cache memory footprint during inference by sharing key and value heads across groups of query heads. Given query tensor Q with num_q_heads heads and key/value tensors K, V each with num_kv_heads heads, compute scaled dot-product attention where every group of num_q_heads / num_kv_heads consecutive query heads attends to the same key and value head. All tensors use float32.

Grouped Query Attention (num_q_heads=4, num_kv_heads=2, groups=2) Q heads Q[0] Q[1] Q[2] Q[3] KV heads K[0], V[0] K[1], V[1] group 0 group 1 Q[0], Q[1] attend to K[0], V[0] Q[2], Q[3] attend to K[1], V[1] scale = 1 / sqrt(head_dim) scores = Q @ K^T * scale weights = softmax(scores) output = weights @ V

Implementation Requirements

Example

With num_q_heads = 4, num_kv_heads = 2 (groups of 2), seq_len = 3, head_dim = 4:

Input:
\(Q_0\) (3×4): \[ \begin{bmatrix} 1 & 0 & 0 & 1 \\ 0 & 1 & 1 & 0 \\ 1 & 1 & 0 & 0 \end{bmatrix} \] \(Q_1\) (3×4): \[ \begin{bmatrix} 0 & 1 & 0 & 1 \\ 1 & 0 & 1 & 0 \\ 0 & 0 & 1 & 1 \end{bmatrix} \] \(Q_2\) (3×4): \[ \begin{bmatrix} -1 & 0 & 0.5 & 0 \\ 0 & -1 & 0 & 0.5 \\ 0.5 & 0 & -1 & 0 \end{bmatrix} \] \(Q_3\) (3×4): \[ \begin{bmatrix} 0 & 0.5 & 0 & -1 \\ 0.5 & 0 & 0 & -1 \\ 0 & 0 & 0.5 & 0.5 \end{bmatrix} \] \(K_0\) (3×4): \[ \begin{bmatrix} 1 & 0 & 1 & 0 \\ 0 & 1 & 0 & 1 \\ 1 & 1 & 1 & 1 \end{bmatrix} \] \(K_1\) (3×4): \[ \begin{bmatrix} 0 & 1 & 0 & -1 \\ -1 & 0 & 1 & 0 \\ 0 & -1 & 0 & 1 \end{bmatrix} \] \(V_0\) (3×4): \[ \begin{bmatrix} 1 & 2 & 3 & 4 \\ 5 & 6 & 7 & 8 \\ 9 & 10 & 11 & 12 \end{bmatrix} \] \(V_1\) (3×4): \[ \begin{bmatrix} -1 & -2 & -3 & -4 \\ 2 & 3 & 4 & 5 \\ 6 & 7 & 8 & 9 \end{bmatrix} \] Groups: \(Q_0, Q_1 \to K_0, V_0\); \quad \(Q_2, Q_3 \to K_1, V_1\)

Output (values rounded to 2 decimal places):
\(\text{output}_0\) (3×4): \[ \begin{bmatrix} 5.71 & 6.71 & 7.71 & 8.71 \\ 5.71 & 6.71 & 7.71 & 8.71 \\ 5.71 & 6.71 & 7.71 & 8.71 \end{bmatrix} \] \(\text{output}_1\) (3×4): \[ \begin{bmatrix} 6.07 & 7.07 & 8.07 & 9.07 \\ 5.00 & 6.00 & 7.00 & 8.00 \\ 5.71 & 6.71 & 7.71 & 8.71 \end{bmatrix} \] \(\text{output}_2\) (3×4): \[ \begin{bmatrix} 2.24 & 2.76 & 3.27 & 3.79 \\ 3.96 & 4.70 & 5.44 & 6.17 \\ 2.40 & 2.60 & 2.79 & 2.98 \end{bmatrix} \] \(\text{output}_3\) (3×4): \[ \begin{bmatrix} 0.76 & 0.58 & 0.40 & 0.22 \\ 1.17 & 1.08 & 1.00 & 0.91 \\ 2.84 & 3.37 & 3.91 & 4.44 \end{bmatrix} \]

Constraints

Guided walkthrough coming soon. Below is the official starter signature in each language; the clickable line-by-line explanation hasn't been authored yet.
Problem text & starters from the official AlphaGPU/leetgpu-challenges. Solve it on leetgpu.com.