← All problems
#111 Softmax Attention Backward Medium

Given the query matrix Q of size M×d, key matrix K of size N×d, value matrix V of size N×d, and upstream gradient dO of size M×d, compute the gradients of the softmax attention operation with respect to Q, K, and V. The forward pass computes $$O = \text{softmax}\!\Bigl(\frac{QK^T}{\sqrt{d}}\Bigr)V,$$ where softmax is applied row-wise. The backward pass produces dQ (M×d), dK (N×d), and dV (N×d) via: $$P = \text{softmax}\!\Bigl(\frac{QK^T}{\sqrt{d}}\Bigr), \quad dV = P^T\,dO,$$ $$dP = dO\,V^T, \quad dS_{ij} = P_{ij}\!\left(dP_{ij} - \sum_k P_{ik}\,dP_{ik}\right), \quad \widetilde{dS} = \frac{dS}{\sqrt{d}},$$ $$dQ = \widetilde{dS}\,K, \quad dK = \widetilde{dS}^T Q.$$

Implementation Requirements

Example

Input:
Q (2×4): \[ \begin{bmatrix} 1.0 & 0.0 & 0.0 & 0.0 \\ 0.0 & 1.0 & 0.0 & 0.0 \end{bmatrix} \] K (3×4): \[ \begin{bmatrix} 1.0 & 0.0 & 0.0 & 0.0 \\ 0.0 & 1.0 & 0.0 & 0.0 \\ 0.0 & 0.0 & 1.0 & 0.0 \end{bmatrix} \] V (3×4): \[ \begin{bmatrix} 1.0 & 2.0 & 3.0 & 4.0 \\ 5.0 & 6.0 & 7.0 & 8.0 \\ 9.0 & 10.0 & 11.0 & 12.0 \end{bmatrix} \] dO (2×4): \[ \begin{bmatrix} 1.0 & 0.0 & 0.0 & 0.0 \\ 0.0 & 1.0 & 0.0 & 0.0 \end{bmatrix} \]

Output:
dQ (2×4): \[ \begin{bmatrix} -0.74 & 0.10 & 0.65 & 0.00 \\ -0.55 & 0.00 & 0.55 & 0.00 \end{bmatrix} \] dK (3×4): \[ \begin{bmatrix} -0.74 & -0.55 & 0.00 & 0.00 \\ 0.10 & 0.00 & 0.00 & 0.00 \\ 0.65 & 0.55 & 0.00 & 0.00 \end{bmatrix} \] dV (3×4): \[ \begin{bmatrix} 0.45 & 0.27 & 0.00 & 0.00 \\ 0.27 & 0.45 & 0.00 & 0.00 \\ 0.27 & 0.27 & 0.00 & 0.00 \end{bmatrix} \]

Constraints

Guided walkthrough coming soon. Below is the official starter signature in each language; the clickable line-by-line explanation hasn't been authored yet.
Problem text & starters from the official AlphaGPU/leetgpu-challenges. Solve it on leetgpu.com.