Given the query matrix Q of size M×d, key matrix K of size
N×d, value matrix V of size N×d, and upstream gradient
dO of size M×d, compute the gradients of the softmax attention operation
with respect to Q, K, and V. The forward pass computes
$$O = \text{softmax}\!\Bigl(\frac{QK^T}{\sqrt{d}}\Bigr)V,$$
where softmax is applied row-wise. The backward pass produces dQ (M×d),
dK (N×d), and dV (N×d) via:
$$P = \text{softmax}\!\Bigl(\frac{QK^T}{\sqrt{d}}\Bigr), \quad dV = P^T\,dO,$$
$$dP = dO\,V^T, \quad dS_{ij} = P_{ij}\!\left(dP_{ij} - \sum_k P_{ik}\,dP_{ik}\right), \quad \widetilde{dS} = \frac{dS}{\sqrt{d}},$$
$$dQ = \widetilde{dS}\,K, \quad dK = \widetilde{dS}^T Q.$$
solve function signature must remain unchangeddQ, dK, and dV
Input:
Q (2×4):
\[
\begin{bmatrix}
1.0 & 0.0 & 0.0 & 0.0 \\
0.0 & 1.0 & 0.0 & 0.0
\end{bmatrix}
\]
K (3×4):
\[
\begin{bmatrix}
1.0 & 0.0 & 0.0 & 0.0 \\
0.0 & 1.0 & 0.0 & 0.0 \\
0.0 & 0.0 & 1.0 & 0.0
\end{bmatrix}
\]
V (3×4):
\[
\begin{bmatrix}
1.0 & 2.0 & 3.0 & 4.0 \\
5.0 & 6.0 & 7.0 & 8.0 \\
9.0 & 10.0 & 11.0 & 12.0
\end{bmatrix}
\]
dO (2×4):
\[
\begin{bmatrix}
1.0 & 0.0 & 0.0 & 0.0 \\
0.0 & 1.0 & 0.0 & 0.0
\end{bmatrix}
\]
Output:
dQ (2×4):
\[
\begin{bmatrix}
-0.74 & 0.10 & 0.65 & 0.00 \\
-0.55 & 0.00 & 0.55 & 0.00
\end{bmatrix}
\]
dK (3×4):
\[
\begin{bmatrix}
-0.74 & -0.55 & 0.00 & 0.00 \\
0.10 & 0.00 & 0.00 & 0.00 \\
0.65 & 0.55 & 0.00 & 0.00
\end{bmatrix}
\]
dV (3×4):
\[
\begin{bmatrix}
0.45 & 0.27 & 0.00 & 0.00 \\
0.27 & 0.45 & 0.00 & 0.00 \\
0.27 & 0.27 & 0.00 & 0.00
\end{bmatrix}
\]
Q and dO are of size M×d; matrices K, V are of size N×dM, N ≤ 100,000d ≤ 128float32M = 8,192, N = 4,096, d = 128