← All problems
#93 Llama Transformer Block Hard

Implement a single Llama-style transformer decoder block. Given an input tensor \(x\) of shape (seq_len, 512), a packed weight buffer, and precomputed RoPE tables, compute the output using pre-norm architecture with Grouped Query Attention (GQA), Rotary Position Embeddings (RoPE), and a SwiGLU feed-forward network.

x (seq_len, 512) residual RMSNorm 1 QKV Projection (GQA) RoPE (Q and K) Causal Attention Output Projection + residual RMSNorm 2 Gate & Up Proj (512→1408) SiLU(gate) ⊙ up Down Proj (1408→512) + output (seq_len, 512)

The block follows Llama's pre-norm architecture. Unlike GPT-2, it uses RMSNorm (no mean subtraction, no additive bias), Grouped Query Attention with 8 query heads and 2 key/value heads, Rotary Position Embeddings applied to Q and K, and a SwiGLU feed-forward network. None of the linear projections have bias terms.

\[ \begin{aligned} x' &= x + \text{Attn}\!\left(\text{RMSNorm}_1(x),\; \cos,\; \sin\right) \\[4pt] \text{output} &= x' + \text{FFN}\!\left(\text{RMSNorm}_2(x')\right) \end{aligned} \]

The sub-operations in detail:

\[ \begin{aligned} \text{RMSNorm}(z, w) &= \frac{z}{\sqrt{\frac{1}{d}\sum_i z_i^2 + \varepsilon}} \odot w, \quad \varepsilon = 10^{-5} \\[8pt] Q &= \text{RMSNorm}_1(x)\, W_Q^\top \in \mathbb{R}^{T \times 512}, \quad \text{reshape to } (T, 8, 64) \\[4pt] K &= \text{RMSNorm}_1(x)\, W_K^\top \in \mathbb{R}^{T \times 128}, \quad \text{reshape to } (T, 2, 64) \\[4pt] V &= \text{RMSNorm}_1(x)\, W_V^\top \in \mathbb{R}^{T \times 128}, \quad \text{reshape to } (T, 2, 64) \\[8pt] \text{RoPE}(q, \cos, \sin) &: \quad [q_1 \mid q_2] \mapsto [q_1 \odot \cos - q_2 \odot \sin \mid q_1 \odot \sin + q_2 \odot \cos] \\[4pt] &\quad q_1 = q[\ldots, {:}32],\; q_2 = q[\ldots, {32:}] \\[8pt] \text{GQA} &: \text{repeat } K,V \text{ along head dim } 4\times \text{ to match 8 Q heads} \\[4pt] \text{head}_i &= \text{softmax}\!\left(\frac{Q_i K_i^\top}{\sqrt{64}} + M_{\text{causal}}\right) V_i \\[8pt] \text{Attn}(x) &= \text{Concat}(\text{head}_1, \ldots, \text{head}_8)\; W_O^\top \\[8pt] \text{FFN}(z) &= \bigl(\text{SiLU}(z\, W_{\text{gate}}^\top) \odot z\, W_{\text{up}}^\top\bigr)\; W_{\text{down}}^\top \end{aligned} \]

where \(M_{\text{causal}}\) is the upper-triangular causal mask (\(-\infty\) above the diagonal) and \(\text{SiLU}(x) = x \cdot \sigma(x)\).

Implementation Requirements

Weight Layout

All parameters are packed into a single contiguous weights buffer (2,819,072 floats) in the order below. All 2-D matrices are stored row-major with shape (out_dim, in_dim). There are no bias terms.

Parameter Shape Size Offset
\(w_1\) (RMSNorm 1 scale) (512,) 512 0
\(W_Q\) (512, 512) 262,144 512
\(W_K\) (128, 512) 65,536 262,656
\(W_V\) (128, 512) 65,536 328,192
\(W_O\) (512, 512) 262,144 393,728
\(w_2\) (RMSNorm 2 scale) (512,) 512 655,872
\(W_{\text{gate}}\) (1408, 512) 720,896 656,384
\(W_{\text{up}}\) (1408, 512) 720,896 1,377,280
\(W_{\text{down}}\) (512, 1408) 720,896 2,098,176

Example

With seq_len = 4, x drawn uniformly from [−1, 1], and randomly initialized weights:

Input:  x.shape       = (4, 512)       # 4 token hidden states
        weights.shape = (2,819,072,)   # packed weight buffer
        cos.shape     = (4, 32)        # precomputed RoPE cosines
        sin.shape     = (4, 32)        # precomputed RoPE sines
        seq_len       = 4
Output: output.shape  = (4, 512)       # transformed token hidden states

Constraints

Guided walkthrough coming soon. Below is the official starter signature in each language; the clickable line-by-line explanation hasn't been authored yet.
Problem text & starters from the official AlphaGPU/leetgpu-challenges. Solve it on leetgpu.com.