Implement a single GPT-2 transformer decoder block. Given an input tensor
\(x\) of shape (seq_len, 768) and a packed weight buffer containing
all block parameters, compute the output using pre-norm architecture with
multi-head self-attention and a feed-forward network with GELU activation.
The block uses GPT-2's pre-norm architecture: LayerNorm is applied before each sub-layer (attention and feed-forward), not after. At a high level:
\[ \begin{aligned} x' &= x + \text{MultiHeadAttn}\!\left(\text{LN}_1(x)\right) \\[4pt] \text{output} &= x' + \text{FeedForward}\!\left(\text{LN}_2(x')\right) \end{aligned} \]where the sub-layers are defined as:
\[ \begin{aligned} \text{LN}(z) &= \frac{z - \mu}{\sqrt{\sigma^2 + \epsilon}} \odot \gamma + \beta, \quad \mu = \frac{1}{d}\sum_i z_i, \quad \sigma^2 = \frac{1}{d}\sum_i (z_i - \mu)^2 \\[8pt] [Q \mid K \mid V] &= \text{LN}_1(x) \cdot W_{qkv} + b_{qkv} \\[4pt] \text{head}_i &= \text{softmax}\!\left(\frac{Q_i K_i^\top}{\sqrt{d_k}}\right) V_i, \quad d_k = 64 \\[4pt] \text{MultiHeadAttn}(z) &= \text{Concat}(\text{head}_1, \ldots, \text{head}_{12}) \cdot W_{\text{attn}} + b_{\text{attn}} \\[8pt] \text{FeedForward}(z) &= \text{GELU}\!\left(z \cdot W_{fc} + b_{fc}\right) \cdot W_{\text{proj}} + b_{\text{proj}} \end{aligned} \]Expanding into individual steps:
(seq_len, 768)solve function signature must remain unchangedoutput tensorAll block parameters are packed into a single contiguous weights buffer
(7,087,872 floats) in the following order. Index into the buffer using the offsets below
(e.g. \(W_{qkv}[i][j]\) is at weights[1536 + i * 2304 + j]).
All 2D matrices are stored in row-major order.
| Parameter | Shape | Size | Offset |
|---|---|---|---|
| \(\gamma_1\) (LN1 weight) | (768,) | 768 | 0 |
| \(\beta_1\) (LN1 bias) | (768,) | 768 | 768 |
| \(W_{qkv}\) | (768, 2304) | 1,769,472 | 1,536 |
| \(b_{qkv}\) | (2304,) | 2,304 | 1,771,008 |
| \(W_{\text{attn}}\) | (768, 768) | 589,824 | 1,773,312 |
| \(b_{\text{attn}}\) | (768,) | 768 | 2,363,136 |
| \(\gamma_2\) (LN2 weight) | (768,) | 768 | 2,363,904 |
| \(\beta_2\) (LN2 bias) | (768,) | 768 | 2,364,672 |
| \(W_{fc}\) | (768, 3072) | 2,359,296 | 2,365,440 |
| \(b_{fc}\) | (3072,) | 3,072 | 4,724,736 |
| \(W_{\text{proj}}\) | (3072, 768) | 2,359,296 | 4,727,808 |
| \(b_{\text{proj}}\) | (768,) | 768 | 7,087,104 |
With seq_len = 4, x uniformly drawn from [−1, 1], and weights randomly initialized
(see Weight Layout for the packing structure):
Input: x.shape = (4, 768) # 4 token embeddings
weights.shape = (7,087,872,) # packed weight buffer
seq_len = 4
Output: output.shape = (4, 768) # transformed token embeddings
d_model = 768, n_heads = 12, ffn_dim = 3,072 (GPT-2 124M architecture)seq_len ≤ 4,096seq_len = 1,024