← All problems
#116 DiT Block Hard

Implement a single DiT (Diffusion Transformer) block, the layer that Stable Diffusion 3, Flux, and the original DiT stack dozens of times to denoise image latents. Given a batch of patch token sequences \(x\) of shape (batch_size, seq_len, 512), a per-sample conditioning vector \(c\) of shape (batch_size, 512) (the timestep plus class/text embedding), and a packed weight buffer, compute the block output. Unlike a language model block, normalization here has no learned affine parameters: the scale, shift, and residual gate are predicted from \(c\) by an adaLN-Zero modulation layer, so every sample in the batch is normalized with different parameters.

c (batch, 512) SiLU Linear 512 → 3072 shift_msa scale_msa gate_msa shift_mlp scale_mlp gate_mlp x (batch, seq_len, 512) LayerNorm (no affine) Modulate (scale, shift) QKV Projection Self-Attention (8 heads) Output Projection ⊙ + LayerNorm (no affine) Modulate (scale, shift) GELU(Linear 512→2048) Linear 2048 → 512 ⊙ + output (batch, seq_len, 512)

The six modulation vectors are produced once per sample and broadcast across all seq_len tokens of that sample:

\[ \begin{aligned} \left[\gamma_{\text{msa}} \mid \alpha_{\text{msa}} \mid g_{\text{msa}} \mid \gamma_{\text{mlp}} \mid \alpha_{\text{mlp}} \mid g_{\text{mlp}}\right] &= \text{SiLU}(c)\, W_{\text{ada}}^\top + b_{\text{ada}} \in \mathbb{R}^{B \times 3072} \end{aligned} \]

where each of the six pieces is a contiguous slice of 512 values (in the order shown) and \(\text{SiLU}(z) = z \cdot \sigma(z)\). The block itself is then:

\[ \begin{aligned} h &= \text{LN}(x) \odot (1 + \alpha_{\text{msa}}) + \gamma_{\text{msa}} \\[4pt] x' &= x + g_{\text{msa}} \odot \text{Attn}(h) \\[4pt] h' &= \text{LN}(x') \odot (1 + \alpha_{\text{mlp}}) + \gamma_{\text{mlp}} \\[4pt] \text{output} &= x' + g_{\text{mlp}} \odot \text{MLP}(h') \end{aligned} \]

with the sub-operations defined as:

\[ \begin{aligned} \text{LN}(z) &= \frac{z - \mu}{\sqrt{\sigma^2 + \varepsilon}}, \quad \mu = \frac{1}{d}\sum_i z_i, \quad \sigma^2 = \frac{1}{d}\sum_i (z_i - \mu)^2, \quad \varepsilon = 10^{-6} \\[8pt] [Q \mid K \mid V] &= h\, W_{qkv}^\top + b_{qkv}, \quad \text{each reshaped to } (B, T, 8, 64) \\[4pt] \text{head}_i &= \text{softmax}\!\left(\frac{Q_i K_i^\top}{\sqrt{64}}\right) V_i \quad \text{(no causal mask)} \\[4pt] \text{Attn}(h) &= \text{Concat}(\text{head}_1, \ldots, \text{head}_8)\, W_o^\top + b_o \\[8pt] \text{MLP}(h') &= \text{GELU}\!\left(h'\, W_{fc1}^\top + b_{fc1}\right) W_{fc2}^\top + b_{fc2} \end{aligned} \]

Attention, normalization, and the MLP are applied independently per batch sample; tokens never mix across samples.

Implementation Requirements

Weight Layout

All block parameters are packed into a single contiguous weights buffer (4,726,272 floats) in the order below. All 2-D matrices are stored row-major with shape (out_dim, in_dim), so \(W_{qkv}[i][j]\) lives at weights[1575936 + i * 512 + j].

Parameter Shape Size Offset
\(W_{\text{ada}}\) (adaLN modulation) (3072, 512) 1,572,864 0
\(b_{\text{ada}}\) (3072,) 3,072 1,572,864
\(W_{qkv}\) (1536, 512) 786,432 1,575,936
\(b_{qkv}\) (1536,) 1,536 2,362,368
\(W_o\) (512, 512) 262,144 2,363,904
\(b_o\) (512,) 512 2,626,048
\(W_{fc1}\) (2048, 512) 1,048,576 2,626,560
\(b_{fc1}\) (2048,) 2,048 3,675,136
\(W_{fc2}\) (512, 2048) 1,048,576 3,677,184
\(b_{fc2}\) (512,) 512 4,725,760

Example

With batch_size = 1, seq_len = 4, x drawn uniformly from [−1, 1], c drawn uniformly from [−2, 2], and randomly initialized weights:

Input:  x.shape       = (1, 4, 512)     # 4 patch tokens
        c.shape       = (1, 512)        # timestep + class conditioning
        weights.shape = (4,726,272,)    # packed weight buffer
        batch_size    = 1
        seq_len       = 4
Output: output.shape  = (1, 4, 512)     # denoised patch tokens

Constraints

Guided walkthrough coming soon. Below is the official starter signature in each language; the clickable line-by-line explanation hasn't been authored yet.
Problem text & starters from the official AlphaGPU/leetgpu-challenges. Solve it on leetgpu.com.