Implement a single DiT (Diffusion Transformer) block, the layer that Stable
Diffusion 3, Flux, and the original DiT stack dozens of times to denoise image latents. Given a
batch of patch token sequences \(x\) of shape (batch_size, seq_len, 512), a
per-sample conditioning vector \(c\) of shape (batch_size, 512) (the timestep plus
class/text embedding), and a packed weight buffer, compute the block output. Unlike a language
model block, normalization here has no learned affine parameters: the scale, shift, and
residual gate are predicted from \(c\) by an adaLN-Zero modulation layer, so
every sample in the batch is normalized with different parameters.
The six modulation vectors are produced once per sample and broadcast across all
seq_len tokens of that sample:
where each of the six pieces is a contiguous slice of 512 values (in the order shown) and \(\text{SiLU}(z) = z \cdot \sigma(z)\). The block itself is then:
\[ \begin{aligned} h &= \text{LN}(x) \odot (1 + \alpha_{\text{msa}}) + \gamma_{\text{msa}} \\[4pt] x' &= x + g_{\text{msa}} \odot \text{Attn}(h) \\[4pt] h' &= \text{LN}(x') \odot (1 + \alpha_{\text{mlp}}) + \gamma_{\text{mlp}} \\[4pt] \text{output} &= x' + g_{\text{mlp}} \odot \text{MLP}(h') \end{aligned} \]with the sub-operations defined as:
\[ \begin{aligned} \text{LN}(z) &= \frac{z - \mu}{\sqrt{\sigma^2 + \varepsilon}}, \quad \mu = \frac{1}{d}\sum_i z_i, \quad \sigma^2 = \frac{1}{d}\sum_i (z_i - \mu)^2, \quad \varepsilon = 10^{-6} \\[8pt] [Q \mid K \mid V] &= h\, W_{qkv}^\top + b_{qkv}, \quad \text{each reshaped to } (B, T, 8, 64) \\[4pt] \text{head}_i &= \text{softmax}\!\left(\frac{Q_i K_i^\top}{\sqrt{64}}\right) V_i \quad \text{(no causal mask)} \\[4pt] \text{Attn}(h) &= \text{Concat}(\text{head}_1, \ldots, \text{head}_8)\, W_o^\top + b_o \\[8pt] \text{MLP}(h') &= \text{GELU}\!\left(h'\, W_{fc1}^\top + b_{fc1}\right) W_{fc2}^\top + b_{fc2} \end{aligned} \]Attention, normalization, and the MLP are applied independently per batch sample; tokens never mix across samples.
solve function signature must remain unchangedoutput tensorAll block parameters are packed into a single contiguous weights buffer
(4,726,272 floats) in the order below. All 2-D matrices are stored row-major with shape
(out_dim, in_dim), so \(W_{qkv}[i][j]\) lives at
weights[1575936 + i * 512 + j].
| Parameter | Shape | Size | Offset |
|---|---|---|---|
| \(W_{\text{ada}}\) (adaLN modulation) | (3072, 512) | 1,572,864 | 0 |
| \(b_{\text{ada}}\) | (3072,) | 3,072 | 1,572,864 |
| \(W_{qkv}\) | (1536, 512) | 786,432 | 1,575,936 |
| \(b_{qkv}\) | (1536,) | 1,536 | 2,362,368 |
| \(W_o\) | (512, 512) | 262,144 | 2,363,904 |
| \(b_o\) | (512,) | 512 | 2,626,048 |
| \(W_{fc1}\) | (2048, 512) | 1,048,576 | 2,626,560 |
| \(b_{fc1}\) | (2048,) | 2,048 | 3,675,136 |
| \(W_{fc2}\) | (512, 2048) | 1,048,576 | 3,677,184 |
| \(b_{fc2}\) | (512,) | 512 | 4,725,760 |
With batch_size = 1, seq_len = 4, x drawn uniformly from
[−1, 1], c drawn uniformly from [−2, 2], and randomly initialized
weights:
Input: x.shape = (1, 4, 512) # 4 patch tokens
c.shape = (1, 512) # timestep + class conditioning
weights.shape = (4,726,272,) # packed weight buffer
batch_size = 1
seq_len = 4
Output: output.shape = (1, 4, 512) # denoised patch tokens
d_model = 512, n_heads = 8, head_dim = 64,
mlp_hidden = 2,048batch_size ≤ 16seq_len ≤ 4,096batch_size = 4, seq_len = 1,024