← All problems
#84 SwiGLU MLP Block Medium

Implement the SwiGLU MLP block — the feedforward network used in LLaMA, Mistral, Gemma, and most modern large language models. Given an input matrix x of shape [M, d_model] and three weight matrices W_gate, W_up (each [d_model, d_ffn]), and W_down ([d_ffn, d_model]), compute: output = (SiLU(x × W_gate) ⊙ (x × W_up)) × W_down, where SiLU(z) = z × sigmoid(z) and ⊙ denotes element-wise multiplication. All tensors are float32.

x [M, d_model] x · W_gate gate projection x · W_up up projection [M, d_ffn] [M, d_ffn] SiLU ⊙ · W_down [M, d_ffn] output [M, d_model] z · sigmoid(z)

Implementation Requirements

Example

Input: M = 2, d_model = 2, d_ffn = 4

\(x\) (float32, \(2 \times 2\)): \[ x = \begin{bmatrix} 1.0 & 0.0 \\ 0.0 & 1.0 \end{bmatrix} \] \(W_\text{gate}\) and \(W_\text{up}\) (both \(2 \times 4\)): \[ W_\text{gate} = W_\text{up} = \begin{bmatrix} 1.0 & 0.0 & 0.0 & 0.0 \\ 0.0 & 1.0 & 0.0 & 0.0 \end{bmatrix} \] \(W_\text{down}\) (\(4 \times 2\)): \[ W_\text{down} = \begin{bmatrix} 1.0 & 0.0 \\ 0.0 & 1.0 \\ 0.0 & 0.0 \\ 0.0 & 0.0 \end{bmatrix} \]

Intermediate steps: \[ \text{gate} = x \cdot W_\text{gate} = \begin{bmatrix} 1.0 & 0.0 & 0.0 & 0.0 \\ 0.0 & 1.0 & 0.0 & 0.0 \end{bmatrix} \] \[ \text{up} = x \cdot W_\text{up} = \begin{bmatrix} 1.0 & 0.0 & 0.0 & 0.0 \\ 0.0 & 1.0 & 0.0 & 0.0 \end{bmatrix} \] \[ \text{SiLU}(1.0) = 1.0 \times \sigma(1.0) \approx 0.7311 \] \[ \text{hidden} = \text{SiLU}(\text{gate}) \odot \text{up} = \begin{bmatrix} 0.7311 & 0.0 & 0.0 & 0.0 \\ 0.0 & 0.7311 & 0.0 & 0.0 \end{bmatrix} \]

Output: \[ \text{output} = \text{hidden} \cdot W_\text{down} \approx \begin{bmatrix} 0.7311 & 0.0 \\ 0.0 & 0.7311 \end{bmatrix} \]

Constraints

Guided walkthrough coming soon. Below is the official starter signature in each language; the clickable line-by-line explanation hasn't been authored yet.
Problem text & starters from the official AlphaGPU/leetgpu-challenges. Solve it on leetgpu.com.