← All problems
#83 Fused Residual Add RMS Norm Medium

Implement a fused kernel that adds an input tensor x to a residual tensor and then RMS-normalizes the result, scaling each feature by a learned per-feature weight vector. Inputs are float32 tensors: x and residual have shape (N, C) where N is the number of tokens and C is the hidden dimension, and weight has shape (C,). This pattern appears before every sublayer in modern LLMs such as LLaMA and Mistral: fusing the addition and normalization into a single kernel saves a full read–write pass over the data compared to two separate operations.

x residual + z = x + residual RMS Norm × weight out

For each row i (token), the kernel must compute:

\[ z_i = x_i + \text{residual}_i, \quad \text{rms}_i = \sqrt{\frac{1}{C}\sum_{j=0}^{C-1} z_{i,j}^2 + \varepsilon}, \quad \text{out}_{i,j} = \frac{z_{i,j}}{\text{rms}_i} \cdot \text{weight}_j \]

Each row's RMS is computed independently, making this a per-token normalization. The challenge is to fuse the addition and normalization into a single GPU kernel, avoiding writing the intermediate sum z to global memory.

Implementation Requirements

Example

N = 1, C = 4
x        = [1.0,  0.0, -1.0,  2.0]
residual = [0.5,  1.5,  0.5, -0.5]
weight   = [1.0,  1.0,  1.0,  1.0]
eps      = 1e-5

z   = [1.5,  1.5, -0.5,  1.5]
rms = sqrt((1.5^2 + 1.5^2 + 0.5^2 + 1.5^2) / 4 + 1e-5)
    = sqrt(1.75 + 1e-5) ≈ 1.32288

out ≈ [1.1339,  1.1339, -0.3780,  1.1339]

Constraints

Guided walkthrough coming soon. Below is the official starter signature in each language; the clickable line-by-line explanation hasn't been authored yet.
Problem text & starters from the official AlphaGPU/leetgpu-challenges. Solve it on leetgpu.com.