← All problems
#105 Group Normalization Medium

Implement Group Normalization for 4D activation tensors, the normalization layer used by Stable Diffusion U-Nets and many ResNet variants. Given an input tensor X of shape (N, C, H, W), the channels are split into G contiguous groups of C/G channels each. For every (batch, group) pair, the mean and variance are computed over all (C/G) × H × W elements, the activations are normalized, then scaled and shifted by per-channel parameters gamma and beta.

For each batch index n and group index g, let \( \mathcal{S}_{n,g} = \{(n, c, h, w) : c \in [g \cdot C/G,\, (g+1) \cdot C/G)\} \). Group Normalization computes: \[ \begin{align} \mu_{n,g} &= \frac{1}{|\mathcal{S}_{n,g}|} \sum_{(n,c,h,w) \in \mathcal{S}_{n,g}} x_{n,c,h,w} \\ \sigma_{n,g}^2 &= \frac{1}{|\mathcal{S}_{n,g}|} \sum_{(n,c,h,w) \in \mathcal{S}_{n,g}} (x_{n,c,h,w} - \mu_{n,g})^2 \\ \hat{x}_{n,c,h,w} &= \frac{x_{n,c,h,w} - \mu_{n,g(c)}}{\sqrt{\sigma_{n,g(c)}^2 + \epsilon}} \\ y_{n,c,h,w} &= \gamma_c \, \hat{x}_{n,c,h,w} + \beta_c \end{align} \] where \( g(c) = \lfloor c \cdot G / C \rfloor \) maps a channel to its group.

Channels split into G groups (here C=8, G=2) group 0 c0 c1 c2 c3 μ, σ² over (C/G)×H×W group 1 c4 c5 c6 c7 μ, σ² over (C/G)×H×W Each group is normalized independently per batch element, then scaled and shifted by per-channel gamma and beta. G=1 reduces to Layer Norm; G=C reduces to Instance Norm.

Implementation Requirements

Example 1:

Input:  N=1, C=4, H=2, W=2, G=2, eps=1e-5
        X[0,0] = [[1, 1], [1, 1]]
        X[0,1] = [[3, 3], [3, 3]]
        X[0,2] = [[2, 2], [2, 2]]
        X[0,3] = [[6, 6], [6, 6]]
        gamma = [1, 1, 1, 1]
        beta  = [0, 0, 0, 0]
Output: Y[0,0] = [[-1, -1], [-1, -1]]
        Y[0,1] = [[ 1,  1], [ 1,  1]]
        Y[0,2] = [[-1, -1], [-1, -1]]
        Y[0,3] = [[ 1,  1], [ 1,  1]]
Note:   Group 0 = channels {0, 1}: mean = 2, var = 1, std = 1
        Group 1 = channels {2, 3}: mean = 4, var = 4, std = 2

Example 2:

Input:  N=1, C=2, H=1, W=2, G=2, eps=1e-5
        X[0,0] = [[1, 3]]
        X[0,1] = [[2, 6]]
        gamma = [2, 1]
        beta  = [0, 0]
Output: Y[0,0] = [[-2,  2]]
        Y[0,1] = [[-1,  1]]
Note:   G=C, so each channel is its own group (Instance Norm).
        Channel 0: mean=2, var=1, std=1
        Channel 1: mean=4, var=4, std=2

Constraints

Guided walkthrough coming soon. Below is the official starter signature in each language; the clickable line-by-line explanation hasn't been authored yet.
Problem text & starters from the official AlphaGPU/leetgpu-challenges. Solve it on leetgpu.com.