← All problems
#106 Token Embedding Layer Medium

Implement the input embedding layer used at the start of transformer models such as BERT. For each token in a batch, gather a row from the token_embeddings table using token_ids, gather a row from the position_embeddings table using position_ids, sum the two vectors, and apply Layer Normalization with learnable scale and shift parameters along the embedding dimension.

Formally, for batch index \(b\) and time step \(t\), let \[ s_{b,t} = E_T[\text{token\_ids}_{b,t}] + E_P[\text{position\_ids}_{t}] \in \mathbb{R}^{D} \] where \(E_T \in \mathbb{R}^{V \times D}\) and \(E_P \in \mathbb{R}^{P \times D}\) are the token and positional embedding tables. The output is then \[ \mu_{b,t} = \frac{1}{D} \sum_{d=1}^{D} s_{b,t,d}, \qquad \sigma^2_{b,t} = \frac{1}{D} \sum_{d=1}^{D} (s_{b,t,d} - \mu_{b,t})^2, \] \[ y_{b,t,d} = \gamma_d \cdot \frac{s_{b,t,d} - \mu_{b,t}}{\sqrt{\sigma^2_{b,t} + \epsilon}} + \beta_d. \]

Implementation Requirements

Example 1:

Input:  B = 1, T = 2, V = 3, P = 2, D = 4, eps = 1e-5
        token_ids        = [[2, 0]]
        position_ids     = [0, 1]
        token_embeddings = [[ 1.0,  2.0,  3.0,  4.0],
                            [ 0.0,  1.0,  0.0, -1.0],
                            [ 2.0,  0.0, -2.0,  0.0]]
        position_embeddings = [[ 0.0,  0.0,  0.0,  0.0],
                               [ 1.0,  0.0, -1.0,  0.0]]
        gamma = [1.0, 1.0, 1.0, 1.0]
        beta  = [0.0, 0.0, 0.0, 0.0]
Output: output = [[[ 1.4142,  0.0000, -1.4142,  0.0000],
                   [-0.5773, -0.5773, -0.5773,  1.7320]]]

Constraints

Guided walkthrough coming soon. Below is the official starter signature in each language; the clickable line-by-line explanation hasn't been authored yet.
Problem text & starters from the official AlphaGPU/leetgpu-challenges. Solve it on leetgpu.com.