Implement the input embedding layer used at the start of transformer models such as BERT.
For each token in a batch, gather a row from the token_embeddings table using
token_ids, gather a row from the position_embeddings table using
position_ids, sum the two vectors, and apply Layer Normalization with learnable
scale and shift parameters along the embedding dimension.
Formally, for batch index \(b\) and time step \(t\), let \[ s_{b,t} = E_T[\text{token\_ids}_{b,t}] + E_P[\text{position\_ids}_{t}] \in \mathbb{R}^{D} \] where \(E_T \in \mathbb{R}^{V \times D}\) and \(E_P \in \mathbb{R}^{P \times D}\) are the token and positional embedding tables. The output is then \[ \mu_{b,t} = \frac{1}{D} \sum_{d=1}^{D} s_{b,t,d}, \qquad \sigma^2_{b,t} = \frac{1}{D} \sum_{d=1}^{D} (s_{b,t,d} - \mu_{b,t})^2, \] \[ y_{b,t,d} = \gamma_d \cdot \frac{s_{b,t,d} - \mu_{b,t}}{\sqrt{\sigma^2_{b,t} + \epsilon}} + \beta_d. \]
solve function signature must remain unchangedtoken_ids has shape (B, T) and position_ids has shape (T), both with int32 valuestoken_embeddings has shape (V, D) and position_embeddings has shape (P, D)gamma and beta each have shape (D)D, not D-1)output tensor with shape (B, T, D)
Input: B = 1, T = 2, V = 3, P = 2, D = 4, eps = 1e-5
token_ids = [[2, 0]]
position_ids = [0, 1]
token_embeddings = [[ 1.0, 2.0, 3.0, 4.0],
[ 0.0, 1.0, 0.0, -1.0],
[ 2.0, 0.0, -2.0, 0.0]]
position_embeddings = [[ 0.0, 0.0, 0.0, 0.0],
[ 1.0, 0.0, -1.0, 0.0]]
gamma = [1.0, 1.0, 1.0, 1.0]
beta = [0.0, 0.0, 0.0, 0.0]
Output: output = [[[ 1.4142, 0.0000, -1.4142, 0.0000],
[-0.5773, -0.5773, -0.5773, 1.7320]]]
B ≤ 64T ≤ 1,024V ≤ 50,000P ≤ 4,096D ≤ 1,024token_ids values < Vposition_ids values < Peps = 1e-5B = 32, T = 512, V = 30,000, P = 2,048, D = 768