Implement a GPU program that computes the Rotary Positional Embedding (RoPE) for a batch of query vectors. RoPE is a method for encoding positional information in transformer models by rotating the query and key vectors using precomputed cosine and sine components.
Mathematically, given a query vector \(x\) and corresponding cosine and sine vectors, the operation is defined as: \[ \text{RoPE}(x) = x \odot \cos + \text{rotate\_half}(x) \odot \sin \]
Where \(\odot\) denotes element-wise multiplication. The \(\text{rotate\_half}(x)\) operation swaps the first and second halves of the vector and negates the first half. For a vector of dimension \(d\): \[ \text{rotate\_half}([x_1, \dots, x_{d/2}, x_{d/2+1}, \dots, x_d]) = [-x_{d/2+1}, \dots, -x_d, x_1, \dots, x_{d/2}] \]
solve function signature must remain unchangedQ, cos, and sin have shape (M, D), where M is the number of tokens and D is the head dimensioncos and sin use half-split layout: values at dimensions j and j + D/2 are identical for each rowD (head dimension) is guaranteed to be an even number(M, D)Input: Q = [[1.0, 2.0, 3.0, 4.0],
[1.0, 1.0, 1.0, 1.0]]
Cos = [[1.0, 1.0, 1.0, 1.0],
[0.0, 0.0, 0.0, 0.0]]
Sin = [[0.0, 0.0, 0.0, 0.0],
[1.0, 1.0, 1.0, 1.0]]
Output: result = [[1.0, 2.0, 3.0, 4.0],
[-1.0, -1.0, 1.0, 1.0]]
(Row 0 is identity via Cos; Row 1 is rotated via Sin)
Q, cos, and sin have identical dimensionsD % 2 == 0M, D ≤ 10,000D = 128, M = 1,048,576