Implement a GPU program that performs Top-K Gating for Mixture of Experts (MoE) models. Given a logit matrix of shape [M, E] where M is the number of tokens and E is the number of experts, identify the k largest values in each row, extract their indices, and apply softmax to get mixing weights.
For each row i, the operation computes: \[ \begin{align} \text{indices}_i, \text{vals}_i &= \text{TopK}(\text{logits}_i, k) \\ \text{vals}_i &= \text{logits}_i[\text{indices}_i] \\ \text{weights}_i &= \text{Softmax}(\text{vals}_i) \end{align} \]
The selected experts must remain ordered by descending logit value, matching the order returned by
topk. The topk_weights array must correspond positionally to
topk_indices in that same order.
solve function signature must remain unchangedtopk_weights and topk_indices arrays
Input:
logits = [[1.0, 2.0, 3.0, 4.0],
[4.0, 3.0, 2.0, 1.0]]
M = 2, E = 4, k = 2
Output:
topk_weights = [[0.7311, 0.2689],
[0.7311, 0.2689]]
topk_indices = [[3, 2],
[0, 1]]
Explanation:
Row 0: Top-2 values are 4.0 and 3.0 at indices 3 and 2.
Softmax([4.0, 3.0]) = [0.7311, 0.2689]
Row 1: Top-2 values are 4.0 and 3.0 at indices 0 and 1.
Softmax([4.0, 3.0]) = [0.7311, 0.2689]
M ≤ 10,000 (number of tokens)E ≤ 256 (number of experts)k ≤ E (top-k selection, typically k=2)M = 1,024, k = 2