← All problems
#85 LoRA Linear Medium

Implement a LoRA (Low-Rank Adaptation) linear layer forward pass. Given an input matrix x of shape batch × d_in, a base weight matrix W of shape d_out × d_in, a LoRA down-projection matrix A of shape rank × d_in, and a LoRA up-projection matrix B of shape d_out × rank, compute output = x × WT + lora_scale × (x × AT) × BT. All tensors are float32.

x B×D_in W D_out×D_in x@Wᵗ B×D_out A rank×D_in x@Aᵗ B×rank B D_out×rank α×(x@Aᵗ)@Bᵗ B×D_out + output B×D_out

Implementation Requirements

Examples

Example 1:

\[ x = \begin{bmatrix} 1 & 0 & -1 & 2 \\ 0 & 1 & 1 & -1 \end{bmatrix},\quad W = \begin{bmatrix} 1 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 \\ 0 & 0 & 1 & 0 \end{bmatrix},\quad A = \begin{bmatrix} 1 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 \end{bmatrix},\quad B = \begin{bmatrix} 1 & 0 \\ 0 & 1 \\ 0 & 0 \end{bmatrix} \]

With lora_scale = 0.5:

\[ \text{output} = x W^T + 0.5 \cdot (x A^T) B^T = \begin{bmatrix} 1 & 0 & -1 \\ 0 & 1 & 1 \end{bmatrix} + 0.5 \cdot \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix} \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \end{bmatrix} = \begin{bmatrix} 1.5 & 0 & -1 \\ 0 & 1.5 & 1 \end{bmatrix} \]

Constraints

Guided walkthrough coming soon. Below is the official starter signature in each language; the clickable line-by-line explanation hasn't been authored yet.
Problem text & starters from the official AlphaGPU/leetgpu-challenges. Solve it on leetgpu.com.