Implement a LoRA (Low-Rank Adaptation) linear layer forward pass. Given an input matrix
x of shape batch × d_in, a base weight matrix W of
shape d_out × d_in, a LoRA down-projection matrix A of shape
rank × d_in, and a LoRA up-projection matrix B of shape
d_out × rank, compute
output = x × WT + lora_scale × (x × AT) × BT.
All tensors are float32.
solve function; do not change its signature.output.Example 1:
\[ x = \begin{bmatrix} 1 & 0 & -1 & 2 \\ 0 & 1 & 1 & -1 \end{bmatrix},\quad W = \begin{bmatrix} 1 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 \\ 0 & 0 & 1 & 0 \end{bmatrix},\quad A = \begin{bmatrix} 1 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 \end{bmatrix},\quad B = \begin{bmatrix} 1 & 0 \\ 0 & 1 \\ 0 & 0 \end{bmatrix} \]
With lora_scale = 0.5:
\[ \text{output} = x W^T + 0.5 \cdot (x A^T) B^T = \begin{bmatrix} 1 & 0 & -1 \\ 0 & 1 & 1 \end{bmatrix} + 0.5 \cdot \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix} \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \end{bmatrix} = \begin{bmatrix} 1.5 & 0 & -1 \\ 0 & 1.5 & 1 \end{bmatrix} \]
batch ≤ 1,024d_in, d_out ≤ 8,192rank ≤ 256; rank < min(d_in, d_out)float32 on GPU.batch = 256, d_in = 4,096, d_out = 4,096, rank = 64