Tell me about LoRA: what it is, how it is trained, and how it works during inference.

LoRA (Low-Rank Adaptation) is a method for adapting large neural networks by training a small number of parameters, which allows for efficient fine-tuning of models without full retraining.

How it works:

• In the original model, the layer weights are represented as matrices.
• LoRA adds low-rank corrections to these matrices (decomposition into two small matrices).
• During training, only these low-rank matrices are updated, while the original weights remain frozen.

Training:

• Only a small number of parameters are trained, reducing resource requirements and training time.
• It allows for quick adaptation of the model to new tasks or data.

Inference:

• During inference, the corrections from LoRA matrices are summed with the original weights, creating an adapted model.
• This slightly increases computations but maintains efficiency and compactness.

It can be roughly represented as: if the original weight matrix is W, then the final weight after LoRA is W + BA, where B and A are low-rank matrices trained separately.