Skip to content

LoRA Primer

MinT trains with LoRA — low-rank adaptation — which updates a small adapter instead of all the base weights. It is far cheaper than full fine-tuning while matching it on the workloads MinT targets.

When LoRA matches full fine-tuning

  • Small-to-medium SFT (instruction tuning, reasoning data): LoRA matches full fine-tuning.
  • Very large SFT datasets: once the data exceeds the adapter's capacity, LoRA falls behind. The gap grows with dataset size rather than hitting a hard floor.
  • Reinforcement learning: LoRA matches full fine-tuning even at low rank — RL needs very little capacity.
  • Large batch sizes: LoRA is less tolerant of very large batches than full fine-tuning, and raising the rank does not fix it (a property of the product-of-matrices parametrisation, not the rank).
  • Where to attach: apply LoRA to all weight matrices (attention and MLP, including MoE layers). Attention-only LoRA underperforms even with the rank raised to match the parameter count.

Learning rate

The learning rate is the hyperparameter that matters most, and LoRA needs a much larger one than full fine-tuning — roughly 10× larger. A common mistake is to keep a full-fine-tuning LR after switching to LoRA and conclude LoRA works poorly.

Pick the LR as: lora_lr ≈ 10 × full_finetune_lr. Conveniently, the optimal LR does not depend on the rank — train at different ranks with the same LR and the first few steps overlap.

What LoRA actually is

Given a weight matrix (W), LoRA replaces it with (W' = W + BA), where (B) is (n \times r) and (A) is (r \times n), with (r \ll n) the rank. MinT's default rank is 32.

The low-rank framing is not the important part; it is simpler to think of LoRA as a random projection of the parameter space that happens to be cheap to train. For RL and small SFT you only learn a small amount of information, and the projected subspace is more than enough.

Choosing a rank

  • Default rank=32 is fine for RL and small SFT.
  • Large SFT datasets: raise the rank. As a rough rule, LoRA works well as long as the number of adapter parameters is at least as large as the number of completion tokens (the weight=1 tokens) you train on.
  • RL: small ranks match large ranks and full fine-tuning.