Parameter-Efficient Finetuning
Full fine-tuning of a seven-billion-parameter model requires storing gradients and Adam's two moment buffers for every single one of those parameters — memory that a single consumer GPU simply doesn't have. Parameter-efficient fine-tuning methods sidestep this by training a tiny fraction of parameters instead, built on a striking empirical observation: the weight updates that fine-tuning actually needs are far lower-rank than the weight matrices themselves.