<ul><li>Large-scale foundation models have shown versatility in various tasks but fine-tuning them completely is computationally expensive.</li><li>A Parameter-Efficient Fine-Tuning (PEFT) method called LoRA introduces low-rank updates to pre-trained weights to reduce computational costs.</li><li>Through singular value decomposition (SVD), it was found that during fine-tuning, top singular values are amplified while the rest remain mostly unchanged, injecting task-specific knowledge into a low-dimensional subspace.</li><li>A novel method leveraging learnable rescaling of top singular directions has been proposed for precise modulation of influential components, leading to consistent improvements across multiple tasks.</li></ul>

Weight Spectra Induced Efficient Model Adaptation

Discover more