A naukri.com initiative
Arxiv
2d
280
Image Credit: Arxiv
FLoRIST: Singular Value Thresholding for Efficient and Accurate Federated Fine-Tuning of Large Language Models
- Integrating Low-Rank Adaptation (LoRA) into federated learning offers a solution for fine-tuning Large Language Models (LLMs) efficiently without sharing local data.
- Challenges in balancing communication efficiency, model accuracy, and computational cost exist in methods designed for federated LoRA, especially among heterogeneous clients.
- Existing methods for federated LoRA either rely on simplistic local adapter averaging introducing noise, require transmitting large local adapters leading to poor communication efficiency, or need computationally expensive decomposition for client-specific low-rank adapters.
- The proposed FLoRIST framework achieves accurate aggregation without high communication or computational overhead by performing singular value decomposition on stacked local adapters separately.
- FLoRIST operates within a compact intermediate space to represent information from local LoRAs and uses tunable singular value thresholding for server-side optimal rank selection to construct global low-rank adapters shared by all clients.
- Empirical evaluations across datasets and LLMs show that FLoRIST balances superior communication efficiency and competitive performance in homogeneous and heterogeneous setups.
Read Full Article
16 Likes
For uninterrupted reading, download the app