<ul><li>Federated fine-tuning (FedFT) is used for adapting large language models (LLMs) to distributed data environments while preserving data privacy.</li><li>Existing FedFT methods primarily employ parameter-efficient fine-tuning techniques but struggle with catastrophic forgetting in distributed environments.</li><li>To address this issue, the proposed FedBE framework integrates an adaptive transformer block expansion mechanism with dynamic trainable-block allocation.</li><li>FedBE shows improved accuracy retention (12-74%) on general tasks after fine-tuning and accelerates model convergence (1.9-3.1x) without compromising downstream task accuracy.</li></ul>

Mitigating Catastrophic Forgetting with Adaptive Transformer Block Expansion in Federated Fine-Tuning

Discover more