<ul data-eligibleForWebStory="true"><li>TreeLoRA (K-D Tree of Low-Rank Adapters) is a novel approach for efficient continual learning (CL) for large pre-trained models (LPMs).</li><li>The approach constructs layer-wise adapters using hierarchical gradient similarity to update models online while preventing catastrophic forgetting.</li><li>Efficiency is crucial for CL due to the computational demands and growing parameter sizes of LPMs.</li><li>To reduce task similarity estimation computational burden, bandit techniques with lower confidence bounds are employed.</li><li>Sparse gradient updates are used to optimize parameters, especially suited for LPMs.</li><li>The approach is justified theoretically, with experiments on vision transformers (ViTs) and large language models (LLMs) showing effectiveness.</li><li>Experiments across vision and natural language processing tasks demonstrate the effectiveness and efficiency of TreeLoRA.</li></ul>

TreeLoRA: Efficient Continual Learning via Layer-Wise LoRAs Guided by a Hierarchical Gradient-Similarity Tree

Discover more