<ul data-eligibleForWebStory="true"><li>Continual learning with large pre-trained models is challenging due to catastrophic forgetting and task interference.</li><li>A new approach called MoRA is proposed to address challenges like interference, redundancy, and ambiguity in existing Mixture-of-Experts (MoE) methods.</li><li>MoRA utilizes a Mixture-of-Rank Adaptive learning approach with self-activated and sparse rank activation to improve continual learning tasks with pre-trained models like CLIP and large language models (LLMs).</li><li>The proposed MoRA approach demonstrates effectiveness in enhancing continual learning with pre-trained models, improving generalization, and mitigating forgetting.</li></ul>

Little By Little: Continual Learning via Self-Activated Sparse Mixture-of-Rank Adaptive Learning

Discover more