<ul><li>Mixture-of-Experts (MoE) has successfully scaled up models while maintaining nearly constant computing costs.</li><li>Efficiency of MoE is challenging to achieve due to imbalanced expert activation and communication overhead.</li><li>The paper proposes a collaboration-constrained routing (C2R) strategy to improve expert utilization and reduce communication costs.</li><li>Experiments show an average performance improvement of 0.51% and 0.33% on two MoE models across ten NLP benchmarks.</li></ul>

Advancing MoE Efficiency: A Collaboration-Constrained Routing (C2R) Strategy for Better Expert Parallelism Design

Discover more