<ul><li>Mixture-of-experts (MoE) is a popular approach in parameter-efficient fine-tuning to balance model capacity and computation overhead.</li><li>Current MoE variants face challenges with heterogeneous datasets due to underutilization of experts' capacity.</li><li>CoMoE, Contrastive Representation for MoE, is introduced to enhance modularization and specialization within MoE by incorporating a contrastive objective.</li><li>Experiments show that CoMoE improves MoE's capacity and promotes modularization among experts across various benchmarks and multi-task settings.</li></ul>

CoMoE: Contrastive Representation for Mixture-of-Experts in Parameter-Efficient Fine-tuning

Discover more