<ul><li>StatsMerging is a new lightweight learning-based model merging method designed to accommodate multiple large models within memory constraints.</li><li>It leverages singular values from singular value decomposition (SVD) to capture task-specific weight distributions and predict task coefficients.</li><li>StatsMerging employs a lightweight learner, StatsMergeLearner, to enhance generalization of weight distributions of task-specific pre-trained models.</li><li>The method introduces Task-Specific Teacher Distillation for merging vision models with different architectures, achieving improved accuracy, generalization, and robustness in experiments across eight tasks.</li></ul>

StatsMerging: Statistics-Guided Model Merging via Task-Specific Teacher Distillation

Discover more