<ul><li>Researchers have introduced Generalized Fisher-Weighted SVD (GFWSVD) as a new post-training technique for compressing large language models (LLMs).</li><li>GFWSVD considers both diagonal and off-diagonal elements of the Fisher information matrix to better reflect parameter importance, addressing the limitations of simple diagonal approximations used in previous methods.</li><li>The method utilizes a scalable adaptation of the Kronecker-factored approximation algorithm for the observed Fisher information, improving compression performance.</li><li>In experiments on LLM compression, GFWSVD outperformed existing compression techniques like FWSVD, SVD-LLM, and ASVD at various compression rates, showcasing improved effectiveness.</li></ul>

Generalized Fisher-Weighted SVD: Scalable Kronecker-Factored Fisher Approximation for Compressing Large Language Models

Discover more