<ul data-eligibleForWebStory="true"><li>MetaTT is a Tensor Train (TT) adapter framework designed for global low-rank fine-tuning of pre-trained transformers.</li><li>Unlike LoRA, MetaTT utilizes a single shared TT to factorize all transformer sub-modules such as query, key, value, projection, and feed-forward layers by indexing structural axes.</li><li>MetaTT adds parameters proportional to the sum across modes for a given rank, resulting in a much more compressed final adapter compared to LoRA.</li><li>Benchmarks comparing MetaTT with LoRA and other tensor-based methods on standard language modeling tasks show that MetaTT achieves a significant reduction in parameters while maintaining similar accuracy to LoRA and even outperforming other methods.</li><li>The TT ansatz benefit from mature optimization routines like DMRG-style rank adaptive minimization and Adam, making training simpler compared to other tensor factorization methods.</li><li>MetaTT allows for cheap appending of new modes, enabling shared adapters across multiple tasks without the need to redesign the core tensor.</li></ul>

MetaTT: A Global Tensor-Train Adapter for Parameter-Efficient Fine-Tuning

Discover more