<ul><li>Researchers propose a method to enhance State-space models (SSMs) by sparsifying them within given computational budgets.</li><li>The method involves a hierarchical sparsification technique called Simba, which prunes tokens in upper layers more than in lower layers to mimic highway behavior.</li><li>By implementing Simba, researchers show improved performance in natural language tasks compared to the baseline model, Mamba, with the same FLOPS.</li><li>The study demonstrates that Simba not only increases efficiency but also enhances information flow across long sequences.</li></ul>

Sparsified State-Space Models are Efficient Highway Networks

Discover more