<ul data-eligibleForWebStory="true"><li>LieRE is introduced as an enhancement to the popular Rotary Position Encoding (RoPE) used in Transformer architectures.</li><li>RoPE has limitations with one-dimensional sequence data and restricted representational capacity, prompting the development of LieRE.</li><li>LieRE generalizes RoPE to high-dimensional rotation matrices by leveraging their Lie group structure.</li><li>Extensive evaluation on image datasets shows LieRE achieving improvement over state-of-the-art baselines in both 2D and 3D classification tasks.</li><li>LieRE offers superior generalization to higher resolutions and is computationally efficient, reproducible on 4 A100 GPUs in 30 minutes on CIFAR100.</li><li>LieRE code is available at https://github.com/StanfordMIMI/LieRE.</li></ul>

LieRE: Lie Rotational Positional Encodings

Discover more