<ul><li>Researchers propose a systematic mathematical framework for Rotary Position Embedding (RoPE), a technique widely adopted in Transformers for encoding relative positions.</li><li>The framework is grounded in Lie group and Lie algebra theory and provides a unified theoretical foundation for RoPE in higher dimensions.</li><li>The study identifies the core properties of RoPE as relativity and reversibility and derives general constraints and constructions for valid RoPE in different dimensions.</li><li>The proposed framework unifies and explains existing RoPE designs and allows for extensions to new modalities and tasks.</li></ul>

Rethinking RoPE: A Mathematical Blueprint for N-dimensional Positional Encoding

Discover more