<ul><li>Position bias is a prevalent issue in modern language models.</li><li>The bias leads to unexpected model failures and affects performance, robustness, and reliability.</li><li>A mechanistic analysis identifies causal attention and relative positional encodings as the sources of bias.</li><li>A training-free zero-shot approach called PINE (Position-INvariant inferencE) is proposed to eliminate the bias and improve performance in downstream tasks.</li></ul>

Eliminating Position Bias of Language Models: A Mechanistic Approach

Discover more