<ul><li>LightOn and Answer.ai have released ModernBERT, an open family of encoder-only models that is a pareto improvement over BERT in terms of speed and accuracy.</li><li>ModernBERT extends the context length to 8,192 tokens, allowing it to handle long-context tasks effectively.</li><li>The model incorporates architectural enhancements such as Flash Attention 2 and rotary positional embeddings (RoPE) to improve computational efficiency and positional understanding.</li><li>ModernBERT outperforms models like RoBERTa and DeBERTa in various benchmarks and is more resource-efficient for inference on GPUs.</li></ul>

LightOn and Answer.ai Releases ModernBERT: A New Model Series that is a Pareto Improvement over BERT with both Speed and Accuracy

Discover more