NeoBERT: A Next-Generation BERT
Lola Le Breton, Quentin Fournier, Mariam El Mezouar, John X. Morris, Sarath Chandar

TL;DR
NeoBERT introduces a next-generation encoder that combines architectural innovations, modern data, and optimized pre-training to outperform existing models like BERT and RoBERTa on multiple benchmarks, with a compact size and seamless integration.
Contribution
It presents NeoBERT, a new encoder model that integrates state-of-the-art advancements, achieving superior performance with a compact design and extended context length.
Findings
Outperforms BERT large and RoBERTa large on MTEB benchmark.
Achieves state-of-the-art results with only 250M parameters.
Demonstrates effectiveness across GLUE and MTEB evaluations.
Abstract
Recent innovations in architecture, pre-training, and fine-tuning have led to the remarkable in-context learning and reasoning abilities of large auto-regressive language models such as LLaMA and DeepSeek. In contrast, encoders like BERT and RoBERTa have not seen the same level of progress despite being foundational for many downstream NLP applications. To bridge this gap, we introduce NeoBERT, a next-generation encoder that redefines the capabilities of bidirectional models by integrating state-of-the-art advancements in architecture, modern data, and optimized pre-training methodologies. NeoBERT is designed for seamless adoption: it serves as a plug-and-play replacement for existing base models, relies on an optimal depth-to-width ratio, and leverages an extended context length of 4,096 tokens. Despite its compact 250M parameter footprint, it achieves state-of-the-art results on the…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsTopic Modeling · Natural Language Processing Techniques · Multimodal Machine Learning Applications
MethodsRefunds@Expedia|||How do I get a full refund from Expedia? · Attention Is All You Need · Adam · Softmax · Dropout · Weight Decay · Attention Dropout · Dense Connections · Linear Layer · Layer Normalization
