RexBERT: Context Specialized Bidirectional Encoders for E-commerce
Rahul Bajaj, Anuj Garg

TL;DR
RexBERT is a family of domain-specific BERT-style encoders optimized for e-commerce, trained on a large curated corpus, outperforming larger general models on relevant tasks.
Contribution
The paper introduces RexBERT, a specialized e-commerce encoder trained on a large domain-specific corpus with a new training recipe, achieving superior performance with fewer parameters.
Findings
RexBERT outperforms larger general-purpose encoders on e-commerce tasks.
A curated 350 billion token corpus enhances domain relevance.
A three-phase training recipe improves domain adaptation.
Abstract
Encoder-only transformers remain indispensable in retrieval, classification, and ranking systems where latency, stability, and cost are paramount. Most general purpose encoders, however, are trained on generic corpora with limited coverage of specialized domains. We introduce RexBERT, a family of BERT-style encoders designed specifically for e-commerce semantics. We make three contributions. First, we release Ecom-niverse, a 350 billion token corpus curated from diverse retail and shopping sources. We describe a modular pipeline that isolates and extracts e-commerce content from FineFineWeb and other open web resources, and characterize the resulting domain distribution. Second, we present a reproducible pretraining recipe building on ModernBERT's architectural advances. The recipe consists of three phases: general pre-training, context extension, and annealed domain specialization.…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsText and Document Classification Technologies · Topic Modeling · Spam and Phishing Detection
