Luxical: High-Speed Lexical-Dense Text Embeddings
DatologyAI: Luke Merrick, Alex Fang, Aldo Carranza, Alvin Deng, Amro Abbas, Brett Larsen, Cody Blakeney, Darren Teh, David Schwab, Fan Pan, Haakon Mongstad, Haoli Yin, Jack Urbanek, Jason Lee, Jason Telanoff, Josh Wills, Kaleigh Mentzer, Paul Burstein, Parth Doshi

TL;DR
Luxical is a high-speed text embedding library that combines lexical features and neural approximation to enable fast, scalable web-scale text organization with quality comparable to neural models.
Contribution
Luxical introduces a novel approach combining sparse TF--IDF features, a small neural network, and knowledge distillation to produce fast, high-quality text embeddings for large-scale applications.
Findings
Achieves 3x to 100x speedups over neural baselines.
Maintains comparable quality to neural embedding models.
Demonstrates effectiveness in web crawl retrieval and data curation tasks.
Abstract
Frontier language model quality increasingly hinges on our ability to organize web-scale text corpora for training. Today's dominant tools trade off speed and flexibility: lexical classifiers (e.g., FastText) are fast but limited to producing classification output scores, while the vector-valued outputs of transformer text embedding models flexibly support numerous workflows (e.g., clustering, classification, and retrieval) but are computationally expensive to produce. We introduce Luxical, a library for high-speed "lexical-dense" text embeddings that aims to recover the best properties of both approaches for web-scale text organization. Luxical combines sparse TF--IDF features, a small ReLU network, and a knowledge distillation training regimen to approximate large transformer embedding models at a fraction of their operational cost. In this technical report, we describe the Luxical…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsTopic Modeling · Sentiment Analysis and Opinion Mining · Computational and Text Analysis Methods
