Some Like It Small: Czech Semantic Embedding Models for Industry Applications

TL;DR

This paper develops small Czech sentence embedding models optimized for industry use, demonstrating their efficiency and effectiveness in real-world search applications with significant size and speed advantages.

Contribution

It introduces and evaluates compact Czech sentence embedding models using innovative training techniques, achieving competitive performance with much smaller and faster models.

Findings

Models are approximately 8 times smaller and 5 times faster than larger counterparts.

The models outperform previous versions in search-related tasks.

Public release of models and evaluation pipeline promotes reproducibility.

Abstract

This article focuses on the development and evaluation of Small-sized Czech sentence embedding models. Small models are important components for real-time industry applications in resource-constrained environments. Given the limited availability of labeled Czech data, alternative approaches, including pre-training, knowledge distillation, and unsupervised contrastive fine-tuning, are investigated. Comprehensive intrinsic and extrinsic analyses are conducted, showcasing the competitive performance of our models compared to significantly larger counterparts, with approximately 8 times smaller size and 5 times faster speed than conventional Base-sized models. To promote cooperation and reproducibility, both the models and the evaluation pipeline are made publicly accessible. Ultimately, this article presents practical applications of the developed sentence embedding models in Seznam.cz, the Czech search engine. These models have effectively replaced previous counterparts, enhancing the overall search experience for instance, in organic search, featured snippets, and image search. This transition has yielded improved performance.