Efficient Ternary Weight Embedding Model: Bridging Scalability and Performance

TL;DR

This paper introduces a novel finetuning framework for ternary-weight embedding models that significantly reduces memory and computational costs while maintaining high performance, suitable for resource-constrained environments.

Contribution

It presents a self-taught knowledge distillation method for applying ternarization to pre-trained embedding models, enhancing efficiency without sacrificing effectiveness.

Findings

Ternary embedding models achieve low memory usage and low latency.

Combining ternary embeddings with ANN search improves accuracy and efficiency.

Extensive experiments validate the effectiveness across text and vision datasets.

Abstract

Embedding models have become essential tools in both natural language processing and computer vision, enabling efficient semantic search, recommendation, clustering, and more. However, the high memory and computational demands of full-precision embeddings pose challenges for deployment in resource-constrained environments, such as real-time recommendation systems. In this work, we propose a novel finetuning framework to ternary-weight embedding models, which reduces memory and computational overhead while maintaining high performance. To apply ternarization to pre-trained embedding models, we introduce self-taught knowledge distillation to finalize the ternary-weights of the linear layers. With extensive experiments on public text and vision datasets, we demonstrated that without sacrificing effectiveness, the ternarized model consumes low memory usage and has low latency in the inference stage with great efficiency. In practical implementations, embedding models are typically integrated with Approximate Nearest Neighbor (ANN) search. Our experiments combining ternary embedding with ANN search yielded impressive improvement in both accuracy and computational efficiency. The repository is available at here.