TernaryCLIP: Efficiently Compressing Vision-Language Models with Ternary Weights and Distilled Knowledge

TL;DR

TernaryCLIP introduces a method to compress large vision-language models by converting weights to ternary format with minimal performance loss, enabling efficient deployment on resource-limited devices.

Contribution

It presents TernaryCLIP, the first framework to convert CLIP's weights into ternary format with quantization-aware training and distillation, achieving high compression and acceleration.

Findings

Achieves 99% ternarized weights with 1.58-bit representation

Provides 16.98× compression ratio and 2.3× inference speedup

Maintains strong zero-shot performance across 41 datasets

Abstract

Recent years have witnessed an increasing interest in image-text contrastive modeling, exemplified by models such as Contrastive Language-Image Pretraining (CLIP). In this paper, we propose the TernaryCLIP, a lightweight computational framework that converts connection weights of both vision and text encoders of CLIP into the ternary format, instead of full-precision or floating ones. TernaryCLIP incorporates quantization-aware training and distillation modules, preventing precision degradation and enabling low-cost and high-efficiency computations. Comprehensive experiments demonstrate that TernaryCLIP can achieve up to 99\% ternarized weights with 1.58-bit representation, 16.98 $\times$ compression ratio, 2.3 $\times$ inference acceleration, 16 $\times$ storage reduction, 10 $\times$ memory optimization, and 60\% sparsity while maintaining promising performance on zero-shot image classification and image-text retrieval tasks across 41 commonly used datasets. Our work highlights the feasibility of extreme quantization for large multimodal models, supporting effective and efficient deployment on resource-constrained devices. The model and code can be accessed from Hugging Face and GitHub.