How to Make Cross Encoder a Good Teacher for Efficient Image-Text Retrieval?

TL;DR

This paper introduces a novel distillation method, CPRD, that enhances dual-encoder models for image-text retrieval by effectively transferring knowledge from a cross-encoder, focusing on relative ranking of hard negatives.

Contribution

The paper proposes Contrastive Partial Ranking Distillation (CPRD), a new approach that improves dual-encoder accuracy by better mimicking cross-encoder ranking behavior, especially for hard negatives.

Findings

CPRD outperforms existing distillation methods in image-text retrieval.

Focusing on hard negatives' relative order is crucial for effective knowledge transfer.

Maintaining coordination between distillation and training losses enhances model performance.

Abstract

Dominant dual-encoder models enable efficient image-text retrieval but suffer from limited accuracy while the cross-encoder models offer higher accuracy at the expense of efficiency. Distilling cross-modality matching knowledge from cross-encoder to dual-encoder provides a natural approach to harness their strengths. Thus we investigate the following valuable question: how to make cross-encoder a good teacher for dual-encoder? Our findings are threefold:(1) Cross-modal similarity score distribution of cross-encoder is more concentrated while the result of dual-encoder is nearly normal making vanilla logit distillation less effective. However ranking distillation remains practical as it is not affected by the score distribution.(2) Only the relative order between hard negatives conveys valid knowledge while the order information between easy negatives has little significance.(3) Maintaining the coordination between distillation loss and dual-encoder training loss is beneficial for knowledge transfer. Based on these findings we propose a novel Contrastive Partial Ranking Distillation (CPRD) method which implements the objective of mimicking relative order between hard negative samples with contrastive learning. This approach coordinates with the training of the dual-encoder effectively transferring valid knowledge from the cross-encoder to the dual-encoder. Extensive experiments on image-text retrieval and ranking tasks show that our method surpasses other distillation methods and significantly improves the accuracy of dual-encoder.