L-CLIPScore: a Lightweight Embedding-based Captioning Metric for Evaluating and Training

TL;DR

L-CLIPScore introduces a lightweight, efficient embedding-based metric for evaluating and training image captioning models, leveraging a compressed and distilled version of CLIP with a novel similarity regulator loss.

Contribution

The paper presents L-CLIPScore, a novel lightweight CLIP variant with a new similarity regulator loss, enabling efficient caption evaluation and training with comparable performance to original CLIP.

Findings

L-CLIPScore achieves similar multi-modal alignment to CLIP with fewer resources.

It effectively evaluates caption quality in experiments.

Using L-CLIPScore alone for training can lead to failure, but combining it with n-gram metrics improves results.

Abstract

We propose a novel embedding-based captioning metric termed as L-CLIPScore that can be used for efficiently evaluating caption quality and training captioning model. L-CLIPScore is calculated from a lightweight CLIP (L-CLIP), which is a dual-encoder architecture compressed and distilled from CLIP. To compress, we apply two powerful techniques which are weight multiplexing and matrix decomposition for reducing the parameters of encoders and word embedding matrix, respectively. To distill, we design a novel multi-modal Similarity Regulator (SR) loss to transfer more vision-language alignment knowledge. Specifically, SR loss amplifies the multi-modal embedding similarity if the given image-text pair is matched and diminishes the similarity if the pair is non-matched. By compressing and distilling by this novel SR loss, our L-CLIP achieves comparable multi-modal alignment ability to the original CLIP while it requires fewer computation resources and running time. We carry out exhaustive experiments to validate the efficiency and effectiveness of L-CLIPScore when using it as the judge to evaluate caption quality. We also discover that when using L-CLIPScore as the supervisor to train the captioning model, it should be mixed up by an n-gram-based metric and meanwhile analyze why using L-CLIPScore only will cause fail training.