On the Difference of BERT-style and CLIP-style Text Encoders

TL;DR

This paper compares BERT-style and CLIP-style text encoders, revealing that CLIP encoders excel in cross-modal association despite underperforming in general NLP tasks, highlighting their unique synesthetic ability.

Contribution

It provides a comprehensive analysis of CLIP-style text encoders, emphasizing their cross-modal capabilities and differences from traditional BERT-style models.

Findings

CLIP-style encoders underperform in general text understanding

CLIP encoders exhibit synesthesia for cross-modal association

BERT encoders outperform in traditional NLP tasks

Abstract

Masked language modeling (MLM) has been one of the most popular pretraining recipes in natural language processing, e.g., BERT, one of the representative models. Recently, contrastive language-image pretraining (CLIP) has also attracted attention, especially its vision models that achieve excellent performance on a broad range of vision tasks. However, few studies are dedicated to studying the text encoders learned by CLIP. In this paper, we analyze the difference between BERT-style and CLIP-style text encoders from three experiments: (i) general text understanding, (ii) vision-centric text understanding, and (iii) text-to-image generation. Experimental analyses show that although CLIP-style text encoders underperform BERT-style ones for general text understanding tasks, they are equipped with a unique ability, i.e., synesthesia, for the cross-modal association, which is more similar to the senses of humans.