Compressing Visual-linguistic Model via Knowledge Distillation

TL;DR

This paper introduces a knowledge distillation method to effectively compress large visual-linguistic models into smaller ones by aligning their representations despite differences in visual region proposals.

Contribution

It proposes a novel distillation approach that aligns hidden states and attention distributions using shared region proposals, improving small VL model performance.

Findings

Achieves 120.8 CIDEr score on COCO captioning, surpassing baseline by 5.1

Attains 69.8 accuracy on VQA 2.0, 0.8 higher than baseline

Demonstrates effectiveness in both pre-training and fine-tuning stages

Abstract

Despite exciting progress in pre-training for visual-linguistic (VL) representations, very few aspire to a small VL model. In this paper, we study knowledge distillation (KD) to effectively compress a transformer-based large VL model into a small VL model. The major challenge arises from the inconsistent regional visual tokens extracted from different detectors of Teacher and Student, resulting in the misalignment of hidden representations and attention distributions. To address the problem, we retrain and adapt the Teacher by using the same region proposals from Student's detector while the features are from Teacher's own object detector. With aligned network inputs, the adapted Teacher is capable of transferring the knowledge through the intermediate representations. Specifically, we use the mean square error loss to mimic the attention distribution inside the transformer block and present a token-wise noise contrastive loss to align the hidden state by contrasting with negative representations stored in a sample queue. To this end, we show that our proposed distillation significantly improves the performance of small VL models on image captioning and visual question answering tasks. It reaches 120.8 in CIDEr score on COCO captioning, an improvement of 5.1 over its non-distilled counterpart; and an accuracy of 69.8 on VQA 2.0, a 0.8 gain from the baseline. Our extensive experiments and ablations confirm the effectiveness of VL distillation in both pre-training and fine-tuning stages.