LXMERT Model Compression for Visual Question Answering

TL;DR

This paper investigates the existence of trainable subnetworks within LXMERT for visual question answering and demonstrates effective pruning of 40-60% with minimal accuracy loss.

Contribution

It combines the lottery ticket hypothesis with LXMERT fine-tuning to identify prunable subnetworks for VQA, providing a size reduction analysis.

Findings

LXMERT can be pruned by 40-60%

Pruning results in only 3% accuracy loss

Subnetworks exist within LXMERT for VQA

Abstract

Large-scale pretrained models such as LXMERT are becoming popular for learning cross-modal representations on text-image pairs for vision-language tasks. According to the lottery ticket hypothesis, NLP and computer vision models contain smaller subnetworks capable of being trained in isolation to full performance. In this paper, we combine these observations to evaluate whether such trainable subnetworks exist in LXMERT when fine-tuned on the VQA task. In addition, we perform a model size cost-benefit analysis by investigating how much pruning can be done without significant loss in accuracy. Our experiment results demonstrate that LXMERT can be effectively pruned by 40%-60% in size with 3% loss in accuracy.