FREE: Fast and Robust Vision Language Models with Early Exits

TL;DR

This paper introduces FREE, an adversarial training method with early exit strategies for vision-language models, significantly speeding up inference while maintaining accuracy and robustness.

Contribution

We propose a novel adversarial training framework for early exit strategies in VLMs, improving inference speed and robustness with minimal performance loss.

Findings

Speeds up inference by over 1.51x

Enhances model robustness against overthinking

Maintains comparable accuracy with faster inference

Abstract

In recent years, Vision-Language Models (VLMs) have shown remarkable performance improvements in Vision-Language tasks. However, their large size poses challenges for real-world applications where inference latency is a concern. To tackle this issue, we propose employing Early Exit (EE) strategies in VLMs. However, training exit classifiers in VLMs is challenging, particularly with limited labeled training data. To address this, we introduce FREE, an adversarial training approach within a GAN-based framework. Here, each exit consists of a transformer layer and a classifier. The transformer layer is adversarially trained to produce feature representations similar to the final layer, while a feature classifier serves as the discriminator. Our method focuses on performing input-adaptive inference that increases inference speed with minimal drop in performance. Experimental results demonstrate the effectiveness of our approach in enhancing accuracy and model robustness by mitigating overthinking and the phenomenon of mid-crisis that we highlight. We experimentally validate that our method speeds up the inference process by more than 1.51x while retaining comparable performance. The source code is available at https://github.com/Div290/FREE.