Unified Reinforcement and Imitation Learning for Vision-Language Models

TL;DR

This paper presents a novel training algorithm called Unified Reinforcement and Imitation Learning (RIL) that creates efficient, lightweight vision-language models by combining reinforcement learning with adversarial imitation, achieving performance close to larger models.

Contribution

The paper introduces RIL, a new unified training method that integrates reinforcement and imitation learning to improve lightweight vision-language models.

Findings

RIL narrows the performance gap with state-of-the-art VLMs.

Student models outperform baseline methods in benchmarks.

RIL enables smaller models to mimic and improve upon large teacher models.

Abstract

Vision-Language Models (VLMs) have achieved remarkable progress, yet their large scale often renders them impractical for resource-constrained environments. This paper introduces Unified Reinforcement and Imitation Learning (RIL), a novel and efficient training algorithm designed to create powerful, lightweight VLMs. RIL distinctively combines the strengths of reinforcement learning with adversarial imitation learning. This enables smaller student VLMs not only to mimic the sophisticated text generation of large teacher models but also to systematically improve their generative capabilities through reinforcement signals. Key to our imitation framework is an LLM-based discriminator that adeptly distinguishes between student and teacher outputs, complemented by guidance from multiple large teacher VLMs to ensure diverse learning. This unified learning strategy, leveraging both reinforcement and imitation, empowers student models to achieve significant performance gains, making them competitive with leading closed-source VLMs. Extensive experiments on diverse vision-language benchmarks demonstrate that RIL significantly narrows the performance gap with state-of-the-art open- and closed-source VLMs and, in several instances, surpasses them.