GenRecal: Generation after Recalibration from Large to Small Vision-Language Models

TL;DR

GenRecal is a versatile distillation framework that aligns diverse vision-language models, enabling smaller models to effectively learn from larger ones and outperform existing systems on various benchmarks.

Contribution

We introduce GenRecal, a novel framework with a Recalibrator that facilitates knowledge transfer across heterogeneous VLM architectures, addressing a key challenge in model distillation.

Findings

GenRecal significantly improves baseline VLM performance.

It outperforms large-scale open- and closed-source VLMs.

Extensive experiments validate its effectiveness across benchmarks.

Abstract

Recent advancements in vision-language models (VLMs) have leveraged large language models (LLMs) to achieve performance on par with closed-source systems like GPT-4V. However, deploying these models in real-world scenarios, particularly on resource-constrained devices, remains challenging due to their substantial computational demands. This has spurred interest in distilling knowledge from large VLMs into smaller, more efficient counterparts. A key challenge arises here from the diversity of VLM architectures, which are built on different LLMs and employ varying token types-differing in vocabulary size, token splits, and token index ordering. To address this challenge of limitation to a specific VLM type, we present Generation after Recalibration (GenRecal), a general-purpose distillation framework for VLMs. GenRecal incorporates a Recalibrator that aligns and adapts feature representations between heterogeneous VLMs, enabling effective knowledge transfer across different types of VLMs. Through extensive experiments on multiple challenging benchmarks, we demonstrate that GenRecal significantly improves baseline performances, eventually outperforming large-scale open- and closed-source VLMs.