HKD4VLM: A Progressive Hybrid Knowledge Distillation Framework for Robust Multimodal Hallucination and Factuality Detection in VLMs

TL;DR

This paper introduces HKD4VLM, a progressive hybrid knowledge distillation framework that improves the robustness and factuality detection of vision-language models through hierarchical distillation and augmentation strategies.

Contribution

The paper proposes a novel hierarchical knowledge distillation framework for VLMs, combining coarse-to-fine alignment and augmentation techniques to enhance factuality and hallucination detection.

Findings

HKD4VLM outperforms larger models on downstream tasks.

Hierarchical distillation improves knowledge transfer efficiency.

Augmentation strategies boost model robustness.

Abstract

Driven by the rapid progress in vision-language models (VLMs), the responsible behavior of large-scale multimodal models has become a prominent research area, particularly focusing on hallucination detection and factuality checking. In this paper, we present the solution for the two tracks of Responsible AI challenge. Inspirations from the general domain demonstrate that a smaller distilled VLM can often outperform a larger VLM that is directly tuned on downstream tasks, while achieving higher efficiency. We thus jointly tackle two tasks from the perspective of knowledge distillation and propose a progressive hybrid knowledge distillation framework termed HKD4VLM. Specifically, the overall framework can be decomposed into Pyramid-like Progressive Online Distillation and Ternary-Coupled Refinement Distillation, hierarchically moving from coarse-grained knowledge alignment to fine-grained refinement. Besides, we further introduce the mapping shift-enhanced inference and diverse augmentation strategies to enhance model performance and robustness. Extensive experimental results demonstrate the effectiveness of our HKD4VLM. Ablation studies provide insights into the critical design choices driving performance gains.