EmoVLM-KD: Fusing Distilled Expertise with Vision-Language Models for Visual Emotion Analysis

TL;DR

EmoVLM-KD introduces a novel approach that combines instruction-tuned vision-language models with a distilled vision module to improve visual emotion analysis performance efficiently.

Contribution

The paper presents EmoVLM-KD, a method that distills knowledge from conventional vision models into vision-language models for enhanced emotion prediction.

Findings

Achieves state-of-the-art results on multiple benchmarks.

Maintains computational efficiency compared to dual-model approaches.

Effectively balances predictions from vision-language and vision models.

Abstract

Visual emotion analysis, which has gained considerable attention in the field of affective computing, aims to predict the dominant emotions conveyed by an image. Despite advancements in visual emotion analysis with the emergence of vision-language models, we observed that instruction-tuned vision-language models and conventional vision models exhibit complementary strengths in visual emotion analysis, as vision-language models excel in certain cases, whereas vision models perform better in others. This finding highlights the need to integrate these capabilities to enhance the performance of visual emotion analysis. To bridge this gap, we propose EmoVLM-KD, an instruction-tuned vision-language model augmented with a lightweight module distilled from conventional vision models. Instead of deploying both models simultaneously, which incurs high computational costs, we transfer the predictive patterns of a conventional vision model into the vision-language model using a knowledge distillation framework. Our approach first fine-tunes a vision-language model on emotion-specific instruction data and then attaches a distilled module to its visual encoder while keeping the vision-language model frozen. Predictions from the vision language model and the distillation module are effectively balanced by a gate module, which subsequently generates the final outcome. Extensive experiments show that EmoVLM-KD achieves state-of-the-art performance on multiple visual emotion analysis benchmark datasets, outperforming the existing methods while maintaining computational efficiency. The code is available in https://github.com/sange1104/EmoVLM-KD.