EmoVIT: Revolutionizing Emotion Insights with Visual Instruction Tuning

TL;DR

EmoVIT introduces a novel visual instruction tuning approach for emotion understanding in images, leveraging GPT-assisted data generation and large language models to improve emotion recognition, affective reasoning, and humor comprehension.

Contribution

This work pioneers emotion-specific visual instruction tuning using GPT-assisted data generation, expanding the capabilities of pre-trained models in visual emotion understanding.

Findings

Enhanced emotion classification accuracy

Improved affective reasoning performance

Effective humor comprehension in visual contexts

Abstract

Visual Instruction Tuning represents a novel learning paradigm involving the fine-tuning of pre-trained language models using task-specific instructions. This paradigm shows promising zero-shot results in various natural language processing tasks but is still unexplored in vision emotion understanding. In this work, we focus on enhancing the model's proficiency in understanding and adhering to instructions related to emotional contexts. Initially, we identify key visual clues critical to visual emotion recognition. Subsequently, we introduce a novel GPT-assisted pipeline for generating emotion visual instruction data, effectively addressing the scarcity of annotated instruction data in this domain. Expanding on the groundwork established by InstructBLIP, our proposed EmoVIT architecture incorporates emotion-specific instruction data, leveraging the powerful capabilities of Large Language Models to enhance performance. Through extensive experiments, our model showcases its proficiency in emotion classification, adeptness in affective reasoning, and competence in comprehending humor. The comparative analysis provides a robust benchmark for Emotion Visual Instruction Tuning in the era of LLMs, providing valuable insights and opening avenues for future exploration in this domain. Our code is available at \url{https://github.com/aimmemotion/EmoVIT}.