SQ-LLaVA: Self-Questioning for Large Vision-Language Assistant

TL;DR

SQ-LLaVA introduces a self-questioning framework that enhances vision-language models by leveraging contextual image information, leading to improved visual understanding and performance without extensive visual instruction data.

Contribution

The paper proposes a novel self-questioning approach for vision-language models, enabling better cross-modality alignment and visual understanding through high-quality question generation.

Findings

SQ-LLaVA can generate meaningful image-related questions.

Self-questioning improves model performance on visual tasks.

Fine-tuning on high-quality data enhances understanding.

Abstract

Recent advances in vision-language models have shown notable generalization in broad tasks through visual instruction tuning. However, bridging the gap between the pre-trained vision encoder and the large language models (LLMs) becomes the whole network's bottleneck. To improve cross-modality alignment, existing works usually consider more visual instruction data covering a broader range of vision tasks to fine-tune the model for question-answering, which, however, is costly to obtain and has not thoroughly explored the rich contextual information contained in images. This paper first attempts to harness the overlooked context within visual instruction data, training the model to self-supervised "learning" how to ask high-quality questions. In this way, we introduce a novel framework named SQ-LLaVA: Self-Questioning for Large Vision-Language Assistant. SQ-LLaVA exhibits proficiency in generating flexible and meaningful image-related questions while analyzing the visual clue and prior language knowledge, signifying an advanced level of generalized visual understanding. Moreover, fine-tuning SQ-LLaVA on higher-quality instruction data shows a performance improvement compared with traditional visual-instruction tuning methods. This improvement highlights the efficacy of self-questioning techniques in achieving a deeper and more nuanced comprehension of visual content across various contexts.