Boosting Visual Knowledge-Intensive Training for LVLMs Through Causality-Driven Visual Object Completion

TL;DR

This paper introduces a causality-driven visual object completion task to enhance LVLMs' visual perception by self-improving through automated instance creation and trial-and-error learning, leading to significant performance gains.

Contribution

The paper proposes a novel self-improvement framework using causality-driven visual object completion to boost LVLMs' visual knowledge and reasoning capabilities.

Findings

Achieved an average of 5.4% and 4.0% improvements on specialized tasks with LLaVA models.

Demonstrated substantial gains across four challenging tasks and four benchmarks.

Utilized automated instance construction without human or GPT-4V assistance.

Abstract

Large Vision-Language Models (LVLMs) have experienced significant advancements in recent years. However, their performance still falls short in tasks requiring deep visual perception, such as identifying subtle differences between images. A potential cause is the scarcity of visual knowledge in popular instruction-tuning corpora, resulting in inadequate visual perception and reasoning capabilities. To address this challenge, we introduce a self-improvement framework grounded in a novel visual knowledge-intensive task, \underline{C}ausality-driven \underline{V}isual object \underline{C}ompletion (CVC). This task requires LVLMs to infer the masked object in an image based on its \textit{causal} relationships with the other visible information. We first obtain rich examples cheaply through our automated instance construction pipeline, without relying on sophisticated LVLMs (\textit{e.g.}, GPT-4V) or human assistance. Then, LVLMs effectively self-improve through trial and error learning using these created instances. Our experiments demonstrate substantial gains across four challenging specialized tasks and four widely-used comprehensive benchmarks. Especially on specialized tasks, our method achieves an average improvement of 5.4\% and 4.0\% compared to the corresponding baselines when utilizing LLaVA-1.5-7B and LLaVA-1.5-13B, respectively. The code is available at https://github.com/XMUDeepLIT/CVC.