Mitigating Hallucinations on Object Attributes using Multiview Images and Negative Instructions

TL;DR

This paper introduces a novel approach using multiview images generated from 3D representations and negative instructions to reduce hallucinations in vision-language models regarding object attributes, improving accuracy.

Contribution

The paper presents MIAVLM, a new model that leverages multiview images and a perceiver module to mitigate hallucinations and bias in LVLMs, a novel combination not previously explored.

Findings

Effective reduction of hallucinations on object attributes.

Improved alignment of multiview visual information with language models.

Demonstrated robustness against input order variations.

Abstract

Current popular Large Vision-Language Models (LVLMs) are suffering from Hallucinations on Object Attributes (HoOA), leading to incorrect determination of fine-grained attributes in the input images. Leveraging significant advancements in 3D generation from a single image, this paper proposes a novel method to mitigate HoOA in LVLMs. This method utilizes multiview images sampled from generated 3D representations as visual prompts for LVLMs, thereby providing more visual information from other viewpoints. Furthermore, we observe the input order of multiple multiview images significantly affects the performance of LVLMs. Consequently, we have devised Multiview Image Augmented VLM (MIAVLM), incorporating a Multiview Attributes Perceiver (MAP) submodule capable of simultaneously eliminating the influence of input image order and aligning visual information from multiview images with Large Language Models (LLMs). Besides, we designed and employed negative instructions to mitigate LVLMs' bias towards ``Yes" responses. Comprehensive experiments demonstrate the effectiveness of our method.