BIMA: Bijective Maximum Likelihood Learning Approach to Hallucination Prediction and Mitigation in Large Vision-Language Models

TL;DR

This paper introduces BIMA, a novel bijective maximum likelihood learning method based on normalizing flows, to effectively reduce hallucinations in large vision-language models, improving their reliability and interpretability.

Contribution

BIMA is the first approach to apply bijective normalizing flow techniques for hallucination mitigation in vision-language models, offering a new direction for trustworthy AI systems.

Findings

Achieves an average F1 score of 85.06% on POPE benchmark

Reduces CHAIRS and CHAIRI hallucination metrics by 7.6% and 2.6%

Demonstrates significant improvements over existing methods

Abstract

Large vision-language models have become widely adopted to advance in various domains. However, developing a trustworthy system with minimal interpretable characteristics of large-scale models presents a significant challenge. One of the most prevalent terms associated with the fallacy functions caused by these systems is hallucination, where the language model generates a response that does not correspond to the visual content. To mitigate this problem, several approaches have been developed, and one prominent direction is to ameliorate the decoding process. In this paper, we propose a new Bijective Maximum Likelihood Learning (BIMA) approach to hallucination mitigation using normalizing flow theories. The proposed BIMA method can efficiently mitigate the hallucination problem in prevailing vision-language models, resulting in significant improvements. Notably, BIMA achieves the average F1 score of 85.06% on POPE benchmark and remarkably reduce CHAIRS and CHAIRI by 7.6% and 2.6%, respectively. To the best of our knowledge, this is one of the first studies that contemplates the bijection means to reduce hallucination induced by large vision-language models.