MM-PoE: Multiple Choice Reasoning via. Process of Elimination using Multi-Modal Models

TL;DR

This paper presents MM-PoE, a novel multi-modal approach that enhances visual reasoning in models by mimicking human elimination strategies, significantly improving performance in visual question-answering tasks across multiple datasets.

Contribution

Introduces MM-PoE, a dual-step elimination method for multi-modal models, enabling improved zero-shot and few-shot visual reasoning beyond traditional independent option evaluation.

Findings

Significant performance gains in zero-shot and few-shot settings

Effective elimination of implausible options improves reasoning accuracy

Broadens application of process of elimination to multi-modal visual reasoning

Abstract

This paper introduces Multiple Choice Reasoning via. Process of Elimination using Multi-Modal models, herein referred to as Multi-Modal Process of Elimination (MM-PoE). This novel methodology is engineered to augment the efficacy of Vision-Language Models (VLMs) in multiple-choice visual reasoning tasks. Diverging from conventional approaches that evaluate each option independently, MM-PoE employs a dual-step scoring paradigm that initially identifies and excludes implausible choices, subsequently concentrating on the most probable remaining options. This method emulates human test-taking strategies, where individuals typically eliminate clearly incorrect answers prior to selecting the optimal response. Our empirical evaluations, conducted across three benchmark datasets, reveal that MM-PoE significantly improves both zero-shot and few-shot performance of contemporary state-of-the-art VLMs. Critically, this approach not only broadens the application of the elimination process to multi-modal contexts but also allows few-shot experiments, thereby addressing two principal limitations concerning usage of PoE only in zero-shot settings and only with a language-only framework. As a result, MM-PoE not only refines the reasoning capabilities of VLMs but also broadens their applicability to complex visual question-answering scenarios. All code and documentation supporting our work are available at https://pypi.org/project/mm-poe/, enabling researchers and practitioners to easily integrate and further develop these techniques.