Mixture of Rationale: Multi-Modal Reasoning Mixture for Visual Question Answering
Tao Li, Linjun Shou, Xuejun Liu

TL;DR
This paper introduces MoR, a multi-modal reasoning approach for zero-shot visual question answering that dynamically generates, retrieves, and fuses multiple rationales, significantly improving accuracy on challenging datasets.
Contribution
MoR is a novel method that mixes multiple rationales using a single frozen VLPM, enhancing reasoning diversity and modality alignment in VQA tasks.
Findings
12.43% accuracy improvement on NLVR2
2.45% accuracy improvement on OKVQA-S
Effective multi-modal reasoning with a single model backbone
Abstract
Zero-shot visual question answering (VQA) is a challenging task that requires reasoning across modalities. While some existing methods rely on a single rationale within the Chain of Thoughts (CoT) framework, they may fall short of capturing the complexity of the VQA problem. On the other hand, some other methods that use multiple rationales may still suffer from low diversity, poor modality alignment, and inefficient retrieval and fusion. In response to these challenges, we propose \emph{Mixture of Rationales (MoR)}, a novel multi-modal reasoning method that mixes multiple rationales for VQA. MoR uses a single frozen Vision-and-Language Pre-trained Models (VLPM) model to {dynamically generate, retrieve and fuse multi-modal thoughts}. We evaluate MoR on two challenging VQA datasets, i.e. NLVR2 and OKVQA, with two representative backbones OFA and VL-T5. MoR achieves a 12.43\% accuracy…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMultimodal Machine Learning Applications · Advanced Image and Video Retrieval Techniques · Constraint Satisfaction and Optimization
MethodsOFA · VL-T5
