Cascaded Mutual Modulation for Visual Reasoning
Yiqun Yao, Jiaming Xu, Feng Wang, Bo Xu

TL;DR
The paper introduces CMM, a novel multi-step visual reasoning model that uses mutual modulation between text and image features, achieving state-of-the-art results on CLEVR and NLVR benchmarks.
Contribution
CMM is the first to incorporate cascaded mutual modulation with multi-step reasoning for improved visual question answering.
Findings
CMM outperforms previous models on CLEVR and NLVR benchmarks.
Multi-step reasoning and visual-guided language modulation are crucial for performance.
Ablation studies validate the effectiveness of the proposed framework.
Abstract
Visual reasoning is a special visual question answering problem that is multi-step and compositional by nature, and also requires intensive text-vision interactions. We propose CMM: Cascaded Mutual Modulation as a novel end-to-end visual reasoning model. CMM includes a multi-step comprehension process for both question and image. In each step, we use a Feature-wise Linear Modulation (FiLM) technique to enable textual/visual pipeline to mutually control each other. Experiments show that CMM significantly outperforms most related models, and reach state-of-the-arts on two visual reasoning benchmarks: CLEVR and NLVR, collected from both synthetic and natural languages. Ablation studies confirm that both our multistep framework and our visual-guided language modulation are critical to the task. Our code is available at https://github.com/FlamingHorizon/CMM-VR.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMultimodal Machine Learning Applications · Advanced Image and Video Retrieval Techniques · Domain Adaptation and Few-Shot Learning
