Calibrating Concepts and Operations: Towards Symbolic Reasoning on Real Images
Zhuowan Li, Elias Stengel-Eskin, Yixiao Zhang, Cihang Xie, Quan Tran,, Benjamin Van Durme, Alan Yuille

TL;DR
This paper introduces Calibrating Concepts and Operations (CCO), a new method that improves neural symbolic reasoning on real images by addressing distribution imbalance and importance of reasoning steps, significantly enhancing performance on the GQA dataset.
Contribution
The paper proposes CCO, a novel framework with learnable concept embeddings and operation calibrator, to better handle real-world visual data in neural symbolic reasoning models.
Findings
CCO boosts neural symbolic model performance on real images.
NSCL with CCO outperforms vanilla NSCL by 9.1% on GQA.
Reduces the gap between symbolic and non-symbolic methods.
Abstract
While neural symbolic methods demonstrate impressive performance in visual question answering on synthetic images, their performance suffers on real images. We identify that the long-tail distribution of visual concepts and unequal importance of reasoning steps in real data are the two key obstacles that limit the models' real-world potentials. To address these challenges, we propose a new paradigm, Calibrating Concepts and Operations (CCO), which enables neural symbolic models to capture underlying data characteristics and to reason with hierarchical importance. Specifically, we introduce an executor with learnable concept embedding magnitudes for handling distribution imbalance, and an operation calibrator for highlighting important operations and suppressing redundant ones. Our experiments show CCO substantially boosts the performance of neural symbolic methods on real images. By…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMultimodal Machine Learning Applications · Advanced Image and Video Retrieval Techniques · Domain Adaptation and Few-Shot Learning
MethodsTest
