Calibrating Concepts and Operations: Towards Symbolic Reasoning on Real Images

TL;DR

This paper introduces Calibrating Concepts and Operations (CCO), a new method that improves neural symbolic reasoning on real images by addressing distribution imbalance and importance of reasoning steps, significantly enhancing performance on the GQA dataset.

Contribution

The paper proposes CCO, a novel framework with learnable concept embeddings and operation calibrator, to better handle real-world visual data in neural symbolic reasoning models.

Findings

CCO boosts neural symbolic model performance on real images.

NSCL with CCO outperforms vanilla NSCL by 9.1% on GQA.

Reduces the gap between symbolic and non-symbolic methods.

Abstract

While neural symbolic methods demonstrate impressive performance in visual question answering on synthetic images, their performance suffers on real images. We identify that the long-tail distribution of visual concepts and unequal importance of reasoning steps in real data are the two key obstacles that limit the models' real-world potentials. To address these challenges, we propose a new paradigm, Calibrating Concepts and Operations (CCO), which enables neural symbolic models to capture underlying data characteristics and to reason with hierarchical importance. Specifically, we introduce an executor with learnable concept embedding magnitudes for handling distribution imbalance, and an operation calibrator for highlighting important operations and suppressing redundant ones. Our experiments show CCO substantially boosts the performance of neural symbolic methods on real images. By evaluating models on the real world dataset GQA, CCO helps the neural symbolic method NSCL outperforms its vanilla counterpart by 9.1% (from 47.0% to 56.1%); this result also largely reduces the performance gap between symbolic and non-symbolic methods. Additionally, we create a perturbed test set for better understanding and analyzing model performance on real images. Code is available at https://github.com/Lizw14/CaliCO.git .