Interactive Visual Reasoning under Uncertainty

TL;DR

This paper introduces IVRE, an interactive environment for testing artificial agents' reasoning under uncertainty, revealing current AI limitations compared to human reasoning in ambiguous scenarios.

Contribution

The paper presents IVRE, a novel environment for evaluating and understanding artificial agents' ability to reason and experiment under uncertain, ambiguous conditions.

Findings

Modern AI agents underperform humans in IVRE scenarios.

Current learning methods struggle with interactive reasoning under uncertainty.

IVRE highlights the need for advanced reasoning capabilities in AI.

Abstract

One of the fundamental cognitive abilities of humans is to quickly resolve uncertainty by generating hypotheses and testing them via active trials. Encountering a novel phenomenon accompanied by ambiguous cause-effect relationships, humans make hypotheses against data, conduct inferences from observation, test their theory via experimentation, and correct the proposition if inconsistency arises. These iterative processes persist until the underlying mechanism becomes clear. In this work, we devise the IVRE (pronounced as "ivory") environment for evaluating artificial agents' reasoning ability under uncertainty. IVRE is an interactive environment featuring rich scenarios centered around Blicket detection. Agents in IVRE are placed into environments with various ambiguous action-effect pairs and asked to determine each object's role. They are encouraged to propose effective and efficient experiments to validate their hypotheses based on observations and actively gather new information. The game ends when all uncertainties are resolved or the maximum number of trials is consumed. By evaluating modern artificial agents in IVRE, we notice a clear failure of today's learning methods compared to humans. Such inefficacy in interactive reasoning ability under uncertainty calls for future research in building human-like intelligence.