Mars: Situated Inductive Reasoning in an Open-World Environment

TL;DR

This paper introduces Mars, an interactive environment designed to evaluate and advance situated inductive reasoning in AI, highlighting the challenges faced by current models and proposing reflection-based methods to improve reasoning capabilities.

Contribution

Mars provides a novel benchmark environment for situated inductive reasoning, incorporating game mechanisms that require active interaction and rule derivation in open-world settings.

Findings

RL and LLM methods struggle on Mars benchmark

Reflection-based induction improves reasoning performance

Mars encourages development of adaptive, context-sensitive AI systems

Abstract

Large Language Models (LLMs) trained on massive corpora have shown remarkable success in knowledge-intensive tasks. Yet, most of them rely on pre-stored knowledge. Inducing new general knowledge from a specific environment and performing reasoning with the acquired knowledge -- \textit{situated inductive reasoning}, is crucial and challenging for machine intelligence. In this paper, we design Mars, an interactive environment devised for situated inductive reasoning. It introduces counter-commonsense game mechanisms by modifying terrain, survival setting and task dependency while adhering to certain principles. In Mars, agents need to actively interact with their surroundings, derive useful rules and perform decision-making tasks in specific contexts. We conduct experiments on various RL-based and LLM-based methods, finding that they all struggle on this challenging situated inductive reasoning benchmark. Furthermore, we explore \textit{Induction from Reflection}, where we instruct agents to perform inductive reasoning from history trajectory. The superior performance underscores the importance of inductive reasoning in Mars. Through Mars, we aim to galvanize advancements in situated inductive reasoning and set the stage for developing the next generation of AI systems that can reason in an adaptive and context-sensitive way.