Learning Concept-Based Causal Transition and Symbolic Reasoning for Visual Planning

TL;DR

This paper introduces a novel interpretable framework for visual planning that combines concept abstraction, symbolic reasoning, and causal transition modeling to improve task planning in complex visual environments.

Contribution

It proposes a new framework integrating concept learning, symbolic reasoning, and causal transition modeling for visual planning, with a large-scale dataset and demonstrated generalization.

Findings

Outperforms existing methods on the CCTP dataset

Generalizes to unseen tasks and object categories

Effective in real-world visual planning scenarios

Abstract

Visual planning simulates how humans make decisions to achieve desired goals in the form of searching for visual causal transitions between an initial visual state and a final visual goal state. It has become increasingly important in egocentric vision with its advantages in guiding agents to perform daily tasks in complex environments. In this paper, we propose an interpretable and generalizable visual planning framework consisting of i) a novel Substitution-based Concept Learner (SCL) that abstracts visual inputs into disentangled concept representations, ii) symbol abstraction and reasoning that performs task planning via the self-learned symbols, and iii) a Visual Causal Transition model (ViCT) that grounds visual causal transitions to semantically similar real-world actions. Given an initial state, we perform goal-conditioned visual planning with a symbolic reasoning method fueled by the learned representations and causal transitions to reach the goal state. To verify the effectiveness of the proposed model, we collect a large-scale visual planning dataset based on AI2-THOR, dubbed as CCTP. Extensive experiments on this challenging dataset demonstrate the superior performance of our method in visual task planning. Empirically, we show that our framework can generalize to unseen task trajectories, unseen object categories, and real-world data. Further details of this work are provided at https://fqyqc.github.io/ConTranPlan/.