Recognising Affordances in Predicted Futures to Plan with Consideration of Non-canonical Affordance Effects

TL;DR

This paper introduces a system that recognizes affordances in predicted futures to improve multi-step action planning by accounting for both canonical and non-canonical effects, enhancing robustness and goal achievement.

Contribution

It presents a novel approach combining affordance recognition with a neural forward model to handle non-canonical effects in planning, learned from experience data.

Findings

System effectively predicts and utilizes non-canonical affordance effects.

Improves planning robustness by avoiding failures caused by non-standard effects.

Validated in simulation on tasks requiring complex affordance considerations.

Abstract

We propose a novel system for action sequence planning based on a combination of affordance recognition and a neural forward model predicting the effects of affordance execution. By performing affordance recognition on predicted futures, we avoid reliance on explicit affordance effect definitions for multi-step planning. Because the system learns affordance effects from experience data, the system can foresee not just the canonical effects of an affordance, but also situation-specific side-effects. This allows the system to avoid planning failures due to such non-canonical effects, and makes it possible to exploit non-canonical effects for realising a given goal. We evaluate the system in simulation, on a set of test tasks that require consideration of canonical and non-canonical affordance effects.