Creative Robot Tool Use by Counterfactual Reasoning

TL;DR

This paper introduces a causal reasoning framework enabling robots to creatively use tools by discovering causal features through simulated experiments and grounding tool use in physics.

Contribution

It presents a novel approach combining causal discovery, counterfactual generation, and keypoint transfer for creative and reliable robot tool use.

Findings

Causal features improve tool selection reliability.

Grounding in physics enhances skill transfer.

The approach outperforms baseline methods in experiments.

Abstract

We propose a causal reasoning framework for creative robot tool use where a suitable tool for a task is correctly identified for use beyond its primary objectives. The proposed framework first discovers the causal relationships between the tool and the task by conducting simulated experiments in a dynamics model. We decouple the causal discovery problem into two complementary components: VLM-based feature suggestion and counterfactual tool generation via targeted geometric and physical feature perturbations. Then, novel objects are classified based on identified causal features, and the tool use skill is transferred via keypoint matching conditioned on the identified causal features. By reconstructing the task in a dynamics model, our approach grounds tool use in the physics of the problem. We illustrate our approach in reaching a distant object with different sticks, scooping candies from a bowl using diverse items, and using different boxes or crates as stepping platforms to retrieve an object from a high shelf. Our baseline comparisons show that identifying causal features and grounding them in physical tool properties leads to more reliable tool selection and stronger skill keypoint transfer.