Creative Robot Tool Use by Counterfactual Reasoning
M. Tuluhan Akbulut, Varun Satheesh, Ahmed Jaafar, Alper Ahmetoglu, Shane Parr, Aditya Ganeshan, Shivam Vats, George Konidaris

TL;DR
This paper introduces a causal reasoning framework enabling robots to creatively use tools by discovering causal features through simulated experiments and grounding tool use in physics.
Contribution
It presents a novel approach combining causal discovery, counterfactual generation, and keypoint transfer for creative and reliable robot tool use.
Findings
Causal features improve tool selection reliability.
Grounding in physics enhances skill transfer.
The approach outperforms baseline methods in experiments.
Abstract
We propose a causal reasoning framework for creative robot tool use where a suitable tool for a task is correctly identified for use beyond its primary objectives. The proposed framework first discovers the causal relationships between the tool and the task by conducting simulated experiments in a dynamics model. We decouple the causal discovery problem into two complementary components: VLM-based feature suggestion and counterfactual tool generation via targeted geometric and physical feature perturbations. Then, novel objects are classified based on identified causal features, and the tool use skill is transferred via keypoint matching conditioned on the identified causal features. By reconstructing the task in a dynamics model, our approach grounds tool use in the physics of the problem. We illustrate our approach in reaching a distant object with different sticks, scooping candies…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
