Detecting Features of Tools, Objects, and Actions from Effects in a Robot using Deep Learning

TL;DR

This paper presents a deep learning-based model enabling robots to identify features of tools, objects, and actions from observed effects, facilitating more autonomous and adaptive tool use through sensory-motor learning.

Contribution

It introduces a novel deep learning approach that learns from effects to detect features of tools, objects, and actions, inspired by infant learning principles.

Findings

Robot can predict images and motions from effects with unknown tools and objects.

Model successfully detects features of tools, objects, and actions from effects.

Demonstrates effective sensory-motor learning for tool use in robots.

Abstract

We propose a tool-use model that can detect the features of tools, target objects, and actions from the provided effects of object manipulation. We construct a model that enables robots to manipulate objects with tools, using infant learning as a concept. To realize this, we train sensory-motor data recorded during a tool-use task performed by a robot with deep learning. Experiments include four factors: (1) tools, (2) objects, (3) actions, and (4) effects, which the model considers simultaneously. For evaluation, the robot generates predicted images and motions given information of the effects of using unknown tools and objects. We confirm that the robot is capable of detecting features of tools, objects, and actions by learning the effects and executing the task.