Applying Learning-from-observation to household service robots: three common-sense formulation

TL;DR

This paper introduces a learning-from-observation system for household robots that uses common sense representations like labanotation, contact-webs, and constraints to generate robot programs from human demonstrations, reducing programming effort.

Contribution

It proposes novel common sense representations and task models for LfO in cluttered household environments, enabling robots to learn tasks by observing humans.

Findings

Effective task models generated from demonstrations

Successful execution of learned tasks on robot hardware

Demonstrated system works in various household scenes

Abstract

Utilizing a robot in a new application requires the robot to be programmed at each time. To reduce such programmings efforts, we have been developing ``Learning-from-observation (LfO)'' that automatically generates robot programs by observing human demonstrations. One of the main issues with introducing this LfO system into the domain of household tasks is the cluttered environments, which cause difficulty in determining which elements are important for task execution when observing demonstrations. To overcome this issue, it is necessary for the system to have common sense shared with the human demonstrator. This paper addresses three relationships that LfO in the household domain should focus on when observing demonstrations and proposes representations to describe the common sense used by the demonstrator for optimal execution of task sequences. Specifically, the paper proposes to use labanotation to describe the postures between the environment and the robot, contact-webs to describe the grasping methods between the robot and the tool, and physical and semantic constraints to describe the motions between the tool and the environment. Then, based on these representations, the paper formulates task models, machine-independent robot programs, that indicate what to do and how to do. Third, the paper explains the task encoder to obtain task models and task decoder to execute the task models on the robot hardware. Finally, this paper presents how the system actually works through several example scenes.