# Semantic Robot Programming for Goal-Directed Manipulation in Cluttered Scenes

## TL;DR

This paper introduces Semantic Robot Programming (SRP), a method combining demonstration-based programming and semantic mapping, enabling robots to understand and achieve user-defined goals in cluttered scenes through scene graph parsing and adaptive planning.

## Contribution

The paper presents SRP as a novel framework for goal-directed manipulation, integrating scene graph parsing with a new perception method DIGEST for scene understanding from RGBD images.

## Key findings

- SRP enables robots to adapt to different initial scene configurations.
- DIGEST accurately infers scene states from RGBD images in cluttered environments.
- Successful demonstration of SRP in tray-setting task with a real robot.

## Abstract

We present the Semantic Robot Programming (SRP) paradigm as a convergence of robot programming by demonstration and semantic mapping. In SRP, a user can directly program a robot manipulator by demonstrating a snapshot of their intended goal scene in workspace. The robot then parses this goal as a scene graph comprised of object poses and inter-object relations, assuming known object geometries. Task and motion planning is then used to realize the user's goal from an arbitrary initial scene configuration. Even when faced with different initial scene configurations, SRP enables the robot to seamlessly adapt to reach the user's demonstrated goal. For scene perception, we propose the Discriminatively-Informed Generative Estimation of Scenes and Transforms (DIGEST) method to infer the initial and goal states of the world from RGBD images. The efficacy of SRP with DIGEST perception is demonstrated for the task of tray-setting with a Michigan Progress Fetch robot. Scene perception and task execution are evaluated with a public household occlusion dataset and our cluttered scene dataset.