Scene Editing as Teleoperation: A Case Study in 6DoF Kit Assembly

TL;DR

This paper introduces SEaT, a scene-centric teleoperation framework that enables non-experts to perform complex 6DoF robot assembly tasks by manipulating digital twins of objects, improving efficiency and success rates.

Contribution

The paper presents a novel scene-centric teleoperation approach that replaces robot control with object manipulation in a virtual scene, making teleoperation accessible to non-experts.

Findings

Improved task success rate in simulation and real-world tests.

Higher user satisfaction and lower workload reported in user study.

Effective scene completion and action refinement algorithms demonstrated.

Abstract

Studies in robot teleoperation have been centered around action specifications -- from continuous joint control to discrete end-effector pose control. However, these robot-centric interfaces often require skilled operators with extensive robotics expertise. To make teleoperation accessible to non-expert users, we propose the framework "Scene Editing as Teleoperation" (SEaT), where the key idea is to transform the traditional "robot-centric" interface into a "scene-centric" interface -- instead of controlling the robot, users focus on specifying the task's goal by manipulating digital twins of the real-world objects. As a result, a user can perform teleoperation without any expert knowledge of the robot hardware. To achieve this goal, we utilize a category-agnostic scene-completion algorithm that translates the real-world workspace (with unknown objects) into a manipulable virtual scene representation and an action-snapping algorithm that refines the user input before generating the robot's action plan. To train the algorithms, we procedurally generated a large-scale, diverse kit-assembly dataset that contains object-kit pairs that mimic real-world object-kitting tasks. Our experiments in simulation and on a real-world system demonstrate that our framework improves both the efficiency and success rate for 6DoF kit-assembly tasks. A user study demonstrates that SEaT framework participants achieve a higher task success rate and report a lower subjective workload compared to an alternative robot-centric interface. Video can be found at https://www.youtube.com/watch?v=-NdR3mkPbQQ .