# Teleoperation System for Service Robots Using a Virtual Reality Headset and 3D Pose Estimation

## TL;DR

This paper introduces a VR-based system for controlling service robots using 3D pose estimation, enabling intuitive and immersive teleoperation.

## Contribution

The novel framework combines VR and 3D pose estimation for low-cost, sensor-driven teleoperation of service robots.

## Key findings

- The system enables reliable arm teleoperation and effective telepresence for real-world object grasping.
- Vision-only estimation struggles with axial rotations like elbow and wrist yaw under occlusions and poor viewpoints.
- The proposed system offers a practical solution for immersive human–robot interaction in dynamic environments.

## Abstract

This paper presents an immersive teleoperation framework for service robots that combines real-time 3D human pose estimation with a Virtual Reality (VR) interface to support intuitive, natural robot control. The operator is tracked using MediaPipe for 2D landmark detection and an Intel RealSense D455 RGB-D (Red-Green-Blue plus Depth) camera for depth acquisition, enabling 3D reconstruction of key joints. Joint angles are computed using efficient vector operations and mapped to the kinematic constraints of an anthropomorphic arm on the CHARMIE service robot. A VR-based telepresence interface provides stereoscopic video and head-motion-based view control to improve situational awareness during manipulation tasks. Experiments in real-world object grasping demonstrate reliable arm teleoperation and effective telepresence; however, vision-only estimation remains limited for axial rotations (e.g., elbow and wrist yaw), particularly under occlusions and unfavorable viewpoints. The proposed system provides a practical pathway toward low-cost, sensor-driven, immersive human–robot interaction for service robotics in dynamic environments.