GraspLook: a VR-based Telemanipulation System with R-CNN-driven Augmentation of Virtual Environment

TL;DR

This paper introduces GraspLook, a VR telemanipulation system enhanced with R-CNN for detecting instruments and augmenting virtual environments, improving operation smoothness and reducing mental load in remote laboratory tasks.

Contribution

It presents a novel VR-based teleoperation system that uses R-CNN for real-time instrument detection and virtual environment augmentation, enhancing remote manipulation efficiency.

Findings

System reduces task execution time

Participants find system less mentally demanding

High user satisfaction with augmented virtual environment

Abstract

The teleoperation of robotic systems in medical applications requires stable and convenient visual feedback for the operator. The most accessible approach to delivering visual information from the remote area is using cameras to transmit a video stream from the environment. However, such systems are sensitive to the camera resolution, limited viewpoints, and cluttered environment bringing additional mental demands to the human operator. The paper proposes a novel system of teleoperation based on an augmented virtual environment (VE). The region-based convolutional neural network (R-CNN) is applied to detect the laboratory instrument and estimate its position in the remote environment to display further its digital twin in the VE, which is necessary for dexterous telemanipulation. The experimental results revealed that the developed system allows users to operate the robot smoother, which leads to a decrease in task execution time when manipulating test tubes. In addition, the participants evaluated the developed system as less mentally demanding (by 11%) and requiring less effort (by 16%) to accomplish the task than the camera-based teleoperation approach and highly assessed their performance in the augmented VE. The proposed technology can be potentially applied for conducting laboratory tests in remote areas when operating with infectious and poisonous reagents.