# Sensorless Hand Guidance using Microsoft Hololens

## TL;DR

This paper introduces a novel sensorless hand guidance method for robots using Microsoft Hololens and AR, enabling trajectory programming without joint-torque sensors by leveraging hand tracking and spatial registration.

## Contribution

The paper presents a new sensorless hand guidance approach using AR and hand tracking, eliminating the need for joint-torque sensors in robot programming.

## Key findings

- Successfully tested on KUKA KR-5 robot
- Achieved sensorless hand guidance with AR device
- Demonstrated practical applicability in industrial robots

## Abstract

Hand guidance of robots has proven to be a useful tool both for programming trajectories and in kinesthetic teaching. However hand guidance is usually relegated to robots possessing joint-torque sensors (JTS). Here we propose to extend hand guidance to robots lacking those sensors through the use of an Augmented Reality (AR) device, namely Microsoft's Hololens. Augmented reality devices have been envisioned as a helpful addition to ease both robot programming and increase situational awareness of humans working in close proximity to robots. We reference the robot by using a registration algorithm to match a robot model to the spatial mesh. The in-built hand tracking capabilities are then used to calculate the position of the hands relative to the robot. By decomposing the hand movements into orthogonal rotations we achieve a completely sensorless hand guidance without any need to build a dynamic model of the robot itself. We did the first tests our approach on a commonly used industrial manipulator, the KUKA KR-5.