# Two-Stage Transfer Learning for Heterogeneous Robot Detection and 3D Joint Position Estimation in a 2D Camera Image using CNN

## TL;DR

This paper introduces a two-stage transfer learning method for real-time detection and 3D joint position estimation of heterogeneous robots using only 2D images, eliminating the need for Eye-to-Hand calibration.

## Contribution

It presents a novel two-stage transfer learning approach to adapt a multi-objective CNN for different robot types with minimal training data, enabling flexible and calibration-free collision detection.

## Key findings

- Real-time robot detection with CNN in 2D images.
- Effective adaptation to multiple robot types with small datasets.
- Potential applications in flexible robot safety systems.

## Abstract

Collaborative robots are becoming more common on factory floors as well as regular environments, however, their safety still is not a fully solved issue. Collision detection does not always perform as expected and collision avoidance is still an active research area. Collision avoidance works well for fixed robot-camera setups, however, if they are shifted around, Eye-to-Hand calibration becomes invalid making it difficult to accurately run many of the existing collision avoidance algorithms. We approach the problem by presenting a stand-alone system capable of detecting the robot and estimating its position, including individual joints, by using a simple 2D colour image as an input, where no Eye-to-Hand calibration is needed. As an extension of previous work, a two-stage transfer learning approach is used to re-train a multi-objective convolutional neural network (CNN) to allow it to be used with heterogeneous robot arms. Our method is capable of detecting the robot in real-time and new robot types can be added by having significantly smaller training datasets compared to the requirements of a fully trained network. We present data collection approach, the structure of the multi-objective CNN, the two-stage transfer learning training and test results by using real robots from Universal Robots, Kuka, and Franka Emika. Eventually, we analyse possible application areas of our method together with the possible improvements.