Real-time Detection of 2D Tool Landmarks with Synthetic Training Data

TL;DR

This paper introduces a real-time deep learning method for detecting 2D tool landmarks using synthetic training data, employing advanced rendering and transfer learning to generalize well to real images.

Contribution

The paper presents the Intermediate Heatmap Model (IHM), a novel architecture that effectively trains on synthetic data to detect tool landmarks in real images, even for unseen tools.

Findings

IHM outperforms existing keypoint detection methods trained on synthetic data.

The model generalizes to unseen tools without needing exact textured 3D models.

Synthetic training combined with advanced rendering achieves high accuracy on real images.

Abstract

In this paper a deep learning architecture is presented that can, in real time, detect the 2D locations of certain landmarks of physical tools, such as a hammer or screwdriver. To avoid the labor of manual labeling, the network is trained on synthetically generated data. Training computer vision models on computer generated images, while still achieving good accuracy on real images, is a challenge due to the difference in domain. The proposed method uses an advanced rendering method in combination with transfer learning and an intermediate supervision architecture to address this problem. It is shown that the model presented in this paper, named Intermediate Heatmap Model (IHM), generalizes to real images when trained on synthetic data. To avoid the need for an exact textured 3D model of the tool in question, it is shown that the model will generalize to an unseen tool when trained on a set of different 3D models of the same type of tool. IHM is compared to two existing approaches to keypoint detection and it is shown that it outperforms those at detecting tool landmarks, trained on synthetic data.