HandyNet: A One-stop Solution to Detect, Segment, Localize & Analyze Driver Hands

TL;DR

HandyNet is a comprehensive CNN that detects, segments, localizes in 3D, and identifies objects held by driver hands, using a novel efficient annotation method for training data generation.

Contribution

This work introduces HandyNet, a multi-task CNN for driver hand analysis and a chroma-keying based annotation method for efficient training data creation.

Findings

Successfully detects and localizes driver hands in 3D

Identifies objects held by driver hands

Uses a cost-effective annotation process

Abstract

Tasks related to human hands have long been part of the computer vision community. Hands being the primary actuators for humans, convey a lot about activities and intents, in addition to being an alternative form of communication/interaction with other humans and machines. In this study, we focus on training a single feedforward convolutional neural network (CNN) capable of executing many hand related tasks that may be of use in autonomous and semi-autonomous vehicles of the future. The resulting network, which we refer to as HandyNet, is capable of detecting, segmenting and localizing (in 3D) driver hands inside a vehicle cabin. The network is additionally trained to identify handheld objects that the driver may be interacting with. To meet the data requirements to train such a network, we propose a method for cheap annotation based on chroma-keying, thereby bypassing weeks of human effort required to label such data. This process can generate thousands of labeled training samples in an efficient manner, and may be replicated in new environments with relative ease.