Looking at Hands in Autonomous Vehicles: A ConvNet Approach using Part Affinity Fields

TL;DR

This paper presents a real-time ConvNet-based system for detecting and localizing driver and passenger hands in autonomous vehicles, enhancing safety by monitoring hand positions under challenging lighting conditions.

Contribution

It introduces a modified, lightweight ConvNet trained on a new autonomous driving dataset to accurately estimate hand joints and affinities at 40 fps.

Findings

Achieves over 95% accuracy in joint localization

Operates in real-time at 40 fps

Performs well under varying lighting conditions

Abstract

In the context of autonomous driving, where humans may need to take over in the event where the computer may issue a takeover request, a key step towards driving safety is the monitoring of the hands to ensure the driver is ready for such a request. This work, focuses on the first step of this process, which is to locate the hands. Such a system must work in real-time and under varying harsh lighting conditions. This paper introduces a fast ConvNet approach, based on the work of original work of OpenPose for full body joint estimation. The network is modified with fewer parameters and retrained using our own day-time naturalistic autonomous driving dataset to estimate joint and affinity heatmaps for driver & passenger's wrist and elbows, for a total of 8 joint classes and part affinity fields between each wrist-elbow pair. The approach runs real-time on real-world data at 40 fps on multiple drivers and passengers. The system is extensively evaluated both quantitatively and qualitatively, showing at least 95% detection performance on joint localization and arm-angle estimation.