ThermoHands: A Benchmark for 3D Hand Pose Estimation from Egocentric Thermal Images

TL;DR

ThermoHands introduces a thermal image-based benchmark for egocentric 3D hand pose estimation, demonstrating thermal imaging's robustness in challenging real-world conditions and proposing a new transformer-based baseline method.

Contribution

This paper presents the first thermal image-based egocentric 3D hand pose benchmark and a novel dual transformer baseline, advancing robustness in complex scenarios.

Findings

ThermoHands dataset includes multi-view, multi-spectral thermal images from 28 subjects.

ThermFormer outperforms existing methods in thermal 3D hand pose estimation.

ThermFormer demonstrates robustness under challenging environmental conditions.

Abstract

Designing egocentric 3D hand pose estimation systems that can perform reliably in complex, real-world scenarios is crucial for downstream applications. Previous approaches using RGB or NIR imagery struggle in challenging conditions: RGB methods are susceptible to lighting variations and obstructions like handwear, while NIR techniques can be disrupted by sunlight or interference from other NIR-equipped devices. To address these limitations, we present ThermoHands, the first benchmark focused on thermal image-based egocentric 3D hand pose estimation, demonstrating the potential of thermal imaging to achieve robust performance under these conditions. The benchmark includes a multi-view and multi-spectral dataset collected from 28 subjects performing hand-object and hand-virtual interactions under diverse scenarios, accurately annotated with 3D hand poses through an automated process. We introduce a new baseline method, TherFormer, utilizing dual transformer modules for effective egocentric 3D hand pose estimation in thermal imagery. Our experimental results highlight TherFormer's leading performance and affirm thermal imaging's effectiveness in enabling robust 3D hand pose estimation in adverse conditions.