PO-GUISE+: Pose and object guided transformer token selection for efficient driver action recognition

TL;DR

PO-GUISE+ is an efficient multi-task transformer model for distracted driving recognition that leverages pose and object interaction cues to reduce computation while maintaining high accuracy, validated on real-world benchmarks.

Contribution

We propose PO-GUISE+, a novel transformer model that integrates pose and object information for efficient driver action recognition, reducing computational costs significantly.

Findings

Outperforms state-of-the-art on Drive&Act, 100-Driver, and 3MDAD datasets.

Reduces computational demands while maintaining or improving accuracy.

Demonstrates effectiveness on Jetson platform across various configurations.

Abstract

We address the task of identifying distracted driving by analyzing in-car videos using efficient transformers. Although transformer models have achieved outstanding performance in human action recognition tasks, their high computational costs limit their application onboard a vehicle. We introduce POGUISE+, a multi-task video transformer that, given an input clip, predicts the distracted driving action, the driver's pose, and the interacting object. Our enhanced features for token selection are specifically adapted to driver actions by leveraging information about object interaction and the driver's pose. With POGUISE+, we significantly reduce the model's computational demands while maintaining or improving baseline accuracy across various computational budgets. Additionally, to evaluate our model's performance in real-world scenarios, we have developed benchmarks on a Jetson computing platform, demonstrating its effectiveness across different configurations and computational budgets. Our model outperforms current state-of-the-art results on the Drive&Act, 100-Driver, and 3MDAD datasets, while having superior efficiency compared to existing video transformer-based methods.