A Two-stage Transformer Framework for Temporal Localization of Distracted Driver Behaviors

TL;DR

This paper introduces a two-stage transformer-based framework for accurately localizing distracted driver behaviors in in-cabin videos, balancing high performance with computational efficiency for practical driver monitoring applications.

Contribution

The work presents a novel two-stage transformer framework combining VideoMAE features with an augmented self-mask attention detector and multi-scale temporal pooling, tailored for driver behavior localization.

Findings

ViT-Giant + SPPF achieves 92.67% mAP.

Lightweight ViT-based model attains 82.55% accuracy with lower computational cost.

The SPPF module enhances localization performance across configurations.

Abstract

The identification of hazardous driving behaviors from in-cabin video streams is essential for enhancing road safety and supporting the detection of traffic violations and unsafe driver actions. However, current temporal action localization techniques often struggle to balance accuracy with computational efficiency. In this work, we develop and evaluate a temporal action localization framework tailored for driver monitoring scenarios, particularly suitable for periodic inspection settings such as transportation safety checkpoints or fleet management assessment systems. Our approach follows a two-stage pipeline that combines VideoMAE-based feature extraction with an Augmented Self-Mask Attention (AMA) detector, enhanced by a Spatial Pyramid Pooling-Fast (SPPF) module to capture multi-scale temporal features. Experimental results reveal a distinct trade-off between model capacity and efficiency. At the feature extraction stage, the ViT-Giant backbone delivers higher representations with 88.09% Top-1 test accuracy, while the ViT-based variant proves to be a practical alternative, achieving 82.55% accuracy with significantly lower computational fine-tuning costs (101.85 GFLOPs/segment compared to 1584.06 GFLOPs/segment for Giant). In the downstream localization task, the integration of SPPF consistently improves performance across all configurations. Notably, the ViT-Giant + SPPF model achieves a peak mAP of 92.67%, while the lightweight ViT-based configuration maintains robust results.