A Comparative Analysis of Decision-Level Fusion for Multimodal Driver Behaviour Understanding

TL;DR

This paper systematically evaluates various decision-level fusion strategies for multimodal driver behavior recognition using video data, aiming to guide the selection of effective fusion methods in vehicle monitoring systems.

Contribution

It provides the first comprehensive empirical comparison of seven late fusion mechanisms for multimodal driver observation, clarifying their relative strengths and weaknesses.

Findings

Score averaging performs robustly across scenarios.

Rank-level fusion offers unique advantages in certain conditions.

The study guides optimal fusion scheme selection for in-vehicle driver monitoring.

Abstract

Visual recognition inside the vehicle cabin leads to safer driving and more intuitive human-vehicle interaction but such systems face substantial obstacles as they need to capture different granularities of driver behaviour while dealing with highly limited body visibility and changing illumination. Multimodal recognition mitigates a number of such issues: prediction outcomes of different sensors complement each other due to different modality-specific strengths and weaknesses. While several late fusion methods have been considered in previously published frameworks, they constantly feature different architecture backbones and building blocks making it very hard to isolate the role of the chosen late fusion strategy itself. This paper presents an empirical evaluation of different paradigms for decision-level late fusion in video-based driver observation. We compare seven different mechanisms for joining the results of single-modal classifiers which have been both popular, (e.g. score averaging) and not yet considered (e.g. rank-level fusion) in the context of driver observation evaluating them based on different criteria and benchmark settings. This is the first systematic study of strategies for fusing outcomes of multimodal predictors inside the vehicles, conducted with the goal to provide guidance for fusion scheme selection.