PFL-LSTR: A privacy-preserving framework for driver intention inference based on in-vehicle and out-vehicle information

TL;DR

This paper presents PFL-LSTR, a privacy-preserving federated learning framework using long short-term transformers to accurately infer driver intentions from in-vehicle and out-of-vehicle videos, enhancing traffic safety without compromising privacy.

Contribution

It introduces a novel personalized federated learning model with a long short-term transformer for driver intention prediction using multimodal video data, ensuring privacy and high accuracy.

Findings

High adaptability and precision of the model.

Out-of-vehicle information, especially rear-mirror actions, improves inference accuracy.

Effective in real-world driving scenarios.

Abstract

Intelligent vehicle anticipation of the movement intentions of other drivers can reduce collisions. Typically, when a human driver of another vehicle (referred to as the target vehicle) engages in specific behaviors such as checking the rearview mirror prior to lane change, a valuable clue is therein provided on the intentions of the target vehicle's driver. Furthermore, the target driver's intentions can be influenced and shaped by their driving environment. For example, if the target vehicle is too close to a leading vehicle, it may renege the lane change decision. On the other hand, a following vehicle in the target lane is too close to the target vehicle could lead to its reversal of the decision to change lanes. Knowledge of such intentions of all vehicles in a traffic stream can help enhance traffic safety. Unfortunately, such information is often captured in the form of images/videos. Utilization of personally identifiable data to train a general model could violate user privacy. Federated Learning (FL) is a promising tool to resolve this conundrum. FL efficiently trains models without exposing the underlying data. This paper introduces a Personalized Federated Learning (PFL) model embedded a long short-term transformer (LSTR) framework. The framework predicts drivers' intentions by leveraging in-vehicle videos (of driver movement, gestures, and expressions) and out-of-vehicle videos (of the vehicle's surroundings - frontal/rear areas). The proposed PFL-LSTR framework is trained and tested through real-world driving data collected from human drivers at Interstate 65 in Indiana. The results suggest that the PFL-LSTR exhibits high adaptability and high precision, and that out-of-vehicle information (particularly, the driver's rear-mirror viewing actions) is important because it helps reduce false positives and thereby enhances the precision of driver intention inference.