Explainable Action Prediction through Self-Supervision on Scene Graphs

TL;DR

This paper introduces a self-supervised method for predicting future driver actions using scene graphs, emphasizing interpretability through attention mechanisms that generate spatial and temporal explanations, and demonstrates improved performance over fully-supervised models.

Contribution

It presents a novel self-supervised framework for action prediction with scene graphs, enhancing interpretability and addressing data scarcity in autonomous driving.

Findings

Outperforms fully-supervised approaches on the ROAD dataset

Generates interpretable spatial and temporal heatmaps

Effectively handles data imbalance in driver action prediction

Abstract

This work explores scene graphs as a distilled representation of high-level information for autonomous driving, applied to future driver-action prediction. Given the scarcity and strong imbalance of data samples, we propose a self-supervision pipeline to infer representative and well-separated embeddings. Key aspects are interpretability and explainability; as such, we embed in our architecture attention mechanisms that can create spatial and temporal heatmaps on the scene graphs. We evaluate our system on the ROAD dataset against a fully-supervised approach, showing the superiority of our training regime.