Optimizing Visual Question Answering Models for Driving: Bridging the Gap Between Human and Machine Attention Patterns

TL;DR

This paper examines how visual question answering models for autonomous driving can be improved by aligning their attention patterns more closely with human focus, leading to better accuracy and trust.

Contribution

It introduces a filter-based method to optimize VQA model attention mechanisms, bridging the gap between human and machine attention patterns in driving scenarios.

Findings

Enhanced model accuracy with filter integration

Attention patterns more aligned with human focus

Improved feature prioritization in VQA models

Abstract

Visual Question Answering (VQA) models play a critical role in enhancing the perception capabilities of autonomous driving systems by allowing vehicles to analyze visual inputs alongside textual queries, fostering natural interaction and trust between the vehicle and its occupants or other road users. This study investigates the attention patterns of humans compared to a VQA model when answering driving-related questions, revealing disparities in the objects observed. We propose an approach integrating filters to optimize the model's attention mechanisms, prioritizing relevant objects and improving accuracy. Utilizing the LXMERT model for a case study, we compare attention patterns of the pre-trained and Filter Integrated models, alongside human answers using images from the NuImages dataset, gaining insights into feature prioritization. We evaluated the models using a Subjective scoring framework which shows that the integration of the feature encoder filter has enhanced the performance of the VQA model by refining its attention mechanisms.