SUTD-TrafficQA: A Question Answering Benchmark and an Efficient Network for Video Reasoning over Traffic Events

TL;DR

This paper introduces SUTD-TrafficQA, a large-scale video question answering benchmark for traffic event reasoning, and proposes Eclipse, an efficient network that improves reasoning performance while reducing computational costs.

Contribution

The paper presents a new traffic video QA dataset with diverse reasoning tasks and introduces Eclipse, a novel efficient network for reliable and cost-effective video reasoning.

Findings

Eclipse achieves superior reasoning performance.

The dataset enables benchmarking of causal inference in traffic videos.

The method significantly reduces computation costs.

Abstract

Traffic event cognition and reasoning in videos is an important task that has a wide range of applications in intelligent transportation, assisted driving, and autonomous vehicles. In this paper, we create a novel dataset, SUTD-TrafficQA (Traffic Question Answering), which takes the form of video QA based on the collected 10,080 in-the-wild videos and annotated 62,535 QA pairs, for benchmarking the cognitive capability of causal inference and event understanding models in complex traffic scenarios. Specifically, we propose 6 challenging reasoning tasks corresponding to various traffic scenarios, so as to evaluate the reasoning capability over different kinds of complex yet practical traffic events. Moreover, we propose Eclipse, a novel Efficient glimpse network via dynamic inference, in order to achieve computation-efficient and reliable video reasoning. The experiments show that our method achieves superior performance while reducing the computation cost significantly. The project page: https://github.com/SUTDCV/SUTD-TrafficQA.