Temporal Object Captioning for Street Scene Videos from LiDAR Tracks

TL;DR

This paper introduces a LiDAR-based captioning method that enhances temporal understanding in street scene videos, improving model performance by focusing on traffic dynamics and reducing static biases.

Contribution

It presents a novel rule-based, template-driven captioning approach using LiDAR data to improve temporal semantics in video captioning for driving scenarios.

Findings

Training SwinBERT with LiDAR-based captions improves temporal understanding.

LiDAR supervision reduces static biases in captioning models.

Method outperforms baseline models across three datasets.

Abstract

Video captioning models have seen notable advancements in recent years, especially with regard to their ability to capture temporal information. While many research efforts have focused on architectural advancements, such as temporal attention mechanisms, there remains a notable gap in understanding how models capture and utilize temporal semantics for effective temporal feature extraction, especially in the context of Advanced Driver Assistance Systems. We propose an automated LiDAR-based captioning procedure that focuses on the temporal dynamics of traffic participants. Our approach uses a rule-based system to extract essential details such as lane position and relative motion from object tracks, followed by a template-based caption generation. Our findings show that training SwinBERT, a video captioning model, using only front camera images and supervised with our template-based captions, specifically designed to encapsulate fine-grained temporal behavior, leads to improved temporal understanding consistently across three datasets. In conclusion, our results clearly demonstrate that integrating LiDAR-based caption supervision significantly enhances temporal understanding, effectively addressing and reducing the inherent visual/static biases prevalent in current state-of-the-art model architectures.