Recognizing Car Fluents from Video

TL;DR

This paper introduces a hierarchical model for recognizing vehicle states from videos, addressing challenges like occlusion and variation, and provides a new annotated dataset for this task.

Contribution

It proposes a novel spatial-temporal And-Or hierarchical model for car fluent recognition and creates the first annotated dataset for this purpose.

Findings

The model outperforms baseline methods in recognizing car fluents.

The approach improves car part localization accuracy.

The dataset enables future research in vehicle state recognition.

Abstract

Physical fluents, a term originally used by Newton [40], refers to time-varying object states in dynamic scenes. In this paper, we are interested in inferring the fluents of vehicles from video. For example, a door (hood, trunk) is open or closed through various actions, light is blinking to turn. Recognizing these fluents has broad applications, yet have received scant attention in the computer vision literature. Car fluent recognition entails a unified framework for car detection, car part localization and part status recognition, which is made difficult by large structural and appearance variations, low resolutions and occlusions. This paper learns a spatial-temporal And-Or hierarchical model to represent car fluents. The learning of this model is formulated under the latent structural SVM framework. Since there are no publicly related dataset, we collect and annotate a car fluent dataset consisting of car videos with diverse fluents. In experiments, the proposed method outperforms several highly related baseline methods in terms of car fluent recognition and car part localization.