Object Detection Difficulty: Suppressing Over-aggregation for Faster and Better Video Object Detection

TL;DR

This paper introduces an Object Detection Difficulty metric to improve video object detection by selecting better reference frames and reducing unnecessary frame aggregation, resulting in faster and more accurate detection.

Contribution

The paper proposes a novel image-level ODD metric and an ODD Scheduler to enhance VOD accuracy and speed by mitigating over-aggregation and selecting optimal reference frames.

Findings

Improves VOD accuracy by selecting better global reference frames.

Increases FPS by an average of 73.3% without accuracy loss.

Achieves state-of-the-art performance in both speed and accuracy.

Abstract

Current video object detection (VOD) models often encounter issues with over-aggregation due to redundant aggregation strategies, which perform feature aggregation on every frame. This results in suboptimal performance and increased computational complexity. In this work, we propose an image-level Object Detection Difficulty (ODD) metric to quantify the difficulty of detecting objects in a given image. The derived ODD scores can be used in the VOD process to mitigate over-aggregation. Specifically, we train an ODD predictor as an auxiliary head of a still-image object detector to compute the ODD score for each image based on the discrepancies between detection results and ground-truth bounding boxes. The ODD score enhances the VOD system in two ways: 1) it enables the VOD system to select superior global reference frames, thereby improving overall accuracy; and 2) it serves as an indicator in the newly designed ODD Scheduler to eliminate the aggregation of frames that are easy to detect, thus accelerating the VOD process. Comprehensive experiments demonstrate that, when utilized for selecting global reference frames, ODD-VOD consistently enhances the accuracy of Global-frame-based VOD models. When employed for acceleration, ODD-VOD consistently improves the frames per second (FPS) by an average of 73.3% across 8 different VOD models without sacrificing accuracy. When combined, ODD-VOD attains state-of-the-art performance when competing with many VOD methods in both accuracy and speed. Our work represents a significant advancement towards making VOD more practical for real-world applications.