Lightweight Multi-Frame Integration for Robust YOLO Object Detection in Videos

TL;DR

This paper introduces a simple multi-frame stacking method for YOLO-based video object detection that enhances robustness and maintains real-time performance, especially benefiting lightweight models in challenging scenarios.

Contribution

It proposes a minimal modification approach to incorporate temporal context into YOLO detectors, improving detection robustness without increasing complexity.

Findings

Enhanced detection robustness in videos with motion blur and occlusions.

Significant performance gains for lightweight models.

Introduction of the BOAT360 dataset for real-world evaluation.

Abstract

Modern image-based object detection models, such as YOLOv7, primarily process individual frames independently, thus ignoring valuable temporal context naturally present in videos. Meanwhile, existing video-based detection methods often introduce complex temporal modules, significantly increasing model size and computational complexity. In practical applications such as surveillance and autonomous driving, transient challenges including motion blur, occlusions, and abrupt appearance changes can severely degrade single-frame detection performance. To address these issues, we propose a straightforward yet highly effective strategy: stacking multiple consecutive frames as input to a YOLO-based detector while supervising only the output corresponding to a single target frame. This approach leverages temporal information with minimal modifications to existing architectures, preserving simplicity, computational efficiency, and real-time inference capability. Extensive experiments on the challenging MOT20Det and our BOAT360 datasets demonstrate that our method improves detection robustness, especially for lightweight models, effectively narrowing the gap between compact and heavy detection networks. Additionally, we contribute the BOAT360 benchmark dataset, comprising annotated fisheye video sequences captured from a boat, to support future research in multi-frame video object detection in challenging real-world scenarios.