Streaming Detection of Queried Event Start
Cristobal Eyzaguirre, Eric Tang, Shyamal Buch, Adrien Gaidon, Jiajun, Wu, Juan Carlos Niebles

TL;DR
This paper introduces SDQES, a new task and benchmark for real-time detection of user-defined events in egocentric videos, utilizing multimodal data and efficient transfer learning methods.
Contribution
It proposes a novel streaming event detection task, a benchmark dataset, and adapter-based models for efficient multimodal video understanding in real-time scenarios.
Findings
Adapter-based models outperform traditional methods in streaming detection accuracy.
The benchmark enables evaluation of diverse event detection in egocentric videos.
Efficient online modeling is achieved through parameter-efficient fine-tuning techniques.
Abstract
Robotics, autonomous driving, augmented reality, and many embodied computer vision applications must quickly react to user-defined events unfolding in real time. We address this setting by proposing a novel task for multimodal video understanding-Streaming Detection of Queried Event Start (SDQES). The goal of SDQES is to identify the beginning of a complex event as described by a natural language query, with high accuracy and low latency. We introduce a new benchmark based on the Ego4D dataset, as well as new task-specific metrics to study streaming multimodal detection of diverse events in an egocentric video setting. Inspired by parameter-efficient fine-tuning methods in NLP and for video tasks, we propose adapter-based baselines that enable image-to-video transfer learning, allowing for efficient online video modeling. We evaluate three vision-language backbones and three adapter…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
Taxonomy
TopicsDistributed systems and fault tolerance · Cloud Computing and Resource Management · Advanced Database Systems and Queries
MethodsAdapter
