Video Stream Retrieval of Unseen Queries using Semantic Memory
Spencer Cappallo, Thomas Mensink, Cees G. M. Snoek

TL;DR
This paper introduces a novel approach for live video stream retrieval using semantic memory, enabling real-time search of unseen queries by leveraging pre-trained classifiers and adaptive memory techniques.
Contribution
It proposes a no-example, semantic relatedness-based retrieval method with memory pooling and welling to handle shifting video content in live streams.
Findings
Effective retrieval on large-scale video datasets
Successful handling of real-time, unseen queries
Improved performance over traditional methods
Abstract
Retrieval of live, user-broadcast video streams is an under-addressed and increasingly relevant challenge. The on-line nature of the problem requires temporal evaluation and the unforeseeable scope of potential queries motivates an approach which can accommodate arbitrary search queries. To account for the breadth of possible queries, we adopt a no-example approach to query retrieval, which uses a query's semantic relatedness to pre-trained concept classifiers. To adapt to shifting video content, we propose memory pooling and memory welling methods that favor recent information over long past content. We identify two stream retrieval tasks, instantaneous retrieval at any particular time and continuous retrieval over a prolonged duration, and propose means for evaluating them. Three large scale video datasets are adapted to the challenge of stream retrieval. We report results for our…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Image and Video Retrieval Techniques · Multimodal Machine Learning Applications · Video Analysis and Summarization
