Task Presentation and Human Perception in Interactive Video Retrieval

TL;DR

This study explores how different task presentation methods in interactive video retrieval affect human perception and retrieval success, highlighting the importance of human factors in evaluation protocols.

Contribution

The paper introduces novel task presentation modes based on media memorability and evaluates their impact through large-scale crowdsourced experiments.

Findings

Presentation style significantly influences retrieval difficulty.

Participants can successfully retrieve videos even with reduced or altered hints.

Results suggest reconsidering evaluation protocols to better account for human perception.

Abstract

Interactive video retrieval is a cooperative process between humans and retrieval systems. Large-scale evaluation campaigns, however, often overlook human factors, such as the effects of perception, attention, and memory, when assessing media retrieval systems. Consequently, their setups fall short of emulating realistic retrieval scenarios. In this paper, we design novel task presentation modes based on concepts in media memorability, implement the pipelines necessary for processing target video segments, and build a custom experimental platform for the final evaluation. In order to study the effects of different task representation schemes, we conduct a large crowdsourced experiment. Our findings demonstrate that the way in which the target of a video retrieval task is presented has a substantial influence on the difficulty of the retrieval task and that individuals can successfully retrieve a target video segment despite reducing or even altering the provided hints, opening up a discussion around future evaluation protocols in the domain of interactive media retrieval.