Exploring EEG for Object Detection and Retrieval

TL;DR

This study demonstrates that EEG signals can be effectively used as a relevance feedback mechanism for object detection and retrieval in complex images, outperforming traditional mouse-based feedback in certain conditions.

Contribution

It introduces a novel approach using EEG for relevance feedback in content-based image retrieval with complex datasets, showing its feasibility and advantages over mouse-based methods.

Findings

EEG signals can detect relevant objects in complex images.

Users with prior knowledge or RSVP experience perform better.

EEG-based feedback outperforms mouse-based feedback with limited annotation time.

Abstract

This paper explores the potential for using Brain Computer Interfaces (BCI) as a relevance feedback mechanism in content-based image retrieval. We investigate if it is possible to capture useful EEG signals to detect if relevant objects are present in a dataset of realistic and complex images. We perform several experiments using a rapid serial visual presentation (RSVP) of images at different rates (5Hz and 10Hz) on 8 users with different degrees of familiarization with BCI and the dataset. We then use the feedback from the BCI and mouse-based interfaces to retrieve localized objects in a subset of TRECVid images. We show that it is indeed possible to detect such objects in complex images and, also, that users with previous knowledge on the dataset or experience with the RSVP outperform others. When the users have limited time to annotate the images (100 seconds in our experiments) both interfaces are comparable in performance. Comparing our best users in a retrieval task, we found that EEG-based relevance feedback outperforms mouse-based feedback. The realistic and complex image dataset differentiates our work from previous studies on EEG for image retrieval.