Where is my Wallet? Modeling Object Proposal Sets for Egocentric Visual Query Localization

TL;DR

This paper introduces CocoFormer, a transformer-based module that enhances egocentric visual query localization by addressing dataset biases, expanding annotations, and considering object proposals context, leading to state-of-the-art results.

Contribution

The paper proposes CocoFormer, a novel transformer module that models object proposal sets for improved egocentric query localization, addressing biases and incorporating context.

Findings

Improved frame-level detection AP from 26.28% to 31.26%.

Achieved top rankings in VQ2D and VQ3D tasks at Ego4D challenge.

State-of-the-art results in Few-Shot Detection (FSD).

Abstract

This paper deals with the problem of localizing objects in image and video datasets from visual exemplars. In particular, we focus on the challenging problem of egocentric visual query localization. We first identify grave implicit biases in current query-conditioned model design and visual query datasets. Then, we directly tackle such biases at both frame and object set levels. Concretely, our method solves these issues by expanding limited annotations and dynamically dropping object proposals during training. Additionally, we propose a novel transformer-based module that allows for object-proposal set context to be considered while incorporating query information. We name our module Conditioned Contextual Transformer or CocoFormer. Our experiments show the proposed adaptations improve egocentric query detection, leading to a better visual query localization system in both 2D and 3D configurations. Thus, we are able to improve frame-level detection performance from 26.28% to 31.26 in AP, which correspondingly improves the VQ2D and VQ3D localization scores by significant margins. Our improved context-aware query object detector ranked first and second in the VQ2D and VQ3D tasks in the 2nd Ego4D challenge. In addition to this, we showcase the relevance of our proposed model in the Few-Shot Detection (FSD) task, where we also achieve SOTA results. Our code is available at https://github.com/facebookresearch/vq2d_cvpr.