Ground-truth or DAER: Selective Re-query of Secondary Information

TL;DR

This paper introduces the seed rejection problem in vision tasks, proposing a method to determine when to reject noisy secondary information to improve model performance and reduce review efforts.

Contribution

It formalizes seed rejection, proposes a novel training method and metrics, and demonstrates effectiveness on viewpoint estimation and classification tasks.

Findings

Reduces seed review requirements by over 23%.

Provides a formal definition and evaluation framework for seed rejection.

Shows improved performance over strong baselines.

Abstract

Many vision tasks use secondary information at inference time -- a seed -- to assist a computer vision model in solving a problem. For example, an initial bounding box is needed to initialize visual object tracking. To date, all such work makes the assumption that the seed is a good one. However, in practice, from crowdsourcing to noisy automated seeds, this is often not the case. We hence propose the problem of seed rejection -- determining whether to reject a seed based on the expected performance degradation when it is provided in place of a gold-standard seed. We provide a formal definition to this problem, and focus on two meaningful subgoals: understanding causes of error and understanding the model's response to noisy seeds conditioned on the primary input. With these goals in mind, we propose a novel training method and evaluation metrics for the seed rejection problem. We then use seeded versions of the viewpoint estimation and fine-grained classification tasks to evaluate these contributions. In these experiments, we show our method can reduce the number of seeds that need to be reviewed for a target performance by over 23% compared to strong baselines.