Active Object Detection with Knowledge Aggregation and Distillation from Large Models

TL;DR

This paper introduces a novel active object detection method that leverages knowledge aggregation and distillation from large models, utilizing informed priors about object interactions to improve detection accuracy in complex scenarios.

Contribution

The paper proposes a knowledge aggregation and distillation framework that incorporates object interaction priors to enhance active object detection performance.

Findings

Achieved state-of-the-art results on four datasets.

Improved detection accuracy in scenarios with subtle visual changes.

Reduced reliance on extra knowledge inputs through distillation.

Abstract

Accurately detecting active objects undergoing state changes is essential for comprehending human interactions and facilitating decision-making. The existing methods for active object detection (AOD) primarily rely on visual appearance of the objects within input, such as changes in size, shape and relationship with hands. However, these visual changes can be subtle, posing challenges, particularly in scenarios with multiple distracting no-change instances of the same category. We observe that the state changes are often the result of an interaction being performed upon the object, thus propose to use informed priors about object related plausible interactions (including semantics and visual appearance) to provide more reliable cues for AOD. Specifically, we propose a knowledge aggregation procedure to integrate the aforementioned informed priors into oracle queries within the teacher decoder, offering more object affordance commonsense to locate the active object. To streamline the inference process and reduce extra knowledge inputs, we propose a knowledge distillation approach that encourages the student decoder to mimic the detection capabilities of the teacher decoder using the oracle query by replicating its predictions and attention. Our proposed framework achieves state-of-the-art performance on four datasets, namely Ego4D, Epic-Kitchens, MECCANO, and 100DOH, which demonstrates the effectiveness of our approach in improving AOD.