CoTDet: Affordance Knowledge Prompting for Task Driven Object Detection

TL;DR

This paper introduces CoTDet, a novel object detection framework that leverages affordance knowledge and multi-level reasoning from large language models to improve detection of task-relevant objects beyond traditional category-based methods.

Contribution

It proposes a new approach using affordance knowledge and multi-level chain-of-thought prompting to enhance object detection for task-driven scenarios.

Findings

Significant performance improvements over state-of-the-art (+15.6 box AP, +14.8 mask AP)

Ability to generate rationales explaining object affordances

Effective utilization of large language models for knowledge extraction

Abstract

Task driven object detection aims to detect object instances suitable for affording a task in an image. Its challenge lies in object categories available for the task being too diverse to be limited to a closed set of object vocabulary for traditional object detection. Simply mapping categories and visual features of common objects to the task cannot address the challenge. In this paper, we propose to explore fundamental affordances rather than object categories, i.e., common attributes that enable different objects to accomplish the same task. Moreover, we propose a novel multi-level chain-of-thought prompting (MLCoT) to extract the affordance knowledge from large language models, which contains multi-level reasoning steps from task to object examples to essential visual attributes with rationales. Furthermore, to fully exploit knowledge to benefit object recognition and localization, we propose a knowledge-conditional detection framework, namely CoTDet. It conditions the detector from the knowledge to generate object queries and regress boxes. Experimental results demonstrate that our CoTDet outperforms state-of-the-art methods consistently and significantly (+15.6 box AP and +14.8 mask AP) and can generate rationales for why objects are detected to afford the task.