Leverage Task Context for Object Affordance Ranking

TL;DR

This paper introduces a novel framework that leverages task context to rank object affordances, enabling more accurate object selection for complex tasks in visual scenes, supported by a large-scale dataset and superior experimental results.

Contribution

It proposes a new context-embed group ranking framework with modules for task relation mining and graph group update, addressing the lack of datasets for task-oriented affordance ranking.

Findings

Demonstrates the effectiveness of task context in affordance ranking

Outperforms state-of-the-art models in saliency ranking and multimodal detection

Provides a large-scale dataset for task-oriented affordance analysis

Abstract

Intelligent agents accomplish different tasks by utilizing various objects based on their affordance, but how to select appropriate objects according to task context is not well-explored. Current studies treat objects within the affordance category as equivalent, ignoring that object affordances vary in priority with different task contexts, hindering accurate decision-making in complex environments. To enable agents to develop a deeper understanding of the objects required to perform tasks, we propose to leverage task context for object affordance ranking, i.e., given image of a complex scene and the textual description of the affordance and task context, revealing task-object relationships and clarifying the priority rank of detected objects. To this end, we propose a novel Context-embed Group Ranking Framework with task relation mining module and graph group update module to deeply integrate task context and perform global relative relationship transmission. Due to the lack of such data, we construct the first large-scale task-oriented affordance ranking dataset with 25 common tasks, over 50k images and more than 661k objects. Experimental results demonstrate the feasibility of the task context based affordance learning paradigm and the superiority of our model over state-of-the-art models in the fields of saliency ranking and multimodal object detection. The source code and dataset will be made available to the public.