Visual Recognition by Request

TL;DR

This paper introduces ViRReq, a new visual recognition paradigm that enables flexible, hierarchical, and request-based recognition by decomposing tasks into atomic requests and utilizing a knowledge base, improving recognition of complex and new concepts.

Contribution

The paper proposes a novel request-based recognition framework that learns hierarchical structures from incomplete data and easily incorporates new concepts, advancing visual recognition capabilities.

Findings

Effective recognition of hierarchical structures on CPP and ADE20K datasets.

Ability to learn from incomplete annotations and add new concepts with minimal effort.

Integration of language-driven recognition into segmentation methods.

Abstract

Humans have the ability of recognizing visual semantics in an unlimited granularity, but existing visual recognition algorithms cannot achieve this goal. In this paper, we establish a new paradigm named visual recognition by request (ViRReq) to bridge the gap. The key lies in decomposing visual recognition into atomic tasks named requests and leveraging a knowledge base, a hierarchical and text-based dictionary, to assist task definition. ViRReq allows for (i) learning complicated whole-part hierarchies from highly incomplete annotations and (ii) inserting new concepts with minimal efforts. We also establish a solid baseline by integrating language-driven recognition into recent semantic and instance segmentation methods, and demonstrate its flexible recognition ability on CPP and ADE20K, two datasets with hierarchical whole-part annotations.