Prototypical Region Proposal Networks for Few-Shot Localization and Classification

TL;DR

This paper introduces PRoPnet, an end-to-end model that combines few-shot segmentation and classification to better localize and classify multiple objects in complex, natural images.

Contribution

It proposes a unified framework that integrates prototype-based segmentation with classification for improved few-shot object localization and recognition.

Findings

Improved accuracy on natural scene datasets with multiple objects

Effective localization of relevant objects in busy images

End-to-end training of segmentation and classification stages

Abstract

Recently proposed few-shot image classification methods have generally focused on use cases where the objects to be classified are the central subject of images. Despite success on benchmark vision datasets aligned with this use case, these methods typically fail on use cases involving densely-annotated, busy images: images common in the wild where objects of relevance are not the central subject, instead appearing potentially occluded, small, or among other incidental objects belonging to other classes of potential interest. To localize relevant objects, we employ a prototype-based few-shot segmentation model which compares the encoded features of unlabeled query images with support class centroids to produce region proposals indicating the presence and location of support set classes in a query image. These region proposals are then used as additional conditioning input to few-shot image classifiers. We develop a framework to unify the two stages (segmentation and classification) into an end-to-end classification model -- PRoPnet -- and empirically demonstrate that our methods improve accuracy on image datasets with natural scenes containing multiple object classes.