Bowtie Networks: Generative Modeling for Joint Few-Shot Recognition and Novel-View Synthesis

TL;DR

This paper introduces Bowtie Networks, a joint model for few-shot object recognition and novel-view image synthesis that leverages a feedback loop between generative and discriminative components to improve performance.

Contribution

It presents a novel joint learning framework with a feedback mechanism that enhances both recognition and view synthesis from limited data.

Findings

Synthesized images are realistic from multiple viewpoints.

Recognition performance improves with data augmentation.

Effective in low-data regimes.

Abstract

We propose a novel task of joint few-shot recognition and novel-view synthesis: given only one or few images of a novel object from arbitrary views with only category annotation, we aim to simultaneously learn an object classifier and generate images of that type of object from new viewpoints. While existing work copes with two or more tasks mainly by multi-task learning of shareable feature representations, we take a different perspective. We focus on the interaction and cooperation between a generative model and a discriminative model, in a way that facilitates knowledge to flow across tasks in complementary directions. To this end, we propose bowtie networks that jointly learn 3D geometric and semantic representations with a feedback loop. Experimental evaluation on challenging fine-grained recognition datasets demonstrates that our synthesized images are realistic from multiple viewpoints and significantly improve recognition performance as ways of data augmentation, especially in the low-data regime. Code and pre-trained models are released at https://github.com/zpbao/bowtie_networks.