Implicit Neural Representation Facilitates Unified Universal Vision Encoding

TL;DR

This paper introduces a unified model that learns representations useful for both recognition and generation by using an implicit neural representation hyper-network, achieving state-of-the-art performance and generative capabilities with compact embeddings.

Contribution

It presents the first model to unify recognition and generation in a single implicit neural representation framework, enhanced with knowledge distillation.

Findings

Outperforms state-of-the-art in image recognition tasks.

Enables high-quality image generation from tiny embeddings.

Learns a highly compressed, versatile embedding space.

Abstract

Models for image representation learning are typically designed for either recognition or generation. Various forms of contrastive learning help models learn to convert images to embeddings that are useful for classification, detection, and segmentation. On the other hand, models can be trained to reconstruct images with pixel-wise, perceptual, and adversarial losses in order to learn a latent space that is useful for image generation. We seek to unify these two directions with a first-of-its-kind model that learns representations which are simultaneously useful for recognition and generation. We train our model as a hyper-network for implicit neural representation, which learns to map images to model weights for fast, accurate reconstruction. We further integrate our INR hyper-network with knowledge distillation to improve its generalization and performance. Beyond the novel training design, the model also learns an unprecedented compressed embedding space with outstanding performance for various visual tasks. The complete model competes with state-of-the-art results for image representation learning, while also enabling generative capabilities with its high-quality tiny embeddings. The code is available at https://github.com/tiktok/huvr.