Shaping Visual Representations with Language for Few-shot Classification

TL;DR

This paper introduces language-shaped learning (LSL), a novel approach that uses language to regularize visual representations for few-shot classification, improving data efficiency and performance when task descriptions are available during training.

Contribution

The paper presents LSL, a new end-to-end model that leverages language to enhance visual representations for few-shot learning, outperforming existing methods.

Findings

LSL outperforms baselines in two few-shot domains.

Language regularization improves visual feature learning.

The approach is more data-efficient and simpler than previous methods.

Abstract

By describing the features and abstractions of our world, language is a crucial tool for human learning and a promising source of supervision for machine learning models. We use language to improve few-shot visual classification in the underexplored scenario where natural language task descriptions are available during training, but unavailable for novel tasks at test time. Existing models for this setting sample new descriptions at test time and use those to classify images. Instead, we propose language-shaped learning (LSL), an end-to-end model that regularizes visual representations to predict language. LSL is conceptually simpler, more data efficient, and outperforms baselines in two challenging few-shot domains.