DeepEMD: Differentiable Earth Mover's Distance for Few-Shot Learning

TL;DR

DeepEMD introduces a differentiable Earth Mover's Distance layer for few-shot image classification, leveraging optimal matching of image regions to improve accuracy and robustness against background clutter and intra-class variations.

Contribution

The paper proposes a novel differentiable EMD-based method for few-shot learning, including a cross-reference mechanism and a structured fully connected layer for end-to-end training.

Findings

Outperforms state-of-the-art on five few-shot benchmarks.

Effective in image retrieval tasks.

Robust to background clutter and intra-class variations.

Abstract

In this work, we develop methods for few-shot image classification from a new perspective of optimal matching between image regions. We employ the Earth Mover's Distance (EMD) as a metric to compute a structural distance between dense image representations to determine image relevance. The EMD generates the optimal matching flows between structural elements that have the minimum matching cost, which is used to calculate the image distance for classification. To generate the important weights of elements in the EMD formulation, we design a cross-reference mechanism, which can effectively alleviate the adverse impact caused by the cluttered background and large intra-class appearance variations. To implement k-shot classification, we propose to learn a structured fully connected layer that can directly classify dense image representations with the EMD. Based on the implicit function theorem, the EMD can be inserted as a layer into the network for end-to-end training. Our extensive experiments validate the effectiveness of our algorithm which outperforms state-of-the-art methods by a significant margin on five widely used few-shot classification benchmarks, namely, miniImageNet, tieredImageNet, Fewshot-CIFAR100 (FC100), Caltech-UCSD Birds-200-2011 (CUB), and CIFAR-FewShot (CIFAR-FS). We also demonstrate the effectiveness of our method on the image retrieval task in our experiments.