Generalized Sum Pooling for Metric Learning

TL;DR

This paper introduces a learnable generalized sum pooling (GSP) method for deep metric learning, enhancing global average pooling by enabling selective focus on semantic features and importance weighting, leading to improved performance.

Contribution

It proposes a novel GSP method based on entropy-smoothed optimal transport that generalizes GAP and is fully differentiable for end-to-end learning.

Findings

GSP outperforms GAP on 4 metric learning benchmarks.

The method effectively learns to ignore nuisance information.

GSP provides a flexible, learnable pooling alternative to traditional GAP.

Abstract

A common architectural choice for deep metric learning is a convolutional neural network followed by global average pooling (GAP). Albeit simple, GAP is a highly effective way to aggregate information. One possible explanation for the effectiveness of GAP is considering each feature vector as representing a different semantic entity and GAP as a convex combination of them. Following this perspective, we generalize GAP and propose a learnable generalized sum pooling method (GSP). GSP improves GAP with two distinct abilities: i) the ability to choose a subset of semantic entities, effectively learning to ignore nuisance information, and ii) learning the weights corresponding to the importance of each entity. Formally, we propose an entropy-smoothed optimal transport problem and show that it is a strict generalization of GAP, i.e., a specific realization of the problem gives back GAP. We show that this optimization problem enjoys analytical gradients enabling us to use it as a direct learnable replacement for GAP. We further propose a zero-shot loss to ease the learning of GSP. We show the effectiveness of our method with extensive evaluations on 4 popular metric learning benchmarks. Code is available at: GSP-DML Framework