Rethinking the Optimization of Average Precision: Only Penalizing Negative Instances before Positive Ones is Enough

TL;DR

This paper introduces a novel loss function, PNP-D, that focuses on penalizing negative instances before positive ones to optimize Average Precision more effectively, achieving state-of-the-art results in image retrieval.

Contribution

The paper proposes the PNP loss, emphasizing negative instances before positives, and systematically investigates gradient assignment strategies, with PNP-D outperforming existing methods.

Findings

PNP-D achieves state-of-the-art performance on retrieval datasets.

Focusing on negative instances before positives improves AP optimization.

Different gradient strategies influence the clustering of relevant instances.

Abstract

Optimizing the approximation of Average Precision (AP) has been widely studied for image retrieval. Limited by the definition of AP, such methods consider both negative and positive instances ranking before each positive instance. However, we claim that only penalizing negative instances before positive ones is enough, because the loss only comes from these negative instances. To this end, we propose a novel loss, namely Penalizing Negative instances before Positive ones (PNP), which can directly minimize the number of negative instances before each positive one. In addition, AP-based methods adopt a fixed and sub-optimal gradient assignment strategy. Therefore, we systematically investigate different gradient assignment solutions via constructing derivative functions of the loss, resulting in PNP-I with increasing derivative functions and PNP-D with decreasing ones. PNP-I focuses more on the hard positive instances by assigning larger gradients to them and tries to make all relevant instances closer. In contrast, PNP-D pays less attention to such instances and slowly corrects them. For most real-world data, one class usually contains several local clusters. PNP-I blindly gathers these clusters while PNP-D keeps them as they were. Therefore, PNP-D is more superior. Experiments on three standard retrieval datasets show consistent results with the above analysis. Extensive evaluations demonstrate that PNP-D achieves the state-of-the-art performance. Code is available at https://github.com/interestingzhuo/PNPloss