InvGC: Robust Cross-Modal Retrieval by Inverse Graph Convolution

TL;DR

This paper identifies the representation degeneration problem in cross-modal retrieval, introduces InvGC, a graph convolution-based post-processing method to enhance representation separation, and demonstrates its effectiveness through extensive experiments.

Contribution

The paper empirically validates the representation degeneration problem and proposes InvGC with LocalAdj to improve cross-modal retrieval performance.

Findings

InvGC significantly mitigates representation degeneration.

InvGC improves retrieval accuracy across multiple benchmarks.

Theoretical analysis confirms recall bounds are enhanced by InvGC.

Abstract

Over recent decades, significant advancements in cross-modal retrieval are mainly driven by breakthroughs in visual and linguistic modeling. However, a recent study shows that multi-modal data representations tend to cluster within a limited convex cone (as representation degeneration problem), which hinders retrieval performance due to the inseparability of these representations. In our study, we first empirically validate the presence of the representation degeneration problem across multiple cross-modal benchmarks and methods. Next, to address it, we introduce a novel method, called InvGC, a post-processing technique inspired by graph convolution and average pooling. Specifically, InvGC defines the graph topology within the datasets and then applies graph convolution in a subtractive manner. This method effectively separates representations by increasing the distances between data points. To improve the efficiency and effectiveness of InvGC, we propose an advanced graph topology, LocalAdj, which only aims to increase the distances between each data point and its nearest neighbors. To understand why InvGC works, we present a detailed theoretical analysis, proving that the lower bound of recall will be improved after deploying InvGC. Extensive empirical results show that InvGC and InvGC w/LocalAdj significantly mitigate the representation degeneration problem, thereby enhancing retrieval performance. Our code is available at https://github.com/yimuwangcs/Better_Cross_Modal_Retrieval