Neural Methods for Point-wise Dependency Estimation

TL;DR

This paper introduces methods for estimating point-wise dependency between outcomes, addressing the limitations of mutual information in capturing local dependencies, and demonstrates their effectiveness in various tasks.

Contribution

The authors propose two novel methods for point-wise dependency estimation that do not rely on optimizing mutual information bounds, improving practical applicability.

Findings

Effective in mutual information estimation

Enhance self-supervised representation learning

Improve cross-modal retrieval performance

Abstract

Since its inception, the neural estimation of mutual information (MI) has demonstrated the empirical success of modeling expected dependency between high-dimensional random variables. However, MI is an aggregate statistic and cannot be used to measure point-wise dependency between different events. In this work, instead of estimating the expected dependency, we focus on estimating point-wise dependency (PD), which quantitatively measures how likely two outcomes co-occur. We show that we can naturally obtain PD when we are optimizing MI neural variational bounds. However, optimizing these bounds is challenging due to its large variance in practice. To address this issue, we develop two methods (free of optimizing MI variational bounds): Probabilistic Classifier and Density-Ratio Fitting. We demonstrate the effectiveness of our approaches in 1) MI estimation, 2) self-supervised representation learning, and 3) cross-modal retrieval task.