A Stable Neural Statistical Dependence Estimator for Autoencoder Feature Analysis

TL;DR

This paper introduces a stable neural dependence estimator for autoencoder features using a variational Gaussian approach, improving stability and efficiency over existing methods for analyzing statistical dependence.

Contribution

The authors propose a novel neural dependence estimator based on orthonormal density-ratio decomposition that avoids input concatenation, reducing computational cost and enhancing stability.

Findings

The method enables meaningful dependence measurement in deterministic autoencoders.

It demonstrates sequential convergence of singular values for feature analysis.

The estimator outperforms MINE in stability and efficiency.

Abstract

Statistical dependence measures like mutual information is ideal for analyzing autoencoders, but it can be ill-posed for deterministic, static, noise-free networks. We adopt the variational (Gaussian) formulation that makes dependence among inputs, latents, and reconstructions measurable, and we propose a stable neural dependence estimator based on an orthonormal density-ratio decomposition. Unlike MINE, our method avoids input concatenation and product-of-marginals re-pairing, reducing computational cost and improving stability. We introduce an efficient NMF-like scalar cost and demonstrate empirically that assuming Gaussian noise to form an auxiliary variable enables meaningful dependence measurements and supports quantitative feature analysis, with a sequential convergence of singular values.