Demystifying Spectral Feature Learning for Instrumental Variable Regression

TL;DR

This paper analyzes spectral feature learning in nonparametric instrumental variable regression, providing a theoretical framework, practical diagnostics, and empirical validation for different performance regimes based on spectral properties.

Contribution

It offers a generalization error bound, a taxonomy of performance scenarios, and a practical method to diagnose spectral conditions in IV regression.

Findings

Performance depends on spectral alignment and eigenvalue decay.

Strong spectral alignment with slow eigenvalue decay yields optimal results.

Weak spectral alignment leads to failure regardless of eigenvalues.

Abstract

We address the problem of causal effect estimation in the presence of hidden confounders, using nonparametric instrumental variable (IV) regression. A leading strategy employs spectral features - that is, learned features spanning the top eigensubspaces of the operator linking treatments to instruments. We derive a generalization error bound for a two-stage least squares estimator based on spectral features, and gain insights into the method's performance and failure modes. We show that performance depends on two key factors, leading to a clear taxonomy of outcomes. In a good scenario, the approach is optimal. This occurs with strong spectral alignment, meaning the structural function is well-represented by the top eigenfunctions of the conditional operator, coupled with this operator's slow eigenvalue decay, indicating a strong instrument. Performance degrades in a bad scenario: spectral alignment remains strong, but rapid eigenvalue decay (indicating a weaker instrument) demands significantly more samples for effective feature learning. Finally, in the ugly scenario, weak spectral alignment causes the method to fail, regardless of the eigenvalues' characteristics. Our synthetic experiments empirically validate this taxonomy. We further introduce a practical procedure to estimate these spectral properties from data, allowing practitioners to diagnose which regime a given problem falls into. We apply this method to the dSprites dataset, demonstrating its utility.