Information Projection and Approximate Inference for Structured Sparse Variables

TL;DR

This paper introduces a novel approach for approximate inference in probabilistic models with structured sparsity, leveraging information projection and submodular optimization to handle complex variable structures efficiently.

Contribution

It extends information projection methods to any variable structure admitting a matroid, providing efficient algorithms with strong guarantees for a broad class of models.

Findings

Superior performance on simulated data

Effective in high-dimensional neuroimaging data

Applicable to various structured sparse models

Abstract

Approximate inference via information projection has been recently introduced as a general-purpose approach for efficient probabilistic inference given sparse variables. This manuscript goes beyond classical sparsity by proposing efficient algorithms for approximate inference via information projection that are applicable to any structure on the set of variables that admits enumeration using a \emph{matroid}. We show that the resulting information projection can be reduced to combinatorial submodular optimization subject to matroid constraints. Further, leveraging recent advances in submodular optimization, we provide an efficient greedy algorithm with strong optimization-theoretic guarantees. The class of probabilistic models that can be expressed in this way is quite broad and, as we show, includes group sparse regression, group sparse principal components analysis and sparse canonical correlation analysis, among others. Moreover, empirical results on simulated data and high dimensional neuroimaging data highlight the superior performance of the information projection approach as compared to established baselines for a range of probabilistic models.