Entropy-Guided Sampling of Flat Modes in Discrete Spaces

TL;DR

This paper introduces EDLP, a novel entropy-guided sampling method for effectively capturing flat modes in discrete spaces, with theoretical guarantees and superior empirical performance across various applications.

Contribution

We propose EDLP, a new sampling algorithm that incorporates local entropy via an auxiliary variable, improving flat mode sampling in discrete distributions.

Findings

EDLP outperforms traditional methods in flat mode sampling.

Theoretical convergence guarantees are established for EDLP.

Empirical results demonstrate EDLP's effectiveness in diverse tasks.

Abstract

Sampling from flat modes in discrete spaces is a crucial yet underexplored problem. Flat modes represent robust solutions and have broad applications in combinatorial optimization and discrete generative modeling. However, existing sampling algorithms often overlook the mode volume and struggle to capture flat modes effectively. To address this limitation, we propose \emph{Entropic Discrete Langevin Proposal} (EDLP), which incorporates local entropy into the sampling process through a continuous auxiliary variable under a joint distribution. The local entropy term guides the discrete sampler toward flat modes with a small overhead. We provide non-asymptotic convergence guarantees for EDLP in locally log-concave discrete distributions. Empirically, our method consistently outperforms traditional approaches across tasks that require sampling from flat basins, including Bernoulli distribution, restricted Boltzmann machines, combinatorial optimization, and binary neural networks.