SparseMAP: Differentiable Sparse Structured Inference

TL;DR

SparseMAP introduces a differentiable, sparse structured inference method that efficiently selects a few plausible structures, enabling better interpretability and integration with neural networks in NLP tasks.

Contribution

It presents SparseMAP, a novel inference technique that combines the benefits of MAP and marginal inference, computable via a MAP oracle, suitable for complex structured prediction.

Findings

Achieves competitive accuracy in dependency parsing and natural language inference.

Enhances interpretability and captures language ambiguities effectively.

Enables efficient gradient backpropagation with sparse structures.

Abstract

Structured prediction requires searching over a combinatorial number of structures. To tackle it, we introduce SparseMAP: a new method for sparse structured inference, and its natural loss function. SparseMAP automatically selects only a few global structures: it is situated between MAP inference, which picks a single structure, and marginal inference, which assigns probability mass to all structures, including implausible ones. Importantly, SparseMAP can be computed using only calls to a MAP oracle, making it applicable to problems with intractable marginal inference, e.g., linear assignment. Sparsity makes gradient backpropagation efficient regardless of the structure, enabling us to augment deep neural networks with generic and sparse structured hidden layers. Experiments in dependency parsing and natural language inference reveal competitive accuracy, improved interpretability, and the ability to capture natural language ambiguities, which is attractive for pipeline systems.