Towards Dynamic Computation Graphs via Sparse Latent Structure

TL;DR

This paper introduces a novel end-to-end learning approach for dynamic computation graphs in NLP, leveraging SparseMAP inference to jointly learn latent structures and downstream tasks without sacrificing differentiability.

Contribution

It presents the first method to enable unrestricted dynamic graph construction from global latent structures while maintaining end-to-end differentiability.

Findings

Enables joint learning of latent structures and downstream predictors.

Maintains differentiability with SparseMAP inference.

Allows dynamic computation graphs based on global latent structures.

Abstract

Deep NLP models benefit from underlying structures in the data---e.g., parse trees---typically extracted using off-the-shelf parsers. Recent attempts to jointly learn the latent structure encounter a tradeoff: either make factorization assumptions that limit expressiveness, or sacrifice end-to-end differentiability. Using the recently proposed SparseMAP inference, which retrieves a sparse distribution over latent structures, we propose a novel approach for end-to-end learning of latent structure predictors jointly with a downstream predictor. To the best of our knowledge, our method is the first to enable unrestricted dynamic computation graph construction from the global latent structure, while maintaining differentiability.