Learning Energy-Based Approximate Inference Networks for Structured Applications in NLP

TL;DR

This paper introduces a framework for training neural networks as inference models for complex structured prediction tasks in NLP, improving speed and accuracy through energy-based models and adversarial learning.

Contribution

It proposes a novel method to jointly learn energy functions and inference networks for structured NLP models, enhancing inference efficiency and effectiveness.

Findings

Effective neural inference networks for structured NLP tasks.

Improved speed/accuracy trade-offs in structured prediction.

Feasible joint learning of energy functions and inference models.

Abstract

Structured prediction in natural language processing (NLP) has a long history. The complex models of structured application come at the difficulty of learning and inference. These difficulties lead researchers to focus more on models with simple structure components (e.g., local classifier). Deep representation learning has become increasingly popular in recent years. The structure components of their method, on the other hand, are usually relatively simple. We concentrate on complex structured models in this dissertation. We provide a learning framework for complicated structured models as well as an inference method with a better speed/accuracy/search error trade-off. The dissertation begins with a general introduction to energy-based models. In NLP and other applications, an energy function is comparable to the concept of a scoring function. In this dissertation, we discuss the concept of the energy function and structured models with different energy functions. Then, we propose a method in which we train a neural network to do argmax inference under a structured energy function, referring to the trained networks as "inference networks" or "energy-based inference networks". We then develop ways of jointly learning energy functions and inference networks using an adversarial learning framework. Despite the inference and learning difficulties of energy-based models, we present approaches in this thesis that enable energy-based models more easily to be applied in structured NLP applications.