Bayesian Optimization of Text Representations

TL;DR

This paper introduces a Bayesian optimization approach to automatically select optimal text representations in NLP, making simple models competitive with complex methods and reducing manual tuning.

Contribution

It formulates text representation selection as a global optimization problem and applies sequential model-based optimization to improve NLP model performance.

Findings

Standard linear models become competitive with advanced methods.

The approach reduces manual hyperparameter tuning.

It demonstrates effectiveness on topic classification and sentiment analysis.

Abstract

When applying machine learning to problems in NLP, there are many choices to make about how to represent input texts. These choices can have a big effect on performance, but they are often uninteresting to researchers or practitioners who simply need a module that performs well. We propose an approach to optimizing over this space of choices, formulating the problem as global optimization. We apply a sequential model-based optimization technique and show that our method makes standard linear models competitive with more sophisticated, expensive state-of-the-art methods based on latent variable models or neural networks on various topic classification and sentiment analysis problems. Our approach is a first step towards black-box NLP systems that work with raw text and do not require manual tuning.