Embeddings and Attention in Predictive Modeling

TL;DR

This paper investigates how embeddings and attention mechanisms can improve predictive modeling of claim severity, demonstrating their utility in feature representation, interpretability, and performance enhancement.

Contribution

It introduces methods for integrating embeddings and attention in predictive models, and shows how they can be used for feature extraction and improved accuracy.

Findings

Embeddings serve as effective pretrained features for linear models.

Attention mechanisms enhance the contextual relevance of embeddings.

Models with attention outperform simpler neural networks in claim severity prediction.

Abstract

We explore in depth how categorical data can be processed with embeddings in the context of claim severity modeling. We develop several models that range in complexity from simple neural networks to state-of-the-art attention based architectures that utilize embeddings. We illustrate the utility of learned embeddings from neural networks as pretrained features in generalized linear models, and discuss methods for visualizing and interpreting embeddings. Finally, we explore how attention based models can contextually augment embeddings, leading to enhanced predictive performance.