Training LayoutLM from Scratch for Efficient Named-Entity Recognition in the Insurance Domain

TL;DR

This paper explores training LayoutLM from scratch with domain-specific data to improve named-entity recognition in insurance documents, demonstrating that domain relevance and smaller models can yield competitive results.

Contribution

It introduces a domain-specific pre-training approach for LayoutLM and shows that smaller models trained on relevant data outperform generic models in insurance NER tasks.

Findings

Domain-relevant pre-training improves NER accuracy.

Smaller, faster models can achieve competitive performance.

Effective training from scratch is feasible with limited in-domain data.

Abstract

Generic pre-trained neural networks may struggle to produce good results in specialized domains like finance and insurance. This is due to a domain mismatch between training data and downstream tasks, as in-domain data are often scarce due to privacy constraints. In this work, we compare different pre-training strategies for LayoutLM. We show that using domain-relevant documents improves results on a named-entity recognition (NER) problem using a novel dataset of anonymized insurance-related financial documents called Payslips. Moreover, we show that we can achieve competitive results using a smaller and faster model.