Fine-tuning and aligning question answering models for complex information extraction tasks

TL;DR

This paper demonstrates that fine-tuning extractive question answering models enhances their ability to reliably extract complex information from German business documents, addressing hallucination issues of generative models.

Contribution

It introduces a fine-tuning approach for German QA models tailored to complex document features, improving extraction accuracy with limited annotated data.

Findings

Fine-tuning improves extraction performance on complex linguistic features.

A combined scoring metric better mimics human evaluation.

Extractive QA models reduce hallucination compared to generative models.

Abstract

The emergence of Large Language Models (LLMs) has boosted performance and possibilities in various NLP tasks. While the usage of generative AI models like ChatGPT opens up new opportunities for several business use cases, their current tendency to hallucinate fake content strongly limits their applicability to document analysis, such as information retrieval from documents. In contrast, extractive language models like question answering (QA) or passage retrieval models guarantee query results to be found within the boundaries of an according context document, which makes them candidates for more reliable information extraction in productive environments of companies. In this work we propose an approach that uses and integrates extractive QA models for improved feature extraction of German business documents such as insurance reports or medical leaflets into a document analysis solution. We further show that fine-tuning existing German QA models boosts performance for tailored extraction tasks of complex linguistic features like damage cause explanations or descriptions of medication appearance, even with using only a small set of annotated data. Finally, we discuss the relevance of scoring metrics for evaluating information extraction tasks and deduce a combined metric from Levenshtein distance, F1-Score, Exact Match and ROUGE-L to mimic the assessment criteria from human experts.