CM1 -- A Dataset for Evaluating Few-Shot Information Extraction with Large Vision Language Models

TL;DR

This paper introduces the CM1 dataset to evaluate few-shot capabilities of Large Vision Language Models in extracting handwritten key-value data from historic documents, demonstrating their advantage over traditional models with limited training data.

Contribution

The paper presents a new dataset and benchmarks for assessing LVLMs' few-shot extraction performance on handwritten forms, highlighting their effectiveness in low-data scenarios.

Findings

LVLMs outperform classical models with few training samples

Traditional full-page models perform well with ample data

CM1 dataset enables evaluation of few-shot information extraction

Abstract

The automatic extraction of key-value information from handwritten documents is a key challenge in document analysis. A reliable extraction is a prerequisite for the mass digitization efforts of many archives. Large Vision Language Models (LVLM) are a promising technology to tackle this problem especially in scenarios where little annotated training data is available. In this work, we present a novel dataset specifically designed to evaluate the few-shot capabilities of LVLMs. The CM1 documents are a historic collection of forms with handwritten entries created in Europe to administer the Care and Maintenance program after World War Two. The dataset establishes three benchmarks on extracting name and birthdate information and, furthermore, considers different training set sizes. We provide baseline results for two different LVLMs and compare performances to an established full-page extraction model. While the traditional full-page model achieves highly competitive performances, our experiments show that when only a few training samples are available the considered LVLMs benefit from their size and heavy pretraining and outperform the classical approach.