On Task-personalized Multimodal Few-shot Learning for Visually-rich Document Entity Retrieval

TL;DR

This paper introduces a novel task-level few-shot learning framework for visually-rich document entity retrieval, emphasizing entity personalization and out-of-distribution handling, supported by a new dataset and improved meta-learning methods.

Contribution

It proposes a task-aware meta-learning framework with hierarchical decoding and contrastive learning for entity-level few-shot VDER, addressing personalization and OOD challenges.

Findings

Significant improvements over baseline meta-learning models.

Enhanced robustness in entity retrieval with few-shot learning.

Introduction of the FewVEX dataset for future research.

Abstract

Visually-rich document entity retrieval (VDER), which extracts key information (e.g. date, address) from document images like invoices and receipts, has become an important topic in industrial NLP applications. The emergence of new document types at a constant pace, each with its unique entity types, presents a unique challenge: many documents contain unseen entity types that occur only a couple of times. Addressing this challenge requires models to have the ability of learning entities in a few-shot manner. However, prior works for Few-shot VDER mainly address the problem at the document level with a predefined global entity space, which doesn't account for the entity-level few-shot scenario: target entity types are locally personalized by each task and entity occurrences vary significantly among documents. To address this unexplored scenario, this paper studies a novel entity-level few-shot VDER task. The challenges lie in the uniqueness of the label space for each task and the increased complexity of out-of-distribution (OOD) contents. To tackle this novel task, we present a task-aware meta-learning based framework, with a central focus on achieving effective task personalization that distinguishes between in-task and out-of-task distribution. Specifically, we adopt a hierarchical decoder (HC) and employ contrastive learning (ContrastProtoNet) to achieve this goal. Furthermore, we introduce a new dataset, FewVEX, to boost future research in the field of entity-level few-shot VDER. Experimental results demonstrate our approaches significantly improve the robustness of popular meta-learning baselines.