Development of the user-friendly decision aid Rule-based Evaluation and Support Tool (REST) for optimizing the resources of an information extraction task

TL;DR

This paper introduces REST, a decision support tool that helps annotators choose between rule-based and machine learning methods for information extraction, aiming to improve sustainability, transferability, and efficiency.

Contribution

The paper presents REST, a novel interactive tool that visualizes entity characteristics and performance metrics to optimize resource use in IE tasks.

Findings

REST demonstrated good reproducibility on a 12-entity use case.

The tool effectively guides annotators in selecting between rules and ML.

Manual annotation efforts are minimized through REST's support.

Abstract

Rules could be an information extraction (IE) default option, compared to ML and LLMs in terms of sustainability, transferability, interpretability, and development burden. We suggest a sustainable and combined use of rules and ML as an IE method. Our approach starts with an exhaustive expert manual highlighting in a single working session of a representative subset of the data corpus. We developed and validated the feasibility and the performance metrics of the REST decision tool to help the annotator choose between rules as a by default option and ML for each entity of an IE task. REST makes the annotator visualize the characteristics of each entity formalization in the free texts and the expected rule development feasibility and IE performance metrics. ML is considered as a backup IE option and manual annotation for training is therefore minimized. The external validity of REST on a 12-entity use case showed good reproducibility.