factgenie: A Framework for Span-based Evaluation of Generated Texts

TL;DR

factgenie is a flexible framework that enables annotation and visualization of span-based phenomena in generated texts, supporting both human and AI annotations through an extensible web interface.

Contribution

it introduces a versatile, extensible framework for span-based annotation and visualization of generated texts, accommodating both human and AI annotations.

Findings

supports annotations of semantic inaccuracies and irrelevant text

integrates human and large language model annotations

provides an easy-to-use web interface for data collection

Abstract

We present factgenie: a framework for annotating and visualizing word spans in textual model outputs. Annotations can capture various span-based phenomena such as semantic inaccuracies or irrelevant text. With factgenie, the annotations can be collected both from human crowdworkers and large language models. Our framework consists of a web interface for data visualization and gathering text annotations, powered by an easily extensible codebase.