iobes: A Library for Span-Level Processing

TL;DR

The paper introduces iobes, an open-source library designed to facilitate parsing, converting, and processing span-level annotations in natural language processing tasks like named entity recognition.

Contribution

It provides a standardized, easy-to-use tool for handling span annotations, improving consistency and efficiency in NLP span labeling tasks.

Findings

iobes simplifies span annotation processing

It enables consistent conversion between span label formats

The library improves reproducibility in span-based NLP tasks

Abstract

Many tasks in natural language processing, such as named entity recognition and slot-filling, involve identifying and labeling specific spans of text. In order to leverage common models, these tasks are often recast as sequence labeling tasks. Each token is given a label and these labels are prefixed with special tokens such as B- or I-. After a model assigns labels to each token, these prefixes are used to group the tokens into spans. Properly parsing these annotations is critical for producing fair and comparable metrics; however, despite its importance, there is not an easy-to-use, standardized, programmatically integratable library to help work with span labeling. To remedy this, we introduce our open-source library, iobes. iobes is used for parsing, converting, and processing spans represented as token level decisions.