AIANO: Enhancing Information Retrieval with AI-Augmented Annotation

TL;DR

AIANO is a specialized annotation tool that integrates AI assistance with human expertise, significantly speeding up dataset creation and improving accuracy for information retrieval tasks.

Contribution

The paper introduces AIANO, a novel AI-augmented annotation workflow and tool that enhances efficiency and quality in creating retrieval datasets.

Findings

AIANO nearly doubled annotation speed compared to baseline tools.

AIANO was easier to use and improved retrieval accuracy.

User study confirmed efficiency and quality improvements.

Abstract

The rise of Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) has rapidly increased the need for high-quality, curated information retrieval datasets. These datasets, however, are currently created with off-the-shelf annotation tools that make the annotation process complex and inefficient. To streamline this process, we developed a specialized annotation tool - AIANO. By adopting an AI-augmented annotation workflow that tightly integrates human expertise with LLM assistance, AIANO enables annotators to leverage AI suggestions while retaining full control over annotation decisions. In a within-subject user study ($n = 15$), participants created question-answering datasets using both a baseline tool and AIANO. AIANO nearly doubled annotation speed compared to the baseline while being easier to use and improving retrieval accuracy. These results demonstrate that AIANO's AI-augmented approach accelerates and enhances dataset creation for information retrieval tasks, advancing annotation capabilities in retrieval-intensive domains.