AIANO: Enhancing Information Retrieval with AI-Augmented Annotation
Sameh Khattab, Marie Bauer, Lukas Heine, Till Rostalski, Jens Kleesiek, and Julian Friedrich

TL;DR
AIANO is a specialized annotation tool that integrates AI assistance with human expertise, significantly speeding up dataset creation and improving accuracy for information retrieval tasks.
Contribution
The paper introduces AIANO, a novel AI-augmented annotation workflow and tool that enhances efficiency and quality in creating retrieval datasets.
Findings
AIANO nearly doubled annotation speed compared to baseline tools.
AIANO was easier to use and improved retrieval accuracy.
User study confirmed efficiency and quality improvements.
Abstract
The rise of Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) has rapidly increased the need for high-quality, curated information retrieval datasets. These datasets, however, are currently created with off-the-shelf annotation tools that make the annotation process complex and inefficient. To streamline this process, we developed a specialized annotation tool - AIANO. By adopting an AI-augmented annotation workflow that tightly integrates human expertise with LLM assistance, AIANO enables annotators to leverage AI suggestions while retaining full control over annotation decisions. In a within-subject user study (), participants created question-answering datasets using both a baseline tool and AIANO. AIANO nearly doubled annotation speed compared to the baseline while being easier to use and improving retrieval accuracy. These results demonstrate that…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsTopic Modeling · Information Retrieval and Search Behavior · Artificial Intelligence in Healthcare and Education
