Efficacy of AI RAG Tools for Complex Information Extraction and Data Annotation Tasks: A Case Study Using Banks Public Disclosures

TL;DR

This study evaluates AI retrieval augmented generation tools in complex data extraction tasks, demonstrating significant improvements in speed and accuracy, especially with interactive use, in a real-world banking disclosures case.

Contribution

It provides empirical evidence on the effectiveness of AI RAG tools in complex annotation tasks and highlights the importance of user skill in AI-assisted workflows.

Findings

AI tools accelerated task completion by up to 10 times.

Interactive AI use improved annotation accuracy.

Potential to save up to 268 hours in large-scale tasks.

Abstract

We utilize a within-subjects design with randomized task assignments to understand the effectiveness of using an AI retrieval augmented generation (RAG) tool to assist analysts with an information extraction and data annotation task. We replicate an existing, challenging real-world annotation task with complex multi-part criteria on a set of thousands of pages of public disclosure documents from global systemically important banks (GSIBs) with heterogeneous and incomplete information content. We test two treatment conditions. First, a "naive" AI use condition in which annotators use only the tool and must accept the first answer they are given. And second, an "interactive" AI treatment condition where annotators use the tool interactively, and use their judgement to follow-up with additional information if necessary. Compared to the human-only baseline, the use of the AI tool accelerated task execution by up to a factor of 10 and enhanced task accuracy, particularly in the interactive condition. We find that when extrapolated to the full task, these methods could save up to 268 hours compared to the human-only approach. Additionally, our findings suggest that annotator skill, not just with the subject matter domain, but also with AI tools, is a factor in both the accuracy and speed of task performance.