VIDEE: Visual and Interactive Decomposition, Execution, and Evaluation of Text Analytics with Intelligent Agents

TL;DR

VIDEE is a system that enables entry-level analysts to perform advanced text analytics using intelligent agents, combining human-in-the-loop reasoning, automated pipeline generation, and evaluation with visualizations.

Contribution

It introduces a novel human-agent collaboration workflow with Monte-Carlo Tree Search, executable pipelines, and LLM-based evaluation tailored for non-expert users.

Findings

VIDEE improves usability for users with varying NLP experience.

Quantitative experiments show VIDEE's effectiveness in text analytics tasks.

User study reveals distinct behaviors and validates practical utility for non-experts.

Abstract

Text analytics has traditionally required specialized knowledge in Natural Language Processing (NLP) or text analysis, which presents a barrier for entry-level analysts. Recent advances in large language models (LLMs) have changed the landscape of NLP by enabling more accessible and automated text analysis (e.g., topic detection, summarization, information extraction, etc.). We introduce VIDEE, a system that supports entry-level data analysts to conduct advanced text analytics with intelligent agents. VIDEE instantiates a human-agent collaroration workflow consisting of three stages: (1) Decomposition, which incorporates a human-in-the-loop Monte-Carlo Tree Search algorithm to support generative reasoning with human feedback, (2) Execution, which generates an executable text analytics pipeline, and (3) Evaluation, which integrates LLM-based evaluation and visualizations to support user validation of execution results. We conduct two quantitative experiments to evaluate VIDEE's effectiveness and analyze common agent errors. A user study involving participants with varying levels of NLP and text analytics experience -- from none to expert -- demonstrates the system's usability and reveals distinct user behavior patterns. The findings identify design implications for human-agent collaboration, validate the practical utility of VIDEE for non-expert users, and inform future improvements to intelligent text analytics systems.