LinguistAgent: A Reflective Multi-Model Platform for Automated Linguistic Annotation

TL;DR

LinguistAgent is a user-friendly platform that automates complex linguistic annotation tasks like metaphor identification using a reflective multi-model architecture, simulating peer review and supporting various LLM paradigms.

Contribution

It introduces a novel multi-model platform with a dual-agent workflow for automated linguistic annotation, integrating multiple LLM paradigms and real-time evaluation.

Findings

LinguistAgent effectively automates metaphor identification with high accuracy.

The platform supports comparative analysis of different LLM paradigms.

Real-time token-level evaluation demonstrates practical utility.

Abstract

Data annotation remains a significant bottleneck in the Humanities and Social Sciences, particularly for complex semantic tasks such as metaphor identification. While Large Language Models (LLMs) show promise, a significant gap remains between the theoretical capability of LLMs and their practical utility for researchers. This paper introduces LinguistAgent, an integrated, user-friendly platform that leverages a reflective multi-model architecture to automate linguistic annotation. The system implements a dual-agent workflow, comprising an Annotator and a Reviewer, to simulate a professional peer-review process. LinguistAgent supports comparative experiments across three paradigms: Prompt Engineering (Zero/Few-shot), Retrieval-Augmented Generation, and Fine-tuning. We demonstrate LinguistAgent's efficacy using the task of metaphor identification as an example, providing real-time token-level evaluation (Precision, Recall, and $F_1$ score) against human gold standards. The application and codes are released on https://github.com/Bingru-Li/LinguistAgent.