Multi-Agent LLMs for Generating Research Limitations

TL;DR

This paper introduces a multi-agent LLM framework that systematically generates and evaluates substantive research limitations by integrating various sources and roles, significantly outperforming zero-shot baselines in coverage.

Contribution

The paper presents a novel multi-agent LLM system that combines explicit limitations, methodological analysis, and literature context to produce deeper, more comprehensive research limitations.

Findings

Achieves +15.51% coverage gain with RAG + multi-agent GPT-4o configuration.

Improves limitation coverage by +4.41% using Llama 3 8B multi-agent setup.

Introduces a pointwise LLM-based evaluation protocol for semantic coverage.

Abstract

Identifying and articulating limitations is essential for transparent and rigorous scientific research. However, zero-shot large language models (LLMs) approach often produce superficial or general limitation statements (e.g., dataset bias or generalizability). They usually repeat limitations reported by authors without looking at deeper methodological issues and contextual gaps. This problem is made worse because many authors disclose only partial or trivial limitations. We propose, a multi-agent LLM framework for generating substantive limitations. It integrates OpenReview comments and author-stated limitations to provide stronger ground truth. It also uses cited and citing papers to capture broader contextual weaknesses. In this setup, different agents have specific roles as sequential role: some extract explicit limitations, others analyze methodological gaps, some simulate the viewpoint of a peer reviewer, and a citation agent places the work within the larger body of literature. A Judge agent refines their outputs, and a Master agent consolidates them into a clear set. This structure allows for systematic identification of explicit, implicit, peer review-focused, and literature-informed limitations. Moreover, traditional NLP metrics like BLEU, ROUGE, and cosine similarity rely heavily on n-gram or embedding overlap. They often overlook semantically similar limitations. To address this, we introduce a pointwise evaluation protocol that uses an LLM-as-a-Judge to measure coverage more accurately. Experiments show that our proposed model substantially improve performance. The RAG + multi-agent GPT-4o mini configuration achieves a +15.51\% coverage gain over zero-shot baselines, while the Llama 3 8B multi-agent setup yields a +4.41\% improvement.