AnalyticsGPT: An LLM Workflow for Scientometric Question Answering

TL;DR

AnalyticsGPT presents an LLM-based workflow tailored for answering complex scientometric questions, integrating retrieval and reasoning to facilitate meta-scientific analysis with expert evaluation.

Contribution

The paper introduces a novel end-to-end LLM workflow for scientometric question answering, combining retrieval-augmented generation, task planning, and expert assessment.

Findings

Effective LLM-based system for scientometric questions

Improved data synthesis into high-level analyses

Insights into LLMs' efficacy for niche scientific tasks

Abstract

This paper introduces AnalyticsGPT, an intuitive and efficient large language model (LLM)-powered workflow for scientometric question answering. This underrepresented downstream task addresses the subcategory of meta-scientific questions concerning the "science of science." When compared to traditional scientific question answering based on papers, the task poses unique challenges in the planning phase. Namely, the need for named-entity recognition of academic entities within questions and multi-faceted data retrieval involving scientometric indices, e.g. impact factors. Beyond their exceptional capacity for treating traditional natural language processing tasks, LLMs have shown great potential in more complex applications, such as task decomposition and planning and reasoning. In this paper, we explore the application of LLMs to scientometric question answering, and describe an end-to-end system implementing a sequential workflow with retrieval-augmented generation and agentic concepts. We also address the secondary task of effectively synthesizing the data into presentable and well-structured high-level analyses. As a database for retrieval-augmented generation, we leverage a proprietary research performance assessment platform. For evaluation, we consult experienced subject matter experts and leverage LLMs-as-judges. In doing so, we provide valuable insights on the efficacy of LLMs towards a niche downstream task. Our (skeleton) code and prompts are available at: https://github.com/lyvykhang/llm-agents-scientometric-qa/tree/acl.