From Facts to Insights: A Study on the Generation and Evaluation of Analytical Reports for Deciphering Earnings Calls

TL;DR

This study investigates how large language models can generate and evaluate analytical reports from earnings calls, using a multi-agent approach to enhance report quality and compare with human assessments.

Contribution

It introduces a multi-agent framework for report generation and evaluation, highlighting the effects of diverse viewpoints and analyzing alignment with human judgments.

Findings

Additional agents improve report insightfulness

Human-written reports are generally preferred

LLMs show significant correlation with human evaluations

Abstract

This paper explores the use of Large Language Models (LLMs) in the generation and evaluation of analytical reports derived from Earnings Calls (ECs). Addressing a current gap in research, we explore the generation of analytical reports with LLMs in a multi-agent framework, designing specialized agents that introduce diverse viewpoints and desirable topics of analysis into the report generation process. Through multiple analyses, we examine the alignment between generated and human-written reports and the impact of both individual and collective agents. Our findings suggest that the introduction of additional agents results in more insightful reports, although reports generated by human experts remain preferred in the majority of cases. Finally, we address the challenging issue of report evaluation, we examine the limitations and strengths of LLMs in assessing the quality of generated reports in different settings, revealing a significant correlation with human experts across multiple dimensions.