Effective Performance Measurement: Challenges and Opportunities in KPI Extraction from Earnings Calls

TL;DR

This paper investigates the challenges of extracting KPIs from unstructured earnings call transcripts, introduces new benchmarks, and proposes an LLM-based system with human-verified extraction accuracy.

Contribution

It introduces three novel benchmarks for KPI extraction from earnings calls and evaluates the generalization of SEC-trained models across domains.

Findings

SEC-trained models struggle with domain shift.

Proposed LLM-based system achieves 79.7% precision in KPI extraction.

New benchmarks support future research in unstructured financial text analysis.

Abstract

Earnings calls are a key source of financial information about public companies. However, extracting information from these calls is difficult. Unlike the templatic filings required by the U.S. Securities and Exchange Commission (SEC) to report a company's financial situation, earnings conference calls have no built-in labels, are unstructured, and feature conversational language. We explore this challenging domain by assessing the information captured by models trained on SEC filings and in-context learning methods. To establish a baseline, we first evaluate the generalization capabilities of SEC-trained models across established SEC datasets. To support our investigation, we introduce three novel benchmarks: (1) SEC Filings Benchmark (SECB), (2) Earnings Calls Benchmark (ECB), and ECB-A, a subset with 2,460 expert annotation groups to support our qualitative analysis. We find that encoder-based models struggle with the domain shift. Finally, we propose a system utilizing LLMs to perform open-ended extraction from unstructured call transcripts, verified by human evaluation (79.7% precision), providing a baseline for this valuable domain through the consistent tracking of emergent KPIs.