Characterizing Multimodal Long-form Summarization: A Case Study on Financial Reports

TL;DR

This paper systematically analyzes the abilities of various large language models in long-form multimodal summarization of financial reports, revealing strengths and biases, especially highlighting Claude 2's superior performance.

Contribution

It introduces a computational framework for characterizing multimodal long-form summarization and provides a comparative analysis of multiple LLMs on financial report summarization.

Findings

GPT-3.5 and Cohere perform poorly on the task

Claude 2 and GPT-4 show different behaviors in extractiveness and bias

Claude 2 effectively recognizes important information after input shuffling

Abstract

As large language models (LLMs) expand the power of natural language processing to handle long inputs, rigorous and systematic analyses are necessary to understand their abilities and behavior. A salient application is summarization, due to its ubiquity and controversy (e.g., researchers have declared the death of summarization). In this paper, we use financial report summarization as a case study because financial reports are not only long but also use numbers and tables extensively. We propose a computational framework for characterizing multimodal long-form summarization and investigate the behavior of Claude 2.0/2.1, GPT-4/3.5, and Cohere. We find that GPT-3.5 and Cohere fail to perform this summarization task meaningfully. For Claude 2 and GPT-4, we analyze the extractiveness of the summary and identify a position bias in LLMs. This position bias disappears after shuffling the input for Claude, which suggests that Claude seems to recognize important information. We also conduct a comprehensive investigation on the use of numeric data in LLM-generated summaries and offer a taxonomy of numeric hallucination. We employ prompt engineering to improve GPT-4's use of numbers with limited success. Overall, our analyses highlight the strong capability of Claude 2 in handling long multimodal inputs compared to GPT-4. The generated summaries and evaluation code are available at https://github.com/ChicagoHAI/characterizing-multimodal-long-form-summarization.