In-Context Learning for Long-Context Sentiment Analysis on Infrastructure Project Opinions

TL;DR

This paper evaluates the performance of leading large language models on long, complex infrastructure project opinions, highlighting their strengths and limitations in zero-shot and few-shot settings.

Contribution

It provides a comparative analysis of GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro on long-context sentiment analysis tasks, revealing their relative capabilities.

Findings

GPT-4o excels in zero-shot for simple, short documents.

Claude 3.5 Sonnet outperforms in complex, sentiment-fluctuating opinions.

Claude 3.5 Sonnet performs best in few-shot scenarios.

Abstract

Large language models (LLMs) have achieved impressive results across various tasks. However, they still struggle with long-context documents. This study evaluates the performance of three leading LLMs: GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro on lengthy, complex, and opinion-varying documents concerning infrastructure projects, under both zero-shot and few-shot scenarios. Our results indicate that GPT-4o excels in zero-shot scenarios for simpler, shorter documents, while Claude 3.5 Sonnet surpasses GPT-4o in handling more complex, sentiment-fluctuating opinions. In few-shot scenarios, Claude 3.5 Sonnet outperforms overall, while GPT-4o shows greater stability as the number of demonstrations increases.