Can Large Language Models be Effective Online Opinion Miners?

TL;DR

This paper introduces a new benchmark dataset and evaluation protocol to assess large language models' effectiveness in mining opinions from complex online content, addressing current challenges in opinion extraction.

Contribution

The paper presents the Online Opinion Mining Benchmark (OOMB), a novel dataset and evaluation framework for testing LLMs' ability to extract and summarize opinions from diverse online sources.

Findings

LLMs show promising capabilities in opinion extraction and summarization.

Certain aspects of opinion mining remain challenging for LLMs.

The benchmark reveals areas where LLMs need improvement for online opinion mining.

Abstract

The surge of user-generated online content presents a wealth of insights into customer preferences and market trends. However, the highly diverse, complex, and context-rich nature of such contents poses significant challenges to traditional opinion mining approaches. To address this, we introduce Online Opinion Mining Benchmark (OOMB), a novel dataset and evaluation protocol designed to assess the ability of large language models (LLMs) to mine opinions effectively from diverse and intricate online environments. OOMB provides extensive (entity, feature, opinion) tuple annotations and a comprehensive opinion-centric summary that highlights key opinion topics within each content, thereby enabling the evaluation of both the extractive and abstractive capabilities of models. Through our proposed benchmark, we conduct a comprehensive analysis of which aspects remain challenging and where LLMs exhibit adaptability, to explore whether they can effectively serve as opinion miners in realistic online scenarios. This study lays the foundation for LLM-based opinion mining and discusses directions for future research in this field.