A Hybrid Supervised-LLM Pipeline for Actionable Suggestion Mining in Unstructured Customer Reviews

TL;DR

This paper presents a hybrid pipeline combining a high-recall classifier and an instruction-tuned LLM to improve extraction and summarization of actionable suggestions from unstructured customer reviews, outperforming existing methods.

Contribution

It introduces a novel hybrid approach that enhances suggestion extraction accuracy and interpretability in unstructured reviews, addressing limitations of prior single-method techniques.

Findings

Outperforms baseline methods in extraction accuracy and cluster coherence.

Human evaluations confirm clarity, faithfulness, and interpretability of suggestions.

Highlights challenges in domain adaptation and deployment efficiency.

Abstract

Extracting actionable suggestions from customer reviews is essential for operational decision-making, yet these directives are often embedded within mixed-intent, unstructured text. Existing approaches either classify suggestion-bearing sentences or generate high-level summaries, but rarely isolate the precise improvement instructions businesses need. We evaluate a hybrid pipeline combining a high-recall RoBERTa classifier trained with a precision-recall surrogate to reduce unrecoverable false negatives with a controlled, instruction-tuned LLM for suggestion extraction, categorization, clustering, and summarization. Across real-world hospitality and food datasets, the hybrid system outperforms prompt-only, rule-based, and classifier-only baselines in extraction accuracy and cluster coherence. Human evaluations further confirm that the resulting suggestions and summaries are clear, faithful, and interpretable. Overall, our results show that hybrid reasoning architectures achieve meaningful improvements fine-grained actionable suggestion mining while highlighting challenges in domain adaptation and efficient local deployment.