Large language models for crowd decision making based on prompt design strategies using ChatGPT: models, analysis and challenges

TL;DR

This paper explores how ChatGPT, guided by prompt design strategies, can be integrated into crowd decision making to extract opinions from social media texts, demonstrating promising results and discussing key challenges.

Contribution

It introduces a novel approach using ChatGPT for crowd decision making, leveraging prompt design strategies for opinion extraction and decision scoring.

Findings

ChatGPT effectively infers opinions from social media texts.

The approach shows promising results in real data scenarios.

Challenges include issues of consistency, sensitivity, and explainability.

Abstract

Social Media and Internet have the potential to be exploited as a source of opinion to enrich Decision Making solutions. Crowd Decision Making (CDM) is a methodology able to infer opinions and decisions from plain texts, such as reviews published in social media platforms, by means of Sentiment Analysis. Currently, the emergence and potential of Large Language Models (LLMs) lead us to explore new scenarios of automatically understand written texts, also known as natural language processing. This paper analyzes the use of ChatGPT based on prompt design strategies to assist in CDM processes to extract opinions and make decisions. We integrate ChatGPT in CDM processes as a flexible tool that infer the opinions expressed in texts, providing numerical or linguistic evaluations where the decision making models are based on the prompt design strategies. We include a multi-criteria decision making scenario with a category ontology for criteria. We also consider ChatGPT as an end-to-end CDM model able to provide a general opinion and score on the alternatives. We conduct empirical experiments on real data extracted from TripAdvisor, the TripR-2020Large dataset. The analysis of results show a promising branch for developing quality decision making models using ChatGPT. Finally, we discuss the challenges of consistency, sensitivity and explainability associated to the use of LLMs in CDM processes, raising open questions for future studies.