Analyzing Large language models chatbots: An experimental approach using a probability test

TL;DR

This research evaluates how two large language model chatbots handle probability-based questions, revealing their strengths in familiar problems and weaknesses in novel probabilistic reasoning tasks.

Contribution

It introduces a novel experimental methodology using probability questions to analyze LLM chatbots' reasoning capabilities.

Findings

ChatGPT and Gemini perform well on classic probabilistic problems.

Both chatbots struggle with applying probabilistic logic to new, unfamiliar problems.

Results suggest reliance on textual cues over logical reasoning in probabilistic tasks.

Abstract

This study consists of qualitative empirical research, conducted through exploratory tests with two different Large Language Models (LLMs) chatbots: ChatGPT and Gemini. The methodological procedure involved exploratory tests based on prompts designed with a probability question. The "Linda Problem", widely recognized in cognitive psychology, was used as a basis to create the tests, along with the development of a new problem specifically for this experiment, the "Mary Problem". The object of analysis is the dataset with the outputs provided by each chatbot interaction. The purpose of the analysis is to verify whether the chatbots mainly employ logical reasoning that aligns with probability theory or if they are more frequently affected by the stereotypical textual descriptions in the prompts. The findings provide insights about the approach each chatbot employs in handling logic and textual constructions, suggesting that, while the analyzed chatbots perform satisfactorily on a well-known probabilistic problem, they exhibit significantly lower performance on new tests that require direct application of probabilistic logic.