A perceptual bias of AI Logical Argumentation Ability in Writing

TL;DR

This paper investigates how human perceptual biases affect evaluations of AI's logical reasoning abilities, revealing that preconceived views significantly influence judgments and highlighting the importance of addressing these biases for better AI understanding.

Contribution

It uncovers perceptual biases in evaluating AI reasoning and demonstrates how prior beliefs shape these assessments, providing insights for improving human-AI interactions.

Findings

Evaluations of AI reasoning are significantly biased by preconceived views.

Frequent AI users are less likely to believe AI undermines independent thinking.

Perceptual biases impact public understanding of AI capabilities.

Abstract

Can machines think? This is a central question in artificial intelligence research. However, there is a substantial divergence of views on the answer to this question. Why do people have such significant differences of opinion, even when they are observing the same real world performance of artificial intelligence? The ability of logical reasoning like humans is often used as a criterion to assess whether a machine can think. This study explores whether human biases influence evaluations of the reasoning abilities of AI. An experiment was conducted where participants assessed two texts on the same topic, one AI generated and one human written,to test for perceptual biases in evaluating logical reasoning. Based on the experimental findings, a questionnaire was designed to quantify the attitudes toward AI.The results reveal a bias in perception. The evaluations of the logical reasoning ability of AI generated texts are significantly influenced by the preconceived views on the logical reasoning abilities of AI. Furthermore, frequent AI users were less likely to believe that AI usage undermines independent thinking.This study highlights the need to address perceptual biases to improve public understanding of AI's capabilities and foster better human AI interactions.