An\'alise de ambiguidade lingu\'istica em modelos de linguagem de grande escala (LLMs)

TL;DR

This study examines how large language models like ChatGPT and Gemini handle linguistic ambiguity in Brazilian Portuguese, revealing their limitations and the need for improved supervised learning methods.

Contribution

It provides a detailed analysis of semantic, syntactic, and lexical ambiguity in LLMs, including a new corpus and evaluation of model responses in Portuguese.

Findings

Models show errors and inconsistencies in ambiguity explanations.

Maximum accuracy achieved was 49.58%.

Highlights the need for further descriptive and supervised learning studies.

Abstract

Linguistic ambiguity continues to represent a significant challenge for natural language processing (NLP) systems, notwithstanding the advancements in architectures such as Transformers and BERT. Inspired by the recent success of instructional models like ChatGPT and Gemini (In 2023, the artificial intelligence was called Bard.), this study aims to analyze and discuss linguistic ambiguity within these models, focusing on three types prevalent in Brazilian Portuguese: semantic, syntactic, and lexical ambiguity. We create a corpus comprising 120 sentences, both ambiguous and unambiguous, for classification, explanation, and disambiguation. The models capability to generate ambiguous sentences was also explored by soliciting sets of sentences for each type of ambiguity. The results underwent qualitative analysis, drawing on recognized linguistic references, and quantitative assessment based on the accuracy of the responses obtained. It was evidenced that even the most sophisticated models, such as ChatGPT and Gemini, exhibit errors and deficiencies in their responses, with explanations often providing inconsistent. Furthermore, the accuracy peaked at 49.58 percent, indicating the need for descriptive studies for supervised learning.