On the Semantics of Large Language Models

TL;DR

This paper investigates whether large language models genuinely understand language semantics by analyzing their internal representations and comparing them with classical semantic theories.

Contribution

It provides a nuanced analysis of LLMs' semantic capabilities through examining their inner workings and relating them to classical semantic theories.

Findings

LLMs' internal representations show partial semantic understanding.

Comparison with classical theories reveals gaps in LLM semantics.

The study offers a framework for assessing LLM semantic comprehension.

Abstract

Large Language Models (LLMs) such as ChatGPT demonstrated the potential to replicate human language abilities through technology, ranging from text generation to engaging in conversations. However, it remains controversial to what extent these systems truly understand language. We examine this issue by narrowing the question down to the semantics of LLMs at the word and sentence level. By examining the inner workings of LLMs and their generated representation of language and by drawing on classical semantic theories by Frege and Russell, we get a more nuanced picture of the potential semantic capabilities of LLMs.