Large language models and (non-)linguistic recursion

TL;DR

This paper investigates whether large language models like GPT-4 can produce and analyze recursive structures, exploring the emergence of human-like meta-linguistic awareness of recursion through specific prompts.

Contribution

It introduces prompt designs to elicit and analyze recursive behavior in LLMs, demonstrating GPT-4's ability to handle recursive structures and explore meta-linguistic awareness.

Findings

GPT-4 can produce recursive structures when prompted.

GPT-4 can analyze recursive structures.

Meta-linguistic awareness of recursion can emerge in large transformers.

Abstract

Recursion is one of the hallmarks of human language. While many design features of language have been shown to exist in animal communication systems, recursion has not. Previous research shows that GPT-4 is the first large language model (LLM) to exhibit metalinguistic abilities (Begu\v{s}, D\k{a}bkowski, and Rhodes 2023). Here, we propose several prompt designs aimed at eliciting and analyzing recursive behavior in LLMs, both linguistic and non-linguistic. We demonstrate that when explicitly prompted, GPT-4 can both produce and analyze recursive structures. Thus, we present one of the first studies investigating whether meta-linguistic awareness of recursion -- a uniquely human cognitive property -- can emerge in transformers with a high number of parameters such as GPT-4.