The Gold Medals in an Empty Room: Diagnosing Metalinguistic Reasoning in LLMs with Camlang

TL;DR

This paper introduces Camlang, a constructed language designed to test whether large language models can develop genuine metalinguistic reasoning, revealing significant gaps between model performance and human-like understanding.

Contribution

The paper presents Camlang, a novel language and evaluation framework that distinguishes between pattern matching and true reasoning in LLMs, highlighting their limitations in mastering grammatical and semantic rules.

Findings

GPT-5 achieves 47% accuracy in Camlang, far below human 87%.

Models mainly rely on shallow lexical cues rather than systematic grammar.

Camlang exposes fundamental gaps in current LLMs' metalinguistic reasoning.

Abstract

Large Language Models (LLMs) achieve gold-medal performance across many benchmarks, yet it remains unclear whether such success reflects genuine reasoning or pattern matching. From a cognitive science perspective, an informative test is whether models can master an unfamiliar language through explicit metalinguistic deductive learning, a paradigm where human learners can reliably internalise grammatical systems through metalinguistic reasoning. We address this question with Camlang, a novel constructed language that exhibits naturalistic yet unattested feature combinations. Camlang consists of two explicit resources, a grammar book and a bilingual dictionary, which mirror adult second-language learning via explicit grammar rules and lexical lookup, and enable us to disentangle errors in morpho-syntax, lexical semantics, and sentence-level reasoning. Human experiments show that these resources are sufficient for participants to acquire Camlang and successfully solve Camlang tasks. To operationalise evaluation, we adapt CommonsenseQA into Camlang, creating Camlang-CSQA-v0, the first task in a broader suite where solving questions requires applying grammar rules and lexical mappings. Experimental results show that GPT-5 achieves 98\% EM accuracy in English but only 47\% in Camlang, far below human performance at 87\%, while other state-of-the-art reasoning LLMs perform even worse. Human verification further reveals that most model successes stem from shallow lexical alignment while GPT-5 shows emerging metalinguistic awareness to a limited extent but not systematic grammatical mastery as humans. Camlang establishes a cognitively grounded evaluation paradigm that exposes fundamental gaps between current models and human metalinguistic competence.

Tables

Table 1. Table 1: The distribution of lexical sourcing and formation as in the dictionary by the experiment time.

Category	#	%
Native Word	698	46.19
Derived Word	588	38.91
Compound Word	103	6.82
Opaque Loan	80	5.29
Transparent Loan	42	2.78

Table 2. Table 2: Averaged ROUGE scores across annotators by round at word and morpheme levels.

Round	Word			Morpheme
	R1	R2	RL	R1	R2	RL
1	46.23	19.09	44.27	67.51	36.27	61.89
2	57.96	33.69	56.67	72.97	50.35	68.39
3	62.91	40.99	60.10	75.04	55.40	69.88

Table 3. Table 3: Camlang’s top neighbours with ≥ \geq 20, ≥ \geq 30, ≥ \geq 40, and ≥ \geq 50 overlapping features. High similarities at this threshold are based on very small overlaps. Results on ≥ \geq 40 and ≥ \geq 50 features are more stable and typologically meaningful. H. East Cushitic is Highland East Cushitic.

Rank	Code	Similarity	$\left|F_X\cap F_Y\right|$	Name	Genus	Family
$\ge$20 overlap
Top–1	awa	0.96	27	Awadhi	Indic	Indo-European
Top–2	gam	0.95	20	Gamo	Ta-Ne-Omotic	Afro-Asiatic
Top–3	kum	0.91	23	Kumauni	Indic	Indo-European
Top–4	knn	0.89	28	Kinnauri	Bodic	Sino-Tibetan
Top–5	bho	0.82	33	Bhojpuri	Indic	Indo-European
$\ge$30 overlap
Top–1	bho	0.82	33	Bhojpuri	Indic	Indo-European
Top–2	sid	0.81	32	Sidaama	H. East Cushitic	Afro-Asiatic
Top–3	sna	0.80	30	Shina	Indic	Indo-European
Top–4	guj	0.75	36	Gujarati	Indic	Indo-European
Top–5	pba	0.75	32	Pima Bajo	Tepiman	Uto-Aztecan
$\ge$40 overlap
Top–1	dji	0.74	42	Djingili	Djingili	Mirndi
Top–2	raw	0.70	40	Rawang	Nungish	Sino-Tibetan
Top–3	tvo	0.68	41	Tatar	Turkic	Turkic
Top–4	mns	0.68	59	Mansi	Ugric	Uralic
Top–5	bel	0.67	43	Belhare	Himalayish	Sino-Tibetan
$\ge$50 overlap
Top–1	mns	0.68	59	Mansi	Ugric	Uralic
Top–2	kty	0.67	66	Khanty	Ugric	Uralic
Top–3	ttu	0.67	51	Tsova-Tush	Nakh	Nakh-Daghestanian
Top–4	pan	0.66	65	Panjabi	Indic	Indo-European
Top–5	dgr	0.65	55	Dagur	Mongolic	Mongolic

Table 4. Table 4: Top neighbours of Camlang compared with their own closest languages under the same overlap thresholds. N. Uto-Aztecan is Northern Uto-Aztecan.

Rank	Code	Similarity	$\left|F_X\cap F_Y\right|$	Name	Genus	Family
Awadhi’s (Indic, Indo-European) top neighbours $\ge$20 overlap
Top–1	ttu	1.00	27	Tsova-Tush	Nakh	Nakh-Daghestanian
Top–2	knn	1.00	25	Kinnauri	Bodic	Sino-Tibetan
Top–3	sna	1.00	25	Shina	Indic	Indo-European
Top–4	guj	1.00	21	Gujarati	Indic	Indo-European
Top–5	kty	0.96	30	Khanty	Ugric	Uralic
…
Top–8	-	0.96	27	Camlang	-	-
Bhojpuri’s (Indic, Indo-European) top neighbours $\ge$30 overlap
Top–1	hin	0.97	33	Hindi	Indic	Indo-European
Top–2	hop	0.91	33	Hopi	N. Uto-Aztecan	Uto-Aztecan
Top–3	kor	0.90	30	Korean	Korean	Korean
Top–4	ttu	0.90	30	Tsova-Tush	Nakh	Nakh-Daghestanian
Top–5	mgg	0.84	31	Mangghuer	Mongolic	Mongolic
…
Top–9	-	0.82	33	Camlang	-	-
Djingili’s (Djingili, Mirndi) top neighbours $\ge$40 overlap
Top–1	kor	0.74	46	Korean	Korean	Korean
Top–2	-	0.74	42	Camlang	-	-
Top–3	hin	0.71	48	Hindi	Indic	Indo-European
Top–4	ket	0.71	41	Ket	Yeniseian	Yeniseian
Top–5	bsq	0.70	47	Basque	Basque	Basque
Mansi (Urgic, Uralic) top neighbours $\ge$50 overlap
Top–1	ntu	0.77	51	Nenets	Samoyedic	Uralic
Top–2	qim	0.74	60	Quechua	Quechuan	Quechuan
Top–3	geo	0.72	58	Georgian	Kartvelian	Kartvelian
Top–4	ket	0.71	52	Ket	Yeniseian	Yeniseian
Top–5	ain	0.69	55	Ainu	Ainu	Ainu
Top–6	-	0.68	59	Camlang	-	-

Table 5. Table 5: Comparison between Camlang and high-resource languages.

Code	Similarity	$\left|F_X\cap F_Y\right|$	Name	Genus	Family
eng	0.44	131	English	Germanic	Indo-European
fre	0.42	130	French	Romance	Indo-European
tur	0.63	133	Turkish	Turkic	Turkic
mnd	0.45	128	Mandarin	Chinese	Sino-Tibetan
jpn	0.49	126	Japanese	Japanese	Japanese

Table 6. Table 6: Exact match results on Camlang-CSQA-v0 (ACC: exact match accuracy, Time: average inference time per instance). The last column reports the accuracy drop from English to Camlang.

Model	Camlang		English		ACC $\downarrow$
	ACC (%)	Time (s)	ACC (%)	Time (s)
4o (tool)	21.28	75.04	89.36	19.49	68.08
4o (context)	21.28	31.98			68.08
DS (context)	40.43	263.63	85.11	58.70	44.68
o4-mini (tool)	25.53	413.84	91.49	14.20	65.96
o3 (tool)	27.66	566.09	97.87	7.86	70.21
o3 (context)	46.81	158.23			51.06
GPT-5 (tool)	42.55	475.34	95.74	16.46	53.19
GPT-5 (context)	46.81	101.70			48.93
Random	16.67	-	16.67	-	-
Human	87.23	670.91	91.49	17.50	4.26

Table 7. Table 7: Human verification results on Camlang-CSQA-v0. SHV: strict correctness and completeness across all aspects. MHV: requires correct parsing but tolerates incomplete semantic understanding. LHV: allows incomplete but not incorrect interpretations, capturing partial or shortcut reasoning. The last column presents exact-match accuracy (EM ACC) for reference.

Model	SHV ACC (%)	MHV ACC (%)	LHV ACC (%)	EM ACC (%)
4o (tool)	0.00	0.00	0.00	21.28
4o (context)	0.00	0.00	0.00	21.28
DS (context)	0.00	2.13	4.26	40.43
o4-mini (tool)	0.00	2.13	6.38	25.53
o3 (tool)	0.00	2.13	6.38	27.66
o3 (context)	0.00	6.38	10.64	46.81
GPT-5 (tool)	2.13	2.13	17.02	42.55
GPT-5 (context)	0.00	19.15	29.79	46.81
Human	55.32	59.57	68.09	87.23

Table 8. Table 8: Distribution of human verification labels for each model, over exact-match-correct instances. Each aspect is broken down into counts (#) and percentages (%). Note that percentages here are computed within each model’s EM-correct instances. (1) Crt+ & Com+, (2) Crt+ & Com-, (3) Crt- & Com+, and (4) Crt- & Com- correspond to (1) correct and complete , (2) correct and incomplete , (3) incorrect and complete , and (4) incorrect and incomplete , respectively.

Model	# EM	Label	Q. Parsing		Q. Meaning		O. Meaning
			#	%	#	%	#	%
4o (tool)	10	Crt+ & Com+	0	0.00	0	0.00	0	0.00
		Crt+ & Com-	0	0.00	0	0.00	0	0.00
		Crt-  & Com+	0	0.00	0	0.00	0	0.00
		Crt-  & Com-	10	100.00	10	100.00	10	100.00
4o (context)	10	Crt+ & Com+	0	0.00	0	0.00	0	0.00
		Crt+ & Com-	0	0.00	0	0.00	0	0.00
		Crt-  & Com+	0	0.00	0	0.00	0	0.00
		Crt-  & Com-	10	100.00	10	100.00	10	100.00
DS (context)	19	Crt+ & Com+	4	21.05	6	31.58	9	47.37
		Crt+ & Com-	1	5.26	0	0.00	2	10.53
		Crt-  & Com+	11	57.89	11	57.89	8	42.11
		Crt-  & Com-	3	15.79	2	10.53	0	0.00
o4-mini (tool)	12	Crt+ & Com+	2	16.67	1	8.33	1	8.33
		Crt+ & Com-	4	33.33	2	16.67	9	75.00
		Crt-  & Com+	4	33.33	6	50.00	0	0.00
		Crt-  & Com-	2	16.67	3	25.00	2	16.67
o3 (tool)	13	Crt+ & Com+	5	38.46	9	69.23	2	15.38
		Crt+ & Com-	4	30.77	2	15.38	5	38.46
		Crt-  & Com+	1	7.69	2	15.38	2	15.38
		Crt-  & Com-	3	23.07	0	0.00	4	30.77
o3 (context)	22	Crt+ & Com+	7	31.82	15	68.18	6	27.27
		Crt+ & Com-	2	9.09	1	4.55	8	36.36
		Crt-  & Com+	8	36.36	5	22.73	8	36.36
		Crt-  & Com-	5	22.73	1	4.55	0	0.00
GPT-5 (tool)	20	Crt+ & Com+	4	20.00	7	35.00	4	20.00
		Crt+ & Com-	8	40.00	6	30.00	8	40.00
		Crt-  & Com+	2	10.00	2	10.00	2	10.00
		Crt-  & Com-	6	30.00	5	25.00	7	35.00
GPT-5 (context)	22	Crt+ & Com+	10	45.45	13	59.09	0	0.00
		Crt+ & Com-	5	22.73	3	13.64	20	90.91
		Crt-  & Com+	4	18.18	3	13.64	0	0.00
		Crt-  & Com-	3	13.64	3	13.64	2	9.09
Human	41	Crt+ & Com+	34	82.93	35	85.37	34	82.93
		Crt+ & Com-	4	9.76	1	2.44	3	7.32
		Crt-  & Com+	2	4.88	5	12.20	3	7.32
		Crt-  & Com-	1	2.44	0	0.00	1	2.44

Table 9. Table 9: Example Camlang case from Camlang-CSQA-v0, with morphemic segmentation and interlinear gloss.

Type	Camlang (English)
Question	nosṇa müś ńer. meni myvá ghöt?
	(A child wants to survive. What does he need to do?)
Option	A. ṇaw ghöt (he needs to study)
	B. dit ghöt (he needs to fall down)
	C. ṇat ghöt (he needs to play)
	D. wecmylirsi irwéc ghöt (he needs to ask questions)
	E. e ghöt (he needs to eat) [correct]
	F. nepli cewmyl nak (none of the above answers)
Morphemic	nos = ṇA. müś -m = jer -$\varnothing$. me -nI. mI- n- fa -m = xöt -$\varnothing$.
Gloss	child = TOP. live-INF=want-3SG. what-ACC. WH=ACC-do-INF=need-3SG
Stems	nos: (n.) child, kid; müś: (v.) live, survive; jer: (v.) want; me: (pro.) Wh; fa: (v.) do, make, create; xöt: (v.) must, need, require; ṇaw: (v.) learn, study (+ACC object); dit: (n.) fall, fall down; ṇat: (n.) play; wecmyl: (n. ORD) question; wec: (v.) ask (someone GEN something ACC); e: (v.) eat; nepli: (adj. ORD) true, correct; cewmyl: (n.) answer; e: (v.) eat; nak: (v.) do not have, do not exist

Table 10. Table 10: Human-annotated labels for the same case as in Table  9 .

Models	Q. Parsing	Q. Meaning	O. Meaning
4o (tool)	Crt-&Com-	Crt-&Com-	Crt-&Com-
DS (context)	Crt-&Com+	Crt+&Com+	Crt+&Com+
o3 (context)	Crt-&Com-	Crt+&Com+	Crt-&Com+
GPT-5 (tool)	Crt-&Com-	Crt-&Com-	Crt-&Com-
GPT-5 (context)	Crt+&Com-	Crt+&Com+	Crt+&Com-
Human	Crt+&Com+	Crt+&Com+	Crt+&Com+

Table 11. Table 11: GPT-4o (tool) output of the case example.

GPT-4o (tool)
### Sentence Analysis
1. ”nosṇa müś ńer.”
- nosṇa: This word wasn’t found in the search results, but based on the structure, it might be a noun or verb.
- müś: This word wasn’t found in the search results, but it could be an adjective or adverb.
- ńer: This word wasn’t found in the search results, but it might be a verb or noun.
2. ”meni myvá ghöt?”
- meni: This is an interrogative form, similar to ”what”.
- myvá: This word wasn’t found in the search results, but it could be a verb or noun.
- ghöt: This word wasn’t found in the search results, but it might be a verb or noun.
### Options Analysis
- A: ṇaw ghöt
- B: dit ghöt
- C: ṇat ghöt
- D: wecmylirsi irwéc ghöt
- E: e ghöt
- F: nepli cewmyl nak
### Reasoning
Given the interrogative ”meni,” the sentence is likely asking a question about an action or state related to ”ghöt.” The options seem to provide different actions or states.
- ”ghöt” appears in several options, suggesting it might be a key verb or noun in the sentence.
- The structure of the sentence and options suggests a focus on an action or state, possibly related to ”ghöt.”
### Conclusion
Without specific translations for each word, the best approach is to match the structure and common elements. Since ”ghöt” appears in multiple options, it is likely central to the question. The options with ”ghöt” are A, B, C, D, and E.
Given the lack of specific translations, the choice is somewhat speculative. However, based on the structure and commonality, Option E: e ghöt seems to be a simple and direct match.

Equations

$${\delta }_f(X,Y)=\{\begin{array}{ll}1,& \text{if }{v}_f(X)=v_f(Y),\\ 0,& \text{otherwise},\end{array}$$

$$\text{sim}(X,Y)=\frac{{\sum }_{f\in F_X\cap F_Y}{\delta }_f(X,Y)}{|F_X\cap F_Y|}.$$

Full text

Note: This ArXiv paper presents ongoing work. The current experimental results are preliminary, based on our existing data. We may further revise Camlang and will extend the tasks in our benchmark, perform additional experiments, and update the analysis in future versions.

1 Introduction

Large Language Models (LLMs) have performed remarkably in language tasks, demonstrating versatility and adaptability (grattafiori2024llama; openai2023gpt4; openai2024gpt4o; deepseekai2025deepseekr1; openai2025gpt5). Yet, a fundamental question arising from this observed performance remains unresolved: are these models truly competent in understanding human language, or are they merely performing statistical pattern matching from their massive training corpora? This debate is central to both AI and cognitive science, and has sparked discussion on whether modern LLMs possess human-like understanding and reasoning (chomsky2023nytimes; hinton2024nobel). Validating such claims requires an evaluation setting where success cannot be gauged exclusively through memory or recalling patterns from training data, but instead depends on the ability to acquire and apply linguistic competence through an understanding of the underlying systems which constitute natural languages.

One natural way to test this ability is to present LLMs in a language they have never encountered before, mirroring the explicit rule–based learning scenarios found in human second language (L2) acquisition (klein1986second; ELLIS2005209). Cognitive science research shows that adult learners can successfully acquire grammatical rules and vocabulary of a new language and apply them to novel cases when they use their native language as an explanatory medium (lichtman2012child; rebuschat2012implicit).111Note this is opposed to infant bi-/multi-lingual acquisition (williams2016implicit; hummel2021introducing), which typically relies on natural, implicit linguistic stimulus from the environment. Such success builds on their domain-general reasoning and metalinguistic skills, which enable learners to systematically internalise the explicitly presented rules and map the new vocabulary onto pre-existing knowledge in the mental lexicon (gombert1992metalinguistic; bialystok2001metalinguistic; nagy2007metalinguistic). Controlled experiments in psycholinguistics and neuroscience further show that human adults can acquire unfamiliar language systems from grammar rules, and subsequently integrate these rules with their existing semantic and reasoning abilities (reber1967implicit; kim1997distinct; musso2003broca). This leads to the central question of this paper: can LLMs learn and use an unseen language when given only explicit grammar rules and vocabulary like human learners?

Existing LLM evaluation protocols are ill-suited to investigate this question. Most LLM benchmarks are built for English (wangglue; wang2019superglue; talmor2019commonsenseqa; chen2024see) and thus inevitably share real-world linguistic distributions with pre-training data (balloccu2024leak), making it difficult to distinguish the application of linguistic knowledge and reasoning from pattern matching. Even evaluations on extremely low-resource or endangered languages are prone to data leakage (artetxe-etal-2020-cross; chen-etal-2023-revisiting; zhang2024teaching), due to the fact that their grammar books, dictionaries, and sample texts are often well-documented and publicly available online.222Such as Kalamang: https://langsci-press.org/catalog/book/344. While a potential alternative is to train an LLM exclusively on one language (e.g., English) and test it on another (e.g., Mandarin Chinese) to ensure novelty in the tested language (chirkova2024zero), this is impractical for modern LLM systems. Most state-of-the-art LLMs are inherently multilingual (katz2024gpt; deepseekai2025deepseekr1), and training a fully monolingual model solely for this evaluation would be prohibitively costly. It would also limit the model’s general capabilities, making the results less informative about the potential of leading systems (qorib-etal-2025-just).

In this work, we introduce Camlang, a typologically novel constructed language. Camlang is constructed through a principled design process that combines typologically attested features in a novel configuration and encodes them in two explicit resources: a grammar book and a bilingual dictionary. The grammar book specifies phonological, morphological, and syntactic rules, while the dictionary provides a lexicon generated and expanded via controlled mechanisms (root generation, derivation, compounding, and loans), ensuring that the language is both plausible and learnable for humans and computational models. Camlang offers three key advantages. First, the novelty ensures that evaluation cannot be reduced to recalling seen distributions, but instead tests whether models can generalise linguistic competence to a genuinely unfamiliar system. Second, the grammar–lexicon separation enables fine-grained diagnosis of model behaviour, making it possible to distinguish failures of structural parsing from those of semantic interpretation or lexical lookup. Furthermore, human learnability guarantees that the benchmark reflects cognitively realistic language acquisition, providing a meaningful point of comparison between human learners and models. Together, these properties make Camlang a cognitively grounded and leakage-free benchmark for testing whether LLMs can go beyond statistical pattern recognition and engage in explicit deductive language learning.

Using the Camlang grammar book and English–Camlang dictionary, we construct a small test set, Camlang-CSQA-v0, by manually translating questions and answers from CommonsenseQA (talmor2019commonsenseqa) into Camlang. Since LLMs already excel at CommonsenseQA in English (krause2023commonsense; davis2023benchmarks), the bottleneck of Camlang-CSQA-v0 shifts from commonsense reasoning to applying explicit grammar rules and lexical mappings. The resulting dataset allows us to isolate the ability to integrate newly learned grammatical knowledge with pre-existing world knowledge in a controlled setting, directly paralleling explicit second-language learning in human adults, and serves as a pilot benchmark for scaling Camlang to broader evaluation tasks.

Our experiments compare multiple state-of-the-art reasoning LLMs, including GPT-5 (openai2025gpt5) and DeepSeek-R1 (deepseekai2025deepseekr1), with a human participant. Results reveal a sharp contrast: while models achieve 85–98% EM accuracy on the English version, their performance drops sharply to 21–47% on Camlang. The human participant reaches near-native accuracy (87%) using only the grammar and dictionary, which also demonstrates the practical learnability of Camlang. This suggests that, unlike human learners who naturally integrate dictionary lookup into their reasoning pipeline, current LLMs struggle to operationalise external resources when faced with Camlang’s morphological complexity, often defaulting to internal pattern matching. Human verification further reveals an important nuance that GPT-5’s reasoning traces are more often incomplete rather than incorrect, indicating a degree of metalinguistic awareness and partial rule-following ability, while it falls short of systematic grammatical mastery. In contrast, other LLMs, such as DeepSeek-R1 and GPT-o3 (openai2024gpt3o), show little evidence of such awareness, with most of their successes attributable to shallow lexical alignment. Together, these findings establish Camlang as both a reliable diagnostic resource and a controlled benchmark for probing reasoning in unfamiliar languages.

In summary, the contribution of this paper includes:

• •

It introduces a typologically plausible yet novel language, equipped with a grammar book and dictionary.

• •

We provide cognitive grounding by empirically validating Camlan’s learnability through human experiments.

• •

By adapting CommonsenseQA into Camlang, we test not only grammatical rule acquisition but also the integration of explicit rules with commonsense reasoning.

• •

We complement standard EM accuracy with human verification of reasoning traces, exposing the reliance of LLMs on shallow alignment rather than genuine metalinguistic deductive reasoning.

Beyond Camlang-CSQA-v0, we are currently extending Camlang to a broader suite of tasks (e.g., translation, parsing, and mathematical reasoning) and are exploiting its explicit grammar and dictionary for fine-grained diagnostic analyses, such as error localisation, rule generalisation, and learning-curve studies, ultimately establishing Camlang as a versatile testbed for probing the limits of LLM reasoning and their convergence with, or divergence from, human-like linguistic intelligence.

2 Related Work

2.1 Natural Language Evaluation and Benchmarks

Early benchmarks such as GLUE (wangglue), SuperGLUE (wang2019superglue), and CommonsenseQA (talmor2019commonsenseqa) focus on general natural language understanding (NLU) and knowledge reasoning. While initially challenging, these benchmarks have been soon surpassed at human-level performance by LLMs (chen-etal-2023-unisumm; hu2024improving; openai2024gpt4o; wang2025hitchhiker), motivating the development of more challenging benchmarks that target multi-step reasoning and specialised knowledge (hendryckstest2021mmlu; srivastava2023beyond; patel-etal-2024-multi; gao2024omnimath; chen2024see). Subsequent evaluations focused on tasks that are challenging even for human experts, yet where LLMs have demonstrated remarkable success, such as the U.S. Bar Exam (katz2024gpt), USMLE (kung2023performance), MATH (hendryckstest2021mmlu), AIME (aime), and even International Mathematical Olympiad problems.

While these results are impressive, they are all conducted in familiar natural languages and established task formats, allowing LLMs to exploit potential memorisation or pattern matching from training data. In contrast, our work employs a controlled constructed language to eliminate prior exposure, enabling a cleaner assessment of a model’s ability to learn and apply entirely new linguistic and reasoning rules from scratch.

2.2 Artificial, Constructed, and Programming Languages in NLP

Constructed language (Conlang) in cognitive science has long been used to study how humans acquire grammatical structure from limited exposure (reber1967implicit; marcus1999rule). Inspired by these paradigms, NLP researchers have proposed controlled synthetic benchmarks such as SCAN (lake2018generalization), NACS (bastings2018jump), and gSCAN (ruis2020benchmark) to evaluate compositional generalisation of NLP systems. More recently, white2021examining and hopkins2022towards use artificial languages to probe the inductive biases of neural language models and to improve ecological validity, respectively. kallini-etal-2024-mission extend this line of work by creating impossible languages via systematic violations of linguistic universals, demonstrating that LLMs learn naturalistic languages far more efficiently than impossible ones. In contrast, our Camlang is deliberately designed to be typologically plausible yet absent from attested natural languages, ensuring novelty without introducing unnatural structures. This allows us to test whether LLMs can integrate explicitly presented grammatical rules with pre-existing semantic and commonsense reasoning to solve downstream tasks, moving beyond pure grammatical learnability to functional language use.

Beyond artificial grammars, programming languages have also been used as testbeds for evaluating LLMs’ rule-following and compositional abilities (chen2021evaluating; li2022competition). While such languages provide a fully formal and unambiguous setting, they differ fundamentally from natural language: programming languages are intentionally engineered for executability and determinism, lacking paraphrastic variation, pragmatic inference, and the typological diversity of human languages (clark1996using; allamanis2018survey; chomsky2014aspects). In contrast, our Camlang is designed to be typologically plausible and naturalistic while still controlled, enabling us to probe whether LLMs can integrate explicit grammatical instruction with semantic and commonsense reasoning in a way that is closer to natural language learning.

Most importantly, the emphasis on naturalness distinguishes our approach from the above systems. If a constructed language were based on unattested or linguistically implausible rules (kallini-etal-2024-mission), the evaluation would risk collapsing into a test of symbol manipulation rather than language learning, producing results that are uninterpretable for both cognitive and NLP perspectives. By ensuring that Camlang’s grammatical features are all attested in human languages and that their combinations respect known typological constraints, we guarantee that it remains within the distribution of humanly possible languages. This design choice makes our evaluation cognitively comparable to adult second-language learning and scientifically credible as evidence of whether LLMs can acquire and apply genuine linguistic competence from explicit description.

2.3 Cross-lingual Evaluation on Low-resource Languages

Evaluation of LLMs on low-resource languages has relied on multilingual benchmarks such as XQuAD (artetxe-etal-2020-cross), TyDi QA (clark2020tydi), and FLORES (guzman-etal-2019-flores), as well as task-specific datasets in certain low-resource languages (adebara2022towards; chen-etal-2023-revisiting; zhang-etal-2025-read). While these datasets cover a broad range of typologically diverse languages, many are derived from publicly available sources such as Wikipedia, parallel corpora, or well-documented analyses and dictionaries. This leads to the data leakage problem that example sentences may have been seen during model pre-training (carlini2023extracting; dong2023abilities), allowing models to recall rather than genuinely learn the target language.

Recent work has turned to controlled settings based on Olympiad-in-Linguistics-style puzzles and unseen-language tasks (chi2024modeling; goyal2025iolbench; zhang2024teaching; lian2025lingbench++), which construct problems from scratch to test grammatical rule induction from minimal examples. However, these puzzles often focus on isolated phenomena without providing a coherent, system-wide grammar. In contrast, our Camlang offers a fully specified, typologically plausible yet unattested language with a systematic grammar and lexicon, enabling controlled and repeatable evaluation of whether LLMs can learn and consistently apply explicit grammatical rules, and integrate them with pre-existing commonsense knowledge in downstream reasoning tasks.

2.4 Human Adults Language Learning and Metalinguistic Skills

Second-language acquisition research shows that adults are capable of learning grammatical rules through explicit instruction (krashen1982principles; ELLIS2005209). Metalinguistic awareness—the ability to reflect on and manipulate linguistic form—has been shown to facilitate grammar learning (tunmer1984development; thomas1992metalinguistic; nagy1995metalinguistic), particularly among bilinguals (bialystok2001bilingualism). Psycholinguistic studies further demonstrate that learners can acquire novel morphosyntactic systems from written descriptions alone, and then apply these rules in new contexts by integrating them with existing semantic and reasoning abilities. In NLP, recent work has examined whether LLMs can similarly make use of explicit linguistic descriptions. For example, prior work has shown that prompting LLMs to generate grammatical explanations (scheinberg2025explain) or providing them with grammar books and dictionaries (behzad-etal-2024-ask; hus2024back; zhang2024teaching) can improve performance on grammar judgment and low-resource translation tasks.

Compared to them, our work greatly differs in two ways. First, instead of relying on existing natural languages, where partial exposure through pre-training cannot be excluded, we use Camlang, a fully constructed, typologically plausible yet unattested language, ensuring zero data contamination. Second, our grammar book is complete and systematically designed, covering multiple levels of linguistic structure and enabling targeted assessment of whether LLMs can internalise and consistently apply explicit grammatical rules. By integrating this with commonsense reasoning tasks, we create a controlled setting to test whether LLMs can combine newly learned linguistic rules with pre-existing world knowledge for functional language use.

3 The Camlang Language

Camlang is a constructed language designed to evaluate whether LLMs can understand and apply new linguistic knowledge from explicit rules, rather than from prior pre-trained knowledge. In this work, the term “constructing the language” refers specifically to the creation of two resources: a grammar book, written in English with Camlang samples, that systematically describes the phonology, morphology, syntax, and other aspects of the language, and an English–Camlang bilingual dictionary, covering all relevant lexical stems used in the evaluation tasks.333Unless otherwise specified, in the later sections, the terms grammar book and dictionary refer to the Camlang resources we create. Note that we describe Camlang as an unseen language not because each of its individual linguistic features is unattested in other natural languages, but because the combination of features in Camlang is, to the best of our knowledge,444We acknowledge the possibility that a language similar to Camlang may exist or have existed, but has not been documented in the linguistic record to our knowledge. absent from any known language.

The subsequent subsections provide a detailed description of the Camlang language and its construction. We first introduce the design principles of Camlang in § 3.1. Then, we detail our construction and implementation in § 3.2. In § 3.3, we present a high-level linguistic description of the language, focusing on its major phonological, morphological, and syntactic characteristics. Finally, in § 3.4, we provide an empirical evaluation of Camlang in a broader context, demonstrating its clarity, learnability and novelty as a naturalistic system while confirming that it is distinct from other known languages.

3.1 Design Principles and Philosophies

As stated, our goal is to simulate adult language learning for a new language in order to evaluate the reasoning abilities of LLMs. To serve this purpose, the language must reasonably resemble genuine human languages rather than arbitrary symbol systems (kallini-etal-2024-mission), which makes naturalness a primary consideration. If the grammar were constructed with arbitrary, unattested rules, the evaluation would reduce to a symbol-manipulation game rather than a meaningful simulation of language learning. In such cases, both humans and models can fail simply because the rules are unnatural or even unlearnable, making results uninterpretable. At the same time, Camlang should be sufficiently rich to cover diverse grammatical phenomena, novel enough to avoid coinciding with attested languages, and compositional so that its rules apply systematically and productively. Thus, the development of Camlang is driven by the below principles:

• •

Naturalness, to ensure plausibility as a human language;

• •

Richness, to cover a wide spectrum of grammatical phenomena;

• •

Novelty, to avoid the overlaps of the overall combination of linguistic features with attested languages;

• •

Compositionality, to guarantee systematic and productive rule application.

Together, these principles provide both a practical foundation and criteria.

Naturalness.

Naturalness requires that the constructed language could, in principle, exist as a human language on Earth, and thus must fall within the distribution of humanly possible languages. We further distinguish three types of criteria for naturalness: (1) naturalness involves fundamental properties of human language, such as design features (hockett1960origin) and the capacity for recursion (doi:10.1126/science.298.5598.1569), and; (2) naturalness requires that the grammatical features employed are attested in human languages. Unattested and empirically unproven innovations (kallini-etal-2024-mission), such as a phonological rule turning every second phoneme into “/a/” if the word begins with “/b/”, should be excluded, and; (3) naturalness concerns the probabilities of feature combinations, where certain features as rarely co-occurring and others as mutually implicative (greenberg1963some). These linguistic universals (greenberg1963some; gorriereduplication; moravcsik1978distribution) constitute the most complex constraints that Camlang must respect to remain naturalistic.

Richness.

We aim to make the language rich enough for our purposes (haspelmath2005world; evans2009myth). In particular, (1) it should possess a diverse range of grammatical features, allowing us to analyse which features are more challenging for LLMs and what kinds of linguistic reasoning are lacking in LLMs, and; (2) it should possess enough vocabulary and grammatical constructions to cover basic language usages, and; (3) it should allow one meaning to be expressed in alternative stylistically differentiated ways. Also, it should be noted that Camlang is not intended to match the full richness and complexity of a natural language such as English. Instead, it is a controlled, moderately complex system designed to probe specific aspects of LLM language learning.

Novelty.

The constructed language is intended to be a novel language, with unseen lexicon and grammar. Therefore, it is important to limit coincidences and overlaps with other languages that we know to our knowledge (comrie1989language; evans2009myth). In particular, we consider two aspects: (1) from the lexicon aspect, the linguistic forms (spellings, in this case) should avoid coinciding with linguistic forms of other languages that mean the same thing, and; (2) from the grammar aspect, it should also be different from those of other languages to a significant extent. We emphasise again that the grammatical features are all attested in human languages, so the novelty refers to novel combinations of features, which means that the underlying logic of this language should differ from other languages.

Compositionality.

The grammar of the constructed language is intended to be learnable with explicit description and demonstrative examples only. Therefore, it must be compositional, in the sense that grammar rules are factorised and composed productively (chomsky2014aspects): any word stem of a given part-of-speech (PoS) can take the specified inflectional slots, and functional categories always co-occur according to fixed linear orders and scope relations, unless exceptional rules or items, which should remain peripheral, are specifically mentioned in the grammar book. This ensures that the legality and linearisation of unseen morpheme combinations are decidable from the grammar book alone.

Other Considerations.

Apart from the above four main principles, the design of the constructed language must also be regular, concise, and straightforward enough to ensure tractability so that both humans and computational models can process it reliably and efficiently. Meanwhile, we deliberately abstract away from peripheral world-building considerations, such as putative geography (janda2013geography), sociocultural embedding (lantolf1995sociocultural), or franchise-specific lore (androic2019dialects) (e.g., Tolkienian settings or Parseltongue), and treat the constructed language purely as a typologically plausible testbed for explicit-rule learning. It should also be noted that the main considerations sometimes lead to contradictions. For example, a more naturalistic feature combination is more common and therefore less novel, and a richer grammar with more features is less naturalistic due to the greater number of uncommon features and marked feature combinations. This requires the four considerations to be balanced in a trade-off in our implementation.

3.2 Construction and Implementation

As discussed in § 2.2, we have not seen any attempt to construct a full-fledged plausibly human language for research purposes. Conlanging, or the construction of new full-fledged languages, is practised mostly for personal recreation, interlanguage communication (Auxlangs such as Esperanto (janton1993esperanto) and Interlingua (lu2018neuralinterlinguamultilingualmachine)), or world-building in fiction, most often without the need for rigorous scientific methodology. Therefore, the construction and implementation of Camlang as a research tool is mostly exploratory and pioneering to our knowledge.

Unlike existing conlangs (lake2018generalization; bastings2018jump), Camlang is designed with explicit operational goals: it must be both learnable by humans and usable by computational models. To this end, Camlang is grounded in two resources: a grammar book and a dictionary. The grammar book specifies how meanings are compositionally expressed through phonology, morphology, and syntax, while the dictionary documents the lexical inventory and its distribution. Together, these artefacts provide the operational foundation that enables both humans and models to learn and use the language. The design of Camlang’s grammar and dictionary is primarily led by a Cambridge linguistics student with prior experience in conlanging. Another three linguistics students from Cambridge and Oxford serve as core contributors in the refinement process. This ensures that all features are systematically analysed from multiple perspectives.

We next describe the conception and organisation of the initial grammar book (§ 3.2.1) and dictionary (§ 3.2.2). We then explain how these resources evolves through iterative human translation exercises and revisions (§ 3.2.3), resulting in the final versions used in our experiments.

3.2.1 Grammar Book

The grammar book is the central artefact of Camlang, specifying how meanings are compositionally expressed through phonology, morphology, and syntax. We first describe how the characteristic features of Camlang are chosen and detailed, and then the organisation of the grammar book.

Feature Selection.

The naturalness and novelty principles dictate that Camlang’s features must all be attested but combined in an innovative configuration. With this goal, we start by creating the defining features of Camlang via the selection of existing features from language groups with distinct typologies, including but not limited to: vowel harmony from Turkic languages (korn1969types), consonant mutation from Celtic languages (green2006independence), and an active left periphery from Romance languages (beninca2004left). To achieve richness and compositionality goals most directly, the author designed Camlang to have highly regular, rich morphology with prevalent simple exponence.

Specification.

With the features preliminarily decided, we translate them into explicit phonological, morphological, and syntactic rules. We deliberately merge morphology and syntax in the grammar book into a single component (“Morphosyntax”), because many syntactic functions in Camlang are morphologically realised, and their description requires treating the two levels jointly.

• •

Phonology Specification. In line with the novelty principle, the phonological traits borrowed from attested languages are systematically reinterpreted within Camlang’s original phoneme inventory and orthography, rather than copied directly. For instance, both voiced and voiceless stops undergo aspirate mutation into voiceless aspirated stops—a pattern distinct from Middle Welsh, where only voiceless stops undergo aspirate mutation while voiced stops remain unaffected.

• •

Morphosyntax Specification. To achieve richness and compositionality, we first identify core grammatical categories such as nominal number, verbal aspect, and clause types. We then associate these categories with plausible morphological and syntactic patterns, including prefixation, suffixation, and verb fronting, and assign functional morphemes phonological shapes consistent with the designed phonology. For instance, yes–no questions are encoded both by verb fronting and by the attachment of a proclitic (“nAs=”) containing a harmonising vowel underspecified for backness, in accordance with vowel harmony.

Organisation.

Language is conventionally described in a modular sequence, proceeding from phonetics, phonology, morphology, and syntax to semantics and pragmatics, gradually moving from linguistic form to linguistic meaning (fromkin2017introduction). In our case, phonetics is first excluded, since Camlang is not intended to be spoken. Accordingly, the initial grammar book is organised as:

• •

Orthography and Phonology, reflecting the rich phonological system and rules that drastically alter the written form;

• •

Nominal Morphology, Verbal Morphology, and Derivational Morphology, capturing the highly productive morphology of Camlang;

• •

Morphosyntax, which subsumes syntax due to the strong interactions between syntactic structures and morphological marking.

It should be noted that semantics and pragmatics are left largely unspecified in the initial version, both because our evaluation focuses on formal structure and because the dictionary captures much of the semantic content.

As about to be discussed in § 3.2.3, the grammar book has been incrementally changed during refinement.

3.2.2 Dictionary

The dictionary records the forms and meanings of Camlang lexemes and multi-word expressions, and thus serves as an indispensable reference for both humans and models. Guided by our design principles, we pursue two objectives: (1) a plausible, naturalistic distribution of lexical items in form and origin; and (2) minimal arbitrariness to support human and machine learnability.

Lexical Sourcing and Formation.

To meet these objectives, we balance five sourcing mechanisms.

• •

Generated Roots (Native Camlang Word). Given the phoneme inventory and phonotactics, we implement a root generator that samples mono- and disyllabic forms from a slot-conditioned probability table (see Appendix § LABEL:sec:root_generator). We select a newly generated form whenever a novel root is required, discarding accidental duplicates unless deliberate homonymy is desired. More basic concepts tend to receive monosyllabic roots, more complex ones disyllabic roots (mimicking synchronically opaque derivations). Sound symbolism occasionally informs choices, e.g., high front vowels for sharp objects vs. low round vowels for round objects (cwiek2022bouba).

• •

Derived Words. We exploit derivational morphology (tyler1989acquisition) to limit the number of independent roots. Many derivatives are synchronically transparent and compositional, e.g., ‘cak’ (‘to work’) $+$ ‘-mA4’ (deverbal nominaliser) $\to$ ‘cakma’ (‘work’); others are conventionalised and semantically opaque, e.g., ‘kök’ (‘to look’) $+$ ‘-GA4s’ (abstract nominaliser) $\to$ ‘kökkys’ (‘hope’), both attested cross-linguistically.555We use “transparent/opaque” in the synchronic sense; c.f., standard discussions of derivational semantics (beard2017derivation). Derivation also includes calques (loan translations), e.g., ‘soruk’ (‘compute’) $+$ ‘pI4-’ (‘agent’) $\to$ ‘pusóruk’ ‘computer’ (shuttleworth2014dictionary).

• •

Compounds. For semantically decomposable complex concepts (fff14662-cc84-3012-8104-515862aabe38), we form compounds from existing words, thereby limiting new roots and maintaining agglutinative consistency, e.g., ‘kityb-chog’ (‘desk’) from ‘kityb’ (‘book’) $+$ ‘cog’ (‘table’), usually orthographically marked with a hyphen and morphologically marked with the ezafe proclitic on the compound head or rightmost item.

• •

Opaque Loans. To simulate superstratal influence (kahane1986typology), a controlled portion of items is adapted from archaic or less familiar sources, e.g., reconstructed forms, whose phonological shapes differ from the native stock. These items model historically plausible borrowing while remaining difficult for models to identify by external knowledge trivially.

• •

Transparent loans. To reflect cross-linguistic diffusion of modern concepts (wierzbicka2006concept), a small set of items is adapted from major languages, e.g., rajdiw ‘radio’, remaining recognisable in meaning despite phonological adaptation. Together with opaque loans, this yields a realistic loan profile.

Semantic Mapping Policy.

We do not engineer semantics to be adversarial. Instead, where possible, Camlang lexemes map one-to-one to English equivalents. Nevertheless, motivated sense splits or mergers are permitted, e.g., ṇat ‘play (with toys)’ vs. get ‘play (sports/instruments)’ and ǵomluṇ (‘usually/generally’), reflecting attested partitions of semantic space via metaphor/metonymy.

Dictionary Structure and Readability.

Entries are alphabetised by the underlying form (the morphophonological base). For each entry, we record: (1) English gloss (and relevant subcategorisation, e.g., genitive vs. accusative complements); (2) part of speech and honorific status; (3) underlying form; (4) surface citation form; and (5) etymology. (5) is not released to models to avoid leakage of external knowledge. (3) and (4) anchor morphological analysis by distinguishing underlying from surface realisation.

Since Camlang is morphologically rich, many surface forms are not listed verbatim. Thus, the dictionary’s underlying forms serve as lemmata for analysis and retrieval, both for humans and models.

3.2.3 Iterative Refinement

Starting from the initial grammar book and dictionary, we perform iterative refinements with three aims: (1) to improve learnability and typological plausibility by removing unnecessary complexities and outliers; (2) to expand the lexicon and specify constructions required for naturalistic expression of complex meanings; and (3) to simulate Camlang’s plausible historical development, so that new features arise via naturalistic grammaticalisation rather than arbitrary design.

In each round, four annotators (linguistics undergraduates from Cambridge and Oxford who have experience in constructed languages and L2 acquisition) independently translated a set of English sentences or dialogues using the current grammar and dictionary. The source material increases in form and expressive complexity across rounds: starting with simple declaratives (e.g., ROCStories (mostafazadeh-etal-2016-corpus)), moving to short daily dialogues, and culminating in longer interview-style exchanges.666https://www.ox.ac.uk/sites/files/oxford/media_wysiwyg/Interview%20Podcast%20-%20Transcript.pdf Translation divergences are then used diagnostically to locate missing constructions, underspecified rules, or structural inconsistencies. The discussion also leads to fixes incorporated into the next grammar and dictionary revision. Note that differences across translations are not treated as errors, since Camlang permits paraphrastic variation and multiple acceptable syntactic and pragmatic realisations. Only divergences indicating under-specification or inconsistency trigger revisions. For lexical expansion, new stems are added only when required to express concepts absent from the current dictionary. After refinement, we further manually review and provide feedback on contradictions or underspecified interactions. We stop after Round 3 when no further feedback beyond minor editorial clarifications is provided. Across all rounds, each annotator works with a total of 2,263 tokens, 737 types, and 207 sentences. While modest in size, this corpus fits our purpose to assess whether the grammar–dictionary specification affords adequate coverage of naturalistic expressions across diverse syntactic structures and discourse contexts.

Here, we summarise the major refinements to the grammar book in each round:

• •

Round 1. The discussion and refinement focus on surface-form consolidation and core nominal/clausal regularisation. The refinement mainly addresses the standardisation of orthography–phonology conventions (venezky2013role) and re-organisation of the nominal system around simple exponence, together with the grammaticalisation of oblique pronouns into pronominal clitics (borer2020syntax) (remaining disharmonic) and a reanalysis of the accusative/genitive contrast with postpositions as a dynamic–stative opposition (koopman2010prepositions). We also fix the basic case assignment and ordering for double-complement predicates.

• •

Round 2. This round focuses on systematising the behaviour of clitics, reworking some high-frequency morphemes’ forms while adding sources of stylistic variation. In particular, we specify clitic stacking and movement rules explicitly (roberts2010agreement), and streamline the core paradigms (pronouns, TAM, information-structure and interrogative markers) to improve distinctiveness and clarity. In addition, we adopt limited periphrastic resources (do-support and light verbs), and replace originally arbitrary evidential contrast and agreement morphology with diachronically motivated ones, while introducing a vowel–glide dissimilation rule to increase phonological opacity (mccarthy1999sympathy). When annotating the dialogues, we recognise that the lack of a pragmatic architecture limited the stylistic scope of Camlang. In response, we devise a topic-based honourific register (agha2006language), introducing a level of controlled pragmatic application to account for the range of contexts in which a conversation might take place (parsons2017systematic).

• •

Round 3. We mainly demarcate functions of some grammatical categories and advanced Camlang’s macro-typology in a diachronically naturalistic way (greenberg1995diachronic). In particular, we separate the definiteness function from the linker by introducing an independent definite marker, and introduce full reduplication to meet typological expectations (haspelmath2005world). We further establish a first-person inclusive/exclusive split and make specificity-driven Differential Object Marking explicit (rodriguez2007syntax). We codify situations of use of mood, modality, evidentiality and epistemic markers. And we turn Camlang from agglutinative to polysynthetic via grammaticalisation of pronominal clitics into obligatory head-markers.777A historically canonical development captured as a morphosyntactic cycle (+1905+285+313). We also structured mathematical expressions for future applications.

After refinement, we also add a General Description section, which summarises the language’s major features. The resulting grammar book contains 453 sentences and 9,477 tokens.

For the dictionary, new entries are added on demand for texts, and forms are amended only when grammar changes (e.g., orthography or derivational rules) impact citation forms. At the time of experiments, the dictionary contains 1,511 entries. We show the distribution of the lexical sourcing and formation as in Table 1.

Using the annotated data from the above procedure, we also measure the cross-annotator consistency across the three rounds as an indirect indicator of Camlang’s learnability under an explicit grammar and dictionary (metrics and results in § 3.4).

3.3 The Overview of Camlang

To avoid potential data leakage to LLMs, we do not release the full grammar book and dictionary at this stage. Instead, we provide a high-level overview of the linguistic profile of Camlang. We also provide a profile of the World Atlas of Language Structures (WALS)-style features in Appendix LABEL:sec:wals.

Phonology.

Camlang has 50 consonantal phonemes, among which 30 do not appear in native citation forms, but result from loaning or consonant mutation, an important morphophonological process that operates at cliticisation boundaries. The inventory is largely symmetric with four places of articulation, and each phoneme can appear at both onset and coda. Camlang has a Turkic-type harmonic 8-vowel system with vowel reduction, which neutralises contrast between three vowels at unstressed positions. Camlang has strict (C)V(C) phonotactics that prohibits adjacent vowels, often satisfied by epenthesis. The combination of these features means learners of Camlang must apply phonological rules in the specified sequence in order to derive surface forms from underlying forms or reversely. For example: \ex \begingl\glpreamblelichéwcymyśür// \glaSegmented: li- chew -cy -my -śür // \glbMorphemic: lI= x= cew -RED $-mA^{4}$ -s =jUr// \glbGloss: 2SG= EZ= answer -PROG -NMLS -GEN =at// \glft‘when you are answering’// \endgl\xeThe Camlang word in Case 3.3 consists of seven morphemes, none of which appear in underlying form. Learners cannot comprehend or produce this word if they do not apply relevant phonological rules, because the ungrammatical result of simple morphemic concatenation, *lixcéwcemasjur, would never appear.

Morphosyntax.

Camlang is a polysynthetic language with obligatory head-marking morphology, consistent pro-drop, and predominant simple exponence. For derivation and inflection, Camlang uses both affixes and clitics extensively. Camlang has a slight preference for suffixation over prefixation and circumfixation, and a slight preference for procliticsation over encliticisation. Camlang’s nouns are marked for many grammatical functions, such as definiteness, plurality, and case, and can take postpositional enclitics. Their possessors are marked with proclitics. Camlang’s verbs are marked categories such as for valency, aspect, tense, evidentiality, mood, and finiteness. Finite subjects are marked on verbs with agreement suffixes while other arguments are marked with proclitics. Nonfinite verb forms like nominalisations are extensively used in complex constructions realised as subordination in English.

The combination of these features means learners of Camlang must use morphosyntactic knowledge to analyse syntactic dependencies and grammatical functions when comprehending or producing agglutinative word forms, which is more challenging for learners with analytic mother tongue languages or training languages like English and Mandarin Chinese. For example, the written word in Case 3.3 contains semantic information equivalent to that of an English subordinate clause due to its morphological complexity, and would be incomprehensible if the learner fails to perform morphological segmentation and analysis according to specific grammar points.

Due to the heavy functional load taken by Camlang’s morphology, its syntactic configurations are relatively simple. Structurally, all projections lower than the Complementiser Phrase 888for Complementiser Phrase, see Chapter 9 of radford2004english in Camlang are head-final, giving SOV default word order. The active left-periphery hosts the information-structural functions of topicalisation and focalisation, and other word orders are used in many marked clause types. Otherwise, word orders are usually fixed and decided by the hierarchical structure.

Sociopragmatics.

Camlang does not exhibit a T/V distinction in second-person reference forms (8407211f-5665-3fa0-bfd2-5965615377de), but showcases lexicalised register as found in languages like Thai  (alma990158258210107026), with some concepts being denotable by an ordinary linguistic variant and an honorific linguistic variant. Moreover, morphosyntactic constructions may be used as stylistic devices to give the text a more native, natural style as opposed to a more foreign, translationese style. Camlang’s sociopragmatic description is currently underdeveloped, but we hope to enrich it from this basis and use it to explore discourse-related tasks.

3.4 Analysis and Evaluation on Camlang

3.4.1 Grammar Clarity and Practical Learnability

We assume that if the grammar–dictionary specification is sufficiently explicit and internally consistent, then the Camlang learners will converge on the same morphological choices and on similar clause-level linearization for a given English input (reber1967implicit; ELLIS2005209; lake2018generalization). In § 3.2.3, we iteratively refine the grammar and dictionary through three rounds of translation by four annotators. We thus analyse the cross-annotator consistency as indirect evidence about the language learnability.999Our use of “learnability” is empirical, different from classical notions, such as (gold1967language) and (valiant1984theory). In L2 acquisition, the learnability/teachability hypothesis has also been discussed (pienemann1989language). We do not claim formal identifiability here and focus on practical learnability under explicit instruction. We calculate the averaged Rouge (lin2004rouge) scores between different annotators at different rounds, and report the results at two granularities. At the word-level, we tokenise the Camlang sentences by spaces. At the morpheme-(surface morph) level, we manually tokenise words into morphemes according to the grammar/dictionary. The results are shown in Table 2.

Generally, from Round 1 to Round 3, Rouge increases consistently at both granularities (word: R1 46.23$\to$62.91, R2 19.09$\to$40.99, RL 44.27$\to$60.10; morpheme: R1 67.51$\to$75.04, R2 36.27$\to$55.40, RL 61.89$\to$69.88). The largest gains in Rouge-2 indicate increasing stability of local combinations (affix order, clitic/particle placement), while gains in Rouge-L point to a more constrained global clause linearisation. These trends persist despite rising source complexity across different rounds (from simple declaratives to longer complex dialogues), suggesting that the grammar–dictionary specification became operationally clearer and thus affords practical learnability under explicit instruction. Morpheme-level scores are consistently higher than word-level ones (e.g., Round 3 RL 69.88 vs. 60.10). Decomposing words exposes shared functional structure (affixes, clitics, and morphotactic slots) and reduces lexical sparsity, thereby increasing the chance of $n$-gram overlap across independently produced translations. This is in line with our morphologically rich design. Accordingly, higher morpheme-level Rouge indicates convergence on the same grammatical machinery (choice of functional morphemes and slot order) rather than mere coincidence at the word level. It should be noted that, because EM metrics are boundary-convention–sensitive, morpheme-level Rouge should be read as a conservative estimate.

We acknowledge that the present analysis only considers the mapping from English into Camlang, i.e., the production aspect. A fuller account of practical learnability would also require examining comprehension in the opposite direction, i.e., the comprehension aspect (vanpatten2004processing; pickering2013integrated). We complement it later in § 5, where one human participant is asked to solve the commonsense question answering task given the grammar book and dictionary, and achieves an accuracy around 87%, thereby addressing this aspect of bidirectional learnability.

3.4.2 Novelty and Typological Positioning

Following (georgi-etal-2010-comparing), we adapt a WALS-style feature-based method to compare the similarity between two languages. WALS (haspelmath2005world) is the most widely used and largest typological database, covering 2,662 languages with 144 chapters of features across domains such as phonology, morphology, and word order.101010https://wals.info/chapter/s1 Since typological inventories are sparse and some features are not directly comparable across languages, we restrict comparison to the intersection of features available for both languages. We manually construct a WALS-style typological profile for Camlang based on its grammar and dictionary, aligning each relevant structural property with the corresponding WALS feature value (Appendix LABEL:sec:wals). Overall, Camlang contains 134 features.

Let $F_{X}$ and $F_{Y}$ be the sets of features encoded for languages $X$ and $Y$, respectively. We restrict comparison to their overlap $F_{X}\cap F_{Y}$, in order to avoid spurious differences due to missing values and to ensure comparability. For each feature $f$ in the overlap, we assign an indicator score:

[TABLE]

where $v_{f}(X)$ and $v_{f}(Y)$ are the values of feature $f$ in languages $X$ and $Y$, respectively. The overall similarity is then defined as the proportion of matched features:

[TABLE]

Following georgi-etal-2010-comparing, we set a minimum overlap threshold and report results across different settings (20, 30, 40 and 50 features). For comparison, we compute the similarities between each top neighbour of Camlang and all other languages under the same minimum-overlap setting, and the similarities between Camlang and high-resource languages.

Table 3 presents Camlang’s closest neighbours under different feature-overlap thresholds. At lower thresholds ($\geq$20, 30), Camlang appears close to Indic languages such as Awadhi, Kumauni, and Bhojpuri, as well as Afro-Asiatic and Sino-Tibetan languages. However, these results are based on small overlaps and are less stable. More meaningful comparisons emerge at higher thresholds ($\geq$40, 50), where Camlang’s closest neighbours include Djingili (Mirndi), Rawang (Sino-Tibetan), Tatar (Turkic), Mansi (Ugric), and Dagur (Mongolic), with similarities in the 0.67–0.74 range. These results suggest that Camlang does not collapse into any single language family, but rather occupies a mixed region of typological space, combining features from multiple families in a stable and coherent way.

Table 4 further compares Camlang’s top-1 neighbours against their own closest languages. Natural languages such as Awadhi (an Indo-Aryan language) and Bhojpuri (also an Indo-Aryan language) have higher similarities with their genealogical relatives (e.g., Indic languages), with near-perfect similarity scores. Conversely, in cases such as Djingili and Mansi, Camlang itself ranks among their top neighbours, often alongside typologically distant families such as Korean, Basque, or Quechuan. This contrast highlights that Camlang is sufficiently naturalistic to be typologically comparable to real languages, yet it is distinct in that it spans across multiple genealogical groups rather than grouping tightly with one family.

Table 5 shows that Camlang maintains relatively low similarity with major high-resource languages such as English (0.44), French (0.42), Mandarin (0.45), and Japanese (0.49), confirming that it does not trivially overlap with the typological profiles of widely studied languages. Camlang shows a higher similarity with Turkish (0.63, based on 133 overlapping features). This is consistent with our design that Camlang incorporates typological traits such as vowel harmony and agglutinative morphology that are characteristic of Turkic languages (§ 3.2.1). Taken together, these results demonstrate that while Camlang remains distinct from common high-resource languages, it naturally aligns more closely with Turkic languages, reinforcing both its novelty as a constructed language.

3.4.3 Richness Analysis

We measure the richness of Camlang by comparing its WALS features with other languages. Figure 1 shows the distribution of feature counts across documented languages. We see that most languages are annotated with only a few dozen features, which can be attributed to limited documentation and scarce resources for many low-resource and endangered languages. As a result, their typological richness in WALS is insufficiently captured. Using such languages for evaluation would cannot provide a comprehensive test of linguistic competence and risk biasing the evaluation towards a narrow subset of phenomena. In contrast, Camlang has 134 features, placing it above 97.52% of WALS languages. Compared to high-resource languages such as English (159), French (158), and Mandarin (153), while Camlang does not aim to replicate the full complexity of natural languages, it ensures sufficient typological richness to support systematic evaluation.

4 The Camlang-CSQA-v0 Test Set

With the grammar book and dictionary stabilised, we construct Camlang-CSQA-v0, a small-scale multiple-choice dataset of commonsense questions in the Camlang language, derived from CommonsenseQA (talmor2019commonsenseqa).

We choose CommonsenseQA (talmor2019commonsenseqa) as the source task because it is a well-established benchmark for evaluating commonsense reasoning in NLP, consisting of daily scenarios that require integrating world knowledge and simple reasoning. Moreover, prior work has shown that the knowledge required to answer CommonsenseQA is already well represented in current LLMs (aggarwal2021explanations; krause2023commonsense; bian2024chatgpt). By translating the task into Camlang, the challenge no longer lies in retrieving commonsense knowledge itself, but in whether models can correctly interpret the grammar book and dictionary and apply explicit rules to integrate this unfamiliar linguistic input with their existing commonsense knowledge.

4.1 Task Definition

The task is multiple-choice question answering in Camlang. Each instance consists of a commonsense question and six candidate answers, including one “none of the above” option. The systems are required to select the single correct answer based on the Camlang input, with access to the grammar book and dictionary.

4.2 Data Selection and Pre-processing

We first randomly select 50 instances from the CommonsenseQA test set.111111https://huggingface.co/datasets/tau/commonsense_qa/ In practice, we observe that some instances are either less reasonable or contain ambiguous answer options. We first modify or replace such options to ensure that the correct answer is more reasonable. Instances with ambiguous questions are removed. Another problem is that the Camlang dictionary is limited and cannot cover all entities in CommonsenseQA. To avoid continuously expanding the dictionary with low-frequency words, we thus replace or modify such entities with semantically similar ones already present in the dictionary, if they are not essential for reasoning or answer selection. For example, given the question “How could a weasel go to another continent?” with the correct answer “by ship”, we replace “weasel” with “cat”, which is already covered by the dictionary. In addition, we insert an extra option “none of the above” into each instance. After this adjustment, we manually select the gold answers, since the CommonsenseQA test set does not provide them. The resulting set of multiple-choice instances is then translated from English into Camlang, yielding a cleaner set of 47 questions.

4.3 Annotation

Given the English questions and multiple-choice options, we ask the core annotator, who participates in Camlang development, to translate them into Camlang, with reference to the grammar book and dictionary. The translation is not required to follow English syntax in a word-to-word manner; instead, the annotator is allowed to adapt the sentences according to the typological characteristics of Camlang. When no reasonable substitution is available in the existing dictionary, the annotator extends the lexicon with new entries. In addition to the translated sentences, we also provide gloss-level annotation, which specifies the morpheme-by-morpheme alignment and thereby ensures both accuracy and interpretability of the annotation. For example, given the question “How could a cat go to another continent?”, we interpret the gloss as “cat=TOP how go-INF=can another continent-GEN?”. After the initial annotation, a second annotator, who is also involved throughout the grammar design process, reviews and revises the translations to ensure consistency and correctness.

4.4 Dataset Statistics

We analyse the length of questions before and after translation. Lengths are measured in words, where both English and Camlang are tokenised by spaces. The distribution is shown in Figure 2. On average, English questions contain 12.34 words, whereas their Camlang counterparts are shorter with 7.32 words.

We observe three main trends. First, both English and Camlang questions are concentrated between 6–15 words, reflecting the original distribution of CommonsenseQA. Second, Camlang exhibits a higher proportion of very short questions (0–5 words), owing to its morphological compactness: information that requires separate function words in English (e.g., auxiliaries, prepositions) is often expressed via affixes or clitics in Camlang. Third, English questions extend to much longer tails (over 20 words), while no Camlang question exceeds 20 words. This indicates that translation into Camlang systematically yields more compact formulations, while preserving the semantic content needed for reasoning.

5 Experiments

With the grammar book, dictionary, and Camlang-CSQA-v0, we conduct experiments to evaluate both human and model performance. This pilot experiment serves two main purposes. First, we assess the Camlang evaluation paradigm by testing whether human participants, given only the grammar book and dictionary, can comprehend questions encoded in Camlang, thereby validating that our method supports metalinguistic deductive learning. Second, we investigate whether LLMs can interpret the explicit grammatical rules and apply them to perform deductive reasoning in an unfamiliar language.

5.1 Experimental Setup

We conduct experiments on Camlang-CSQA-v0 in a zero-shot setting.

5.1.1 Baseline Systems and Implementation Details

We consider strong reasoning models as our baseline systems. For comparison, we also include a general-purpose model and human participant:

• •

GPT-4o (2024-11-20): A general-purpose LLM from OpenAI (openai2024gpt4o).

• •

GPT-o4-mini (2025-04-16): A lightweight, efficient reasoning LLM (openai2024gpt3o).

• •

GPT-o3 (2025-04-16): OpenAI’s most capable model in the o-series of reasoning models (openai2024gpt3o).

• •

GPT-5 (2025-08-07): The most recent LLMs from OpenAI, featuring substantial improvements in coding and reasoning (openai2025gpt5).

• •

DeepSeek-R1 (2025-05-28): An open-source reasoning LLM from DeepSeek, focusing on strong logical and reasoning capabilities (deepseekai2025deepseekr1).

• •

Human Participant: A first-year undergraduate student in Linguistics at Oxford.

For OpenAI models, we consider two configurations enabled by the API interface: (1) Tools-enabled: we upload the grammar book (.md) and the dictionary (.xlsx) as files and allow the model to call File Search and Code Interpreter, respectively, and; (2) Context-only: we prepend the full text of the grammar book and a textual export of the dictionary to the input as context, with tools disabled. For DeepSeek-R1, we use the context-only configuration as it does not support such tools.

For reasoning models (o3, o4-mini, and GPT-5), we set the reasoning effort to high. For GPT-4o, we set temperature and top-p to 0. For the other models, these two parameters are not exposed, and we therefore use the default configuration. We add prompts before the question to instruct the LLMs to explicitly make use of the grammar book and dictionary, and conclude its output with a sentence of the form “The final answer is X”, where $X$ is an option (e.g., “A” or “B”). The detailed prompts are shown in Appendix LABEL:sec:prompts. In addition to choosing an answer option, the human participant is asked to write explanations to demonstrate their choice, simulating the reasoning steps of LLM, which is later used for evaluation and analysis.

We also run the systems on the English version of our testing instances without the grammar book and dictionary to provide a controlled comparison. For human participant, to avoid leakage, we ask the annotator to complete the English questions after the Camlang questions.

5.2 Exact Match Results

5.2.1 Evaluation Method

Following standard practice in reasoning benchmarks, we report average accuracy using exact matching (EM). In this setting, a prediction is considered as correct if and only if the model’s output contains a final answer of the form “The final answer is X”, and the value $X$ exactly matches the gold option. If a response contains multiple such phrases, we take the last one as the final prediction.

5.2.2 Result Discussion

We first discuss the EM results as shown in Table 6. All observations in this subsection are based on the EM accuracy only, which should not be interpreted as direct evidence of metalinguistic deductive reasoning.

Performance Gap between Camlang and English.

All models achieve strong performance on the English version of CommonsenseQA (85–98% accuracy), confirming that the underlying commonsense reasoning task itself is not the bottleneck. The sharp performance drop on Camlang (21–47%) therefore isolates the difficulty to the metalinguistic reasoning from grammar and dictionary into meaning, which current LLMs fail to operationalise. Since Camlang requires first parsing questions through explicit grammar and dictionary lookup before reasoning about commonsense relation and knowledge, the drop in accuracy reflects models’ inability to perform metalinguistic reasoning in an unknown language.

Reasoning Models vs. General-Purpose Models.

Among LLMs, reasoning systems such as GPT-5 and GPT-o3 achieve the highest accuracies on Camlang (both 46.81% in the context-only setting), while DeepSeek-R1 reaches 40.43%. In contrast, GPT-4o performs near chance level (21.28%), only slightly above the random baseline of 16.67%. This suggests that reasoning-oriented models show greater robustness when faced with Camlang’s grammar-mediated encoding, though their performance remains far from human. This is consistent with that Camlang-CSQA indeed requires grammar rule processing before commonsense deduction. However, whether models actually follow this reasoning path is further examined through human verification in § 5.3.

Impact of Resource Access Methods.

Interestingly, we observe that tool use does not always improve the model performance. For both GPT-5 and GPT-o3, the context-only configuration outperforms the tools-enabled configuration (46.81 vs. 42.55 for GPT-5; 46.81 vs. 27.66 for GPT-o3). The longer inference times in tool mode do not correlate with higher accuracy, indicating a clear decoupling between time cost and successful rule use.

While CommonsenseQA questions are relatively short multiple-choice instances, the tool setting requires the systems to repeatedly query the grammar book or dictionary, which can add complexity and latency without improving final accuracy. Another salient factor is that tool-based lookup operates over dictionary stems, while Camlang is morphologically rich and produces many inflected surface forms that may not directly appear in the dictionary. In our setup, unless the model can lemmatise first based on the grammar book, direct tool queries may fail to retrieve relevant entries. By contrast, in the context-only setting, the full grammar and dictionary are embedded directly into the input, allowing the model to match morphologically related forms through context and thereby avoiding lookup failures. The result suggests that LLMs struggle to operationalise grammar book and dictionary lookup as part of their reasoning pipeline, even when the API explicitly provides such capabilities.

Human Performance and Learnability Evidence.

Compared to LLMs, the human participant achieves much higher accuracy on Camlang (87.23%), only slightly lower than their 91.49% on English (a drop of 4.26%). This sharp contrast shows that the difficulty for LLMs lies not in the commonsense reasoning component itself, but in the metalinguistic deductive reasoning of Camlang. Although solving Camlang questions takes considerably longer for humans (670.9s vs. 17.7s), they are able to systematically parse the grammar, consult the dictionary, and integrate the results into reasoning. These results are consistent with Camlang being learnable via explicit use of the grammar and dictionary as in § 3.2.3 (cross-annotator translation consistency in the English $\to$ Camlang direction).

5.3 Human Verification and Analysis on Model Outputs

Although reasoning LLMs such as GPT-5 and GPT-o3 achieve nearly 47% EM accuracy on Camlang, these numbers alone do not reveal whether correct answers result from genuine reasoning, or from alternative shortcuts such as exploiting surface regularities or relying on English priors. We therefore turn to human verification of reasoning traces to assess whether correct predictions indeed reflect rule mastery.

5.3.1 Evaluation Method

We further verify whether the intermediate reasoning steps of model outputs are correct when their final answers match the gold label. In particular, our analysis considers three aspects:

• •

Parsing of the Question: whether the model correctly and completely identified the syntactic structure of the input question (e.g., clause boundaries, argument roles) according to the grammar book.

• •

Understanding of the Question Meaning: whether the model accurately captured the intended semantic interpretation of the question.

• •

Understanding of answer Option Meaning: whether the model correctly interpreted the meanings of all answer options from the dictionary.

We explicitly separate parsing from meaning to distinguish errors in recognising formal structure (syntactic misparsing) from errors in semantic interpretation (misunderstanding word or sentence meaning). For instance, a model may arrive at the right meaning despite an incomplete parse by directly mapping key words from the dictionary, or conversely, may parse a structure correctly but still misinterpret semantic roles (see § 6, where we present case studies with annotations). We do not examine the parsing of options, since most options consist of a single Camlang word translated from one or two English words, leaving little structural ambiguity.

For each aspect, model outputs are categorised into four labels: (1) correct and complete (Crt+&Com+); (2) incorrect and complete (Crt-&Com+); (3) correct and incomplete (Crt+&Com-); and (4) incorrect and incomplete (Crt-&Com-). Each instance is first annotated by one annotator and then reviewed by another, and disagreements are subsequently resolved through discussion to reach a final consensus.121212Across 169 instances $times$ 3 aspects (507 annotation in total), only 9 required discussion.

To provide a better comparison, we report three human verification-based metrics by jointly evaluating all three aspects (question parsing, question meaning, and answer option understanding):

• •

Strict Human-Verified Accuracy (SHV ACC): an instance is correct only if it is labelled as Crt+&Com+ in all three aspects, reflecting full and faithful rule understanding and application.

• •

Moderate Human-Verified Accuracy (MHV ACC): an instance is counted as correct if the question parsing is Crt+&Com+, while question and option understanding may contain incomplete but not incorrect or incorrect and incomplete labels. This metric requires accurate parsing but tolerates minor omissions in semantic interpretation.

• •

Lenient Human-Verified Accuracy (LHV ACC): an instance is counted as correct if question and option understanding contain only incomplete (but not incorrect) labels, regardless of whether parsing is incomplete. This captures cases where reasoning succeeds through partial lexical mapping or shallow shortcuts without full grammatical parsing.

Note that SHV/MHV/LHV are accuracy-style metrics, defined as the number of instances meeting the respective criteria divided by all test instances (not just EM-correct ones). Such a tiered comparison allows us to distinguish genuine rule mastery (SHV), parsing-sensitive correctness (MHV), and shortcut-based reasoning (LHV).

5.3.2 Result Discussion

We report the human verified results in Table 7. Overall, we find a salient gap between humans and LLMs. Humans maintain strong performance across all verification criteria, confirming that Camlang is operationally learnable through explicit use of the grammar and dictionary. By contrast, LLMs achieve non-trivial EM accuracy scores but almost entirely fail under SHV and MHV, indicating that most correct predictions do not arise from metalinguistic deductive reasoning. These trends highlight two general conclusions: (1) EM substantially overestimates models’ reasoning ability, and; (2) Camlang effectively exposes the limits of LLMs in integrating explicit formal rules into their reasoning pipeline. We next discuss the results in detail.

Sharp decline from EM to SHV.

Across all LLMs, SHV accuracy is near to zero, even for GPT-5 (context), which achieves 46.81% EM. This sharp drop suggests that many EM-correct predictions are not supported by faithful grammatical reasoning, but more likely arise from shortcuts such as partial lexical matching or reliance on English priors. By contrast, humans reach 55.32% SHV, demonstrating that they can systematically parse questions, reconstruct meanings, and interpret options using explicit rules. The contrast reveals that EM alone substantially overstates LLM competence, while SHV directly tests whether Camlang’s grammar is truly mastered.

At the same time, we acknowledge that SHV is deliberately strict. LLMs and humans may have partially engaged with the grammar but omitted intermediate explanations they deemed unnecessary, in particular for reasoning LLMs, leading to incomplete labels. Thus, we further analyse MHV and LHV.

Partial Success and Shortcut Reasoning under MHV and LHV.

Under MHV, which tolerates incomplete semantics but requires correct parsing, only GPT-5 (context) reaches 19.15%, while others remain close to zero. This confirms that LLMs rarely manage accurate structural parsing. LHV further relaxes the criteria by including incomplete-but-not-incorrect interpretations. Here, GPT-5 (context) rises to 29.79% and GPT-5 (tool) to 17.02%, while o3 (context) reaches 10.64%. These gains suggest that stronger models sometimes align Camlang words with dictionary entries sufficiently to guess the right answer, even without full parsing. In other words, their success often comes from shallow alignment rather than genuine metalinguistic deduction.

Moreover, the divergence from EM to LHV makes clear that EM can substantially overestimate grammatical reasoning ability. In particular, EM remains non-trivial for LLMs such as DeepSeek-R1 and o4-mini, while LHV is near zero. This comparison indicates that, for those LLMs, many EM-correct cases arise from shallow heuristics rather than rule-guided parsing.

Human vs. LLMs.

Humans consistently outperform all LLMs by wide margins under every metric: 55.32% SHV, 59.57% MHV, and 68.09% LHV. The drop between EM and SHV indicates humans also resort to shortcuts, such as inferring from lexical cues or skipping minor structures. However, the high LHV, MHV, and SHV further support that humans can reliably parse the surface syntax and identify the intended hierarchical structure of Camlang sentences.

5.4 Error Analysis

We now show the fine-grained error analysis (Table 8), examining the distribution of correctness and completeness labels across question parsing, question meaning, and option meaning as evaluated in § 5.3. Table 8 shows the distribution statistics.

5.4.1 Analysis across Aspects

The distribution of verification labels reveals systematic asymmetries across dimensions.

Correct Parsing vs. Correct Meaning.

Across most LLMs (except o4-mini), parsing is disproportionately weaker than meaning. For example, DeepSeek-R1 shows 26.31% (21.05% Crt+&Com+ + 5.26% Crt+&Com-) of EM-correct cases as in parsing, while 31.58% in question meaning. This indicates a tendency to infer plausible semantics directly from lexical cues or commonsense priors without faithfully reconstructing syntactic structure, a form of “meaning without parsing”. In contrast, humans maintain high correctness in both parsing (82.93% Crt+&Com+ + 9.76% Crt+&Com- = 92.69%) and meaning (85.37 % Crt+&Com+ + 2.44 Crt+&Com- = 87.81%), confirming that they rely on explicit metalinguistic reasoning and interpretation rather than semantic guessing.

Question vs. Option Meaning.

Another consistent pattern is that some models perform better on interpreting options than on questions. For instance, o4-mini achieves 8.33% Crt+&Com+ and 75% Crt+&Com- in option meaning, indicating that despite missing interpretation, its understanding of options is mostly correct. This contrast suggests that understanding full questions is considerably harder than interpreting isolated options, because questions require reconstructing relationships among multiple words according to the grammar book, rather than mapping single words to dictionary entries. In other words, option meaning can often be resolved through shallow lexical lookup, whereas question interpretation demands faithful parsing of clause structures and argument roles. Humans, by contrast, show no such imbalance, with parsing and both meaning dimensions converging at high levels.

Complete vs. Incomplete Reasoning.

Strong reasoning models such as GPT-5 (context) reach relatively high proportions of Crt+&Com+ in parsing (45.45%) and question meaning (59.09%), yet collapse to 0 in option meaning. Instead, almost all option cases fall into Crt+&Com-, reflecting reasoning traces where partial lexical alignment is visible but the full interpretation is not explicitly documented. It suggests that models often omit parts of the reasoning they deem unnecessary, leaving incomplete chains in the written output.

5.4.2 Analysis across Systems

As a general-purpose LLM, both tool and context settings of GPT-4o fail almost completely, where all EM-correct cases fall into the Crt-&Com- category across parsing, question meaning, and option meaning. This indicates that GPT-4o shows almost no evidence of rule-based reasoning in its traces, and its EM-level success likely comes from random guessing or coincidental alignment with commonsense priors.

Amongst reasoning LLMs, both DS and o4-mini exhibit partial success via shortcuts rather than full parsing. For DS, option interpretation shows some strength: 11 out of 19 EM-correct cases contain at least partially correct option meanings, yet parsing is mostly incorrect (Crt-&Com+). This reflects a “meaning without parsing” behaviour: the model often guesses semantics from lexical cues and world knowledge, bypassing formal structure. In contrast, o4-mini’s parsing and question meaning remain poor, but 10 (1+9) out of 12 EM-correct cases contain at least partially correct option meanings (mostly Crt+&Com-), indicating reliance on shallow lexical shortcuts—matching familiar words without reconstructing the question’s syntax.

Across both settings, o3 produces partially correct outputs in all three aspects, but rarely aligns them into a complete reasoning step. In the context setting (#EM=22), question meaning is its relative strength (Crt+&Com+ 15/22 = 68.18%), while question parsing is more scattered (Crt+&Com+ 31.82%, Crt-&Com+ 36.36%), indicating that the model often delivers a complete but incorrect parse. Option meaning is fragmented (Crt+&Com+ 27.27%, Crt+&Com- 36.36%, Crt-&Com+ 36.36%). The tool setting (#EM=13) follows a similar pattern: question meaning is relatively strong (Crt+&Com+ 69.23%), but option meaning shows weaker completeness (Crt+&Com+ 15.38%, Crt+&Com- 38.46%).

GPT-5, particularly in the context setting (#EM=22), achieves the highest totals across parsing (Crt+&Com+ 45.45%, Crt+&Com- 22.73%) and question meaning (Crt+&Com+ 59.09%). However, option meaning is almost entirely incomplete (Crt+&Com- 20/22 = 90.91%, Crt+&Com+ 0%), showing that while the model can consistently map options to dictionary entries, it rarely produces a full, verifiable reasoning trace. This pattern suggests emerging metalinguistic awareness to a limited extent, where GPT-5 is often incomplete rather than incorrect, but still lacks the systematic, end-to-end grammatical mastery achieved by humans. The tool setting (#EM=20) performs weaker overall, with higher proportions of Crt-&Com- across all aspects (30% in parsing, 25% in question meaning, 35% in option meaning), reflecting failures in handling morphological variation and retrieval. Such evidence shows that GPT-5 has the strongest potential, but its reasoning remains fragmented, revealing only partial metalinguistic competence compared to humans’ systematic grammar-based reasoning.

Humans consistently rely on structured grammar-based reasoning, in sharp contrast to LLMs’ shortcut-based strategies. Over 80% of their EM-correct cases are fully correct and complete in parsing, question meaning, and option meaning.