Which Programming Language and What Features at Pre-training Stage Affect Downstream Logical Inference Performance?

TL;DR

This study investigates how pre-training on programming languages versus natural languages influences large language models' ability to perform logical inference, highlighting the importance of language features and syntax structure.

Contribution

It systematically compares the effects of different programming languages and syntax features on logical reasoning performance in LLMs, revealing key factors that enhance inference abilities.

Findings

Models trained on programming languages outperform those trained on natural languages in logical tasks.

Programming language models better follow instructions than natural language models.

Syntax tree depth influences the logical reasoning performance of models.

Abstract

Recent large language models (LLMs) have demonstrated remarkable generalization abilities in mathematics and logical reasoning tasks. Prior research indicates that LLMs pre-trained with programming language data exhibit high mathematical and reasoning abilities; however, this causal relationship has not been rigorously tested. Our research aims to verify which programming languages and features during pre-training affect logical inference performance. Specifically, we pre-trained decoder-based language models from scratch using datasets from ten programming languages (e.g., Python, C, Java) and three natural language datasets (Wikipedia, Fineweb, C4) under identical conditions. Thereafter, we evaluated the trained models in a few-shot in-context learning setting on logical reasoning tasks: FLD and bAbi, which do not require commonsense or world knowledge. The results demonstrate that nearly all models trained with programming languages consistently outperform those trained with natural languages, indicating that programming languages contain factors that elicit logic inference performance. In addition, we found that models trained with programming languages exhibit a better ability to follow instructions compared to those trained with natural languages. Further analysis reveals that the depth of Abstract Syntax Trees representing parsed results of programs also affects logical reasoning performance. These findings will offer insights into the essential elements of pre-training for acquiring the foundational abilities of LLMs.