Assessment of Pre-Trained Models Across Languages and Grammars

TL;DR

This paper evaluates how multilingual large language models learn syntax across different languages and grammatical structures, revealing insights into their encoding of syntactic information and the impact of pretraining data.

Contribution

It introduces a sequence labeling approach for parsing and systematically assesses multiple LLMs on diverse syntactic treebanks across languages.

Findings

Framework is consistent across encodings

Pre-trained vectors do not favor constituency over dependency syntax

Sub-word tokenization is essential for syntax representation

Abstract

We present an approach for assessing how multilingual large language models (LLMs) learn syntax in terms of multi-formalism syntactic structures. We aim to recover constituent and dependency structures by casting parsing as sequence labeling. To do so, we select a few LLMs and study them on 13 diverse UD treebanks for dependency parsing and 10 treebanks for constituent parsing. Our results show that: (i) the framework is consistent across encodings, (ii) pre-trained word vectors do not favor constituency representations of syntax over dependencies, (iii) sub-word tokenization is needed to represent syntax, in contrast to character-based models, and (iv) occurrence of a language in the pretraining data is more important than the amount of task data when recovering syntax from the word vectors.