Syntax Representation in Word Embeddings and Neural Networks -- A Survey

TL;DR

This survey reviews how neural networks in NLP capture syntax implicitly, evaluating various models and representations across monolingual and multilingual tasks to understand their syntactic knowledge.

Contribution

It provides a comprehensive overview of methods for assessing syntactic information in neural network representations across different architectures and languages.

Findings

Neural networks encode syntax without explicit supervision.

Certain pre-trained models are more effective for syntactic transfer.

Multilingual models show varied syntactic encoding capabilities.

Abstract

Neural networks trained on natural language processing tasks capture syntax even though it is not provided as a supervision signal. This indicates that syntactic analysis is essential to the understating of language in artificial intelligence systems. This overview paper covers approaches of evaluating the amount of syntactic information included in the representations of words for different neural network architectures. We mainly summarize re-search on English monolingual data on language modeling tasks and multilingual data for neural machine translation systems and multilingual language models. We describe which pre-trained models and representations of language are best suited for transfer to syntactic tasks.