Analysis and Visualization of Linguistic Structures in Large Language Models: Neural Representations of Verb-Particle Constructions in BERT

TL;DR

This paper explores how BERT's layers encode verb-particle constructions, revealing that middle layers best capture syntactic nuances and highlighting variability across verb categories, advancing understanding of neural language representations.

Contribution

It provides a detailed analysis of BERT's internal representations of verb-particle structures, demonstrating layer-specific encoding and variability across different verb categories.

Findings

Middle layers most effectively capture syntactic structures

Significant variability in representational accuracy across verb categories

Challenging the assumption of uniform neural processing of linguistic elements

Abstract

This study investigates the internal representations of verb-particle combinations within transformer-based large language models (LLMs), specifically examining how these models capture lexical and syntactic nuances at different neural network layers. Employing the BERT architecture, we analyse the representational efficacy of its layers for various verb-particle constructions such as 'agree on', 'come back', and 'give up'. Our methodology includes a detailed dataset preparation from the British National Corpus, followed by extensive model training and output analysis through techniques like multi-dimensional scaling (MDS) and generalized discrimination value (GDV) calculations. Results show that BERT's middle layers most effectively capture syntactic structures, with significant variability in representational accuracy across different verb categories. These findings challenge the conventional uniformity assumed in neural network processing of linguistic elements and suggest a complex interplay between network architecture and linguistic representation. Our research contributes to a better understanding of how deep learning models comprehend and process language, offering insights into the potential and limitations of current neural approaches to linguistic analysis. This study not only advances our knowledge in computational linguistics but also prompts further research into optimizing neural architectures for enhanced linguistic precision.