Do We Really Need GNNs with Explicit Structural Modeling? MLPs Suffice for Language Model Representations

TL;DR

This paper questions the necessity of GNNs for structural modeling in NLP, demonstrating that MLPs can effectively encode linguistic information and serve as scalable alternatives, challenging the presumed superiority of GNNs.

Contribution

The study introduces a modular probing framework to isolate and evaluate the roles of message-passing and feature-transformation in GNNs, revealing MLPs' effectiveness in structure-aware tasks.

Findings

MLPs improve linguistic knowledge in LM representations.

Models with feature-transformation enhance syntactic and semantic encoding.

Message-passing alone often underperforms or harms probing results.

Abstract

Explicit structural information has been proven to be encoded by Graph Neural Networks (GNNs), serving as auxiliary knowledge to enhance model capabilities and improve performance in downstream NLP tasks. However, recent studies indicate that GNNs fail to fully utilize structural information, whereas Multi-Layer Perceptrons (MLPs), despite lacking the message-passing mechanisms inherent to GNNs, exhibit a surprising ability in structure-aware tasks. Motivated by these findings, this paper introduces a comprehensive probing framework from an information-theoretic perspective. The framework is designed to systematically assess the role of explicit structural modeling in enhancing language model (LM) representations and to investigate the potential of MLPs as efficient and scalable alternatives to GNNs. We extend traditional probing classifiers by incorporating a control module that allows for selective use of either the full GNN model or its decoupled components, specifically, the message-passing and feature-transformation operations.This modular approach isolates and assesses the individual contributions of these operations, avoiding confounding effects from the complete GNN architecture. Using the Edge Probing Suite, a diagnostic tool for evaluating the linguistic knowledge encoded in LMs, we find that MLPs, when used as feature-transformation modules, consistently improve the linguistic knowledge captured in LM representations across different architectures. They effectively encode both syntactic and semantic patterns. Similarly, GNNs that incorporate feature-transformation operations show beneficial effects. In contrast, models that rely solely on message-passing operations tend to underperform, often leading to negative impacts on probing task performance.