Improving LLM Final Representations with Inter-Layer Geometry

TL;DR

This paper introduces Cayley-Encoder, a novel, efficient graph-based method for leveraging intermediate LLM layers, significantly improving prediction accuracy across multiple tasks while maintaining low complexity.

Contribution

It proposes Cayley-Encoder, a mathematically grounded, sparse graph approach that enhances inter-layer communication in LLMs, outperforming existing methods with minimal additional parameters.

Findings

Cayley-Encoder outperforms baselines on 13 tasks and 9 LLMs.

Achieves up to 40 percentage points accuracy improvement.

Operates effectively in few-shot regimes and surpasses LoRA fine-tuning.

Abstract

The standard in LLM-based prediction is to use the final-layer representation as the input to a downstream predictor. However, intermediate layers may encode complementary task-relevant signals. Existing approaches therefore either search for the best layer for each task or apply expensive attention-based mechanisms to learn inter-layer aggregation. In this work, we first show that such complexity is unnecessary: a lightweight Graph Neural Network over a fully connected graph of LLM layers is more efficient and achieves significantly stronger predictive performance than existing approaches. We then introduce the Cayley-Encoder, which further improves both efficiency and predictive performance by replacing the fully connected graph with a Cayley graph over SL(2, Zn). These Cayley graphs provide a mathematically grounded topology that is sparse, regular by construction, and has low diameter. This enables effective communication across layers while constraining the aggregation structure and reducing the risk of GNN overfitting. In an evaluation of Cayley-Encoder across 13 tasks and 9 LLMs, Cayley-Encoder consistently outperforms baselines, achieving improvements of up to 40 percentage points in accuracy, while introducing at most 0.1% additional parameters relative to the LLM size. We further show that Cayley-Encoder is effective in few-shot regimes. Finally, we show that Cayley-Encoder outperforms LoRA fine-tuning while operating on the frozen LLM. We conclude with an explainability analysis showing that multiple layers contribute meaningfully to the final prediction, supporting our hypothesis.