Efficient OpAmp Adaptation for Zoom Attention to Golden Contexts

TL;DR

This paper introduces an OpAmp-inspired adapter mechanism for Transformer models that improves focus on relevant context in noisy data, leading to superior question-answering performance without extensive retraining.

Contribution

We propose an OpAmp-based adapter for Transformers that enhances relevance focus in noisy contexts, outperforming existing models in retrieval-augmented tasks.

Findings

Qwen2.5-OpAmp-72B surpasses state-of-the-art models

Improves focus on golden context in noisy data

Achieves better QA performance without costly retraining

Abstract

Large language models (LLMs) have shown significant promise in question-answering (QA) tasks, particularly in retrieval-augmented generation (RAG) scenarios and long-context applications. However, their performance is hindered by noisy reference documents, which often distract from essential information. Despite fine-tuning efforts, Transformer-based architectures struggle to prioritize relevant content. This is evidenced by their tendency to allocate disproportionate attention to irrelevant or later-positioned documents. Recent work proposes the differential attention mechanism to address this issue, but this mechanism is limited by an unsuitable common-mode rejection ratio (CMRR) and high computational costs. Inspired by the operational amplifier (OpAmp), we propose the OpAmp adaptation to address these challenges, which is implemented with adapters efficiently. By integrating the adapter into pre-trained Transformer blocks, our approach enhances focus on the golden context without costly training from scratch. Empirical evaluations on noisy-context benchmarks reveal that our Qwen2.5-OpAmp-72B model, trained with our OpAmp adaptation, surpasses the performance of state-of-the-art LLMs, including DeepSeek-V3 and GPT-4o.