HyperEdit: Unlocking Instruction-based Text Editing in LLMs via Hypernetworks

TL;DR

HyperEdit introduces hypernetworks and difference-aware regularization to improve instruction-based text editing in LLMs, achieving significant accuracy gains with minimal over-editing.

Contribution

It presents a novel hypernetwork-based approach with regularization techniques to enhance fidelity and precision in text editing tasks for LLMs.

Findings

Achieves 9%-30% BLEU improvement over baselines.

Uses only 3B parameters for effective editing.

Reduces over-editing while maintaining edit accuracy.

Abstract

Instruction-based text editing is increasingly critical for real-world applications such as code editors (e.g., Cursor), but Large Language Models (LLMs) continue to struggle with this task. Unlike free-form generation, editing requires faithfully implementing user instructions while preserving unchanged content, as even minor unintended modifications can break functionality. Existing approaches treat editing as generic text generation, leading to two key failures: they struggle to faithfully align edits with diverse user intents, and they often over-edit unchanged regions. We propose HyperEdit to address both issues. First, we introduce hypernetwork-based dynamic adaptation that generates request-specific parameters, enabling the model to tailor its editing strategy to each instruction. Second, we develop difference-aware regularization that focuses supervision on modified spans, preventing over-editing while ensuring precise, minimal changes. HyperEdit achieves a 9%--30% relative improvement in BLEU on modified regions over state-of-the-art baselines, despite utilizing only 3B parameters.