Virtual Parameter Sharpening: Dynamic Low-Rank Perturbations for Inference-Time Reasoning Enhancement

TL;DR

Virtual Parameter Sharpening (VPS) is a novel inference-time method that dynamically augments transformer layers with activation-conditioned low-rank perturbations, enabling test-time adaptation and reasoning enhancement without parameter updates.

Contribution

VPS introduces a dynamic, activation-conditioned low-rank perturbation technique for inference-time reasoning enhancement in transformers, differing from static fine-tuning methods like LoRA.

Findings

VPS effectively improves reasoning capabilities in large language models.

Theoretical analysis reveals spectral properties of the perturbations.

Adaptive policies modulate perturbation strength based on activation energy.

Abstract

I introduce Virtual Parameter Sharpening (VPS), an inference-time technique that augments frozen transformer linear layers with dynamic, activation-conditioned low-rank perturbations. Unlike parameter-efficient fine-tuning methods such as LoRA, which learn static low-rank adapters, VPS constructs its perturbation factors on the fly from batch activation statistics and optional gradient signals, enabling test-time adaptation without persistent parameter updates. The perturbation takes the form Delta W = gamma * W^T V U^T W, where selector matrices U and V are constructed via sparse activation-guided selection or Sylvester-coupled regression. We provide a theoretical analysis of the perturbation's spectral properties and describe an adaptive policy system that modulates perturbation magnitude based on activation energy and token-level entropy. This system incorporates multi-objective verification with iterative refinement for tasks with ground-truth supervision. We present the complete algorithmic framework, analyze its mathematical foundations, and discuss the mechanisms by which activation-conditioned computation may enhance reasoning capabilities in large language models. Implementation and experimental code are available at https://github.com/Saba-Kublashvili/vps-virtual-parameter-synthesis .