History-Aware Cross-Attention Reinforcement: Self-Supervised Multi Turn and Chain-of-Thought Fine-Tuning with vLLM

TL;DR

This paper introduces CAGSR-vLLM-MTC, a framework that enhances large language models with self-supervised reinforcement learning for multi-turn dialogues and reasoning, leveraging attention signals during generation.

Contribution

It extends the CAGSR framework to vLLM, enabling asynchronous attention capture and self-supervised training for complex multi-turn and chain-of-thought tasks.

Findings

Effective attention signal accumulation over conversations

Improved multi-turn dialogue reasoning capabilities

Practical mechanisms to prevent attention collapse

Abstract

We present CAGSR-vLLM-MTC, an extension of our Self-Supervised Cross-Attention-Guided Reinforcement (CAGSR) framework, now implemented on the high-performance vLLM runtime, to address both multi-turn dialogue and chain-of-thought reasoning. Building upon our original single-turn approach, we first instrumented vLLM's C++/CUDA kernels to asynchronously capture per-layer, per-head cross-attention weights during generation. We then generalized our self-supervised reward function to accumulate attention signals over entire conversation histories and intermediate chain-of-thought steps. We discuss practical trade-offs, including an entropy-based clamping mechanism to prevent attention collapse on early context, and outline future directions for multi-party dialogues and hierarchical reasoning.