Retrieval-augmented reasoning with lean language models

TL;DR

This paper presents a lightweight retrieval-augmented reasoning system that combines domain-specific fine-tuning with synthetic data and document compression, achieving near-frontier performance in resource-constrained environments.

Contribution

It introduces a novel lean language model architecture integrating retrieval and reasoning, optimized for privacy and deployment in limited-resource settings.

Findings

Significant improvement in answer accuracy over non-reasoning models

Approaches frontier-level performance with domain-specific fine-tuning

Effective use of synthetic data and document compression techniques

Abstract

This technical report details a novel approach to combining reasoning and retrieval augmented generation (RAG) within a single, lean language model architecture. While existing RAG systems typically rely on large-scale models and external APIs, our work addresses the increasing demand for performant and privacy-preserving solutions deployable in resource-constrained or secure environments. Building on recent developments in test-time scaling and small-scale reasoning models, we develop a retrieval augmented conversational agent capable of interpreting complex, domain-specific queries using a lightweight backbone model. Our system integrates a dense retriever with fine-tuned Qwen2.5-Instruct models, using synthetic query generation and reasoning traces derived from frontier models (e.g., DeepSeek-R1) over a curated corpus, in this case, the NHS A-to-Z condition pages. We explore the impact of summarisation-based document compression, synthetic data design, and reasoning-aware fine-tuning on model performance. Evaluation against both non-reasoning and general-purpose lean models demonstrates that our domain-specific fine-tuning approach yields substantial gains in answer accuracy and consistency, approaching frontier-level performance while remaining feasible for local deployment. All implementation details and code are publicly released to support reproducibility and adaptation across domains.