Merlin: Deterministic Byte-Exact Deduplication for Lossless Context Optimization in Large Language Model Inference

TL;DR

Merlin is a high-throughput, byte-exact deduplication engine that significantly reduces redundant text data in large language model workflows without data loss.

Contribution

It introduces Merlin, a novel SIMD-friendly deduplication system with a secure architecture for efficient large-scale text processing in LLM applications.

Findings

Input reduction of 13.9% to over 71% in redundant datasets

Achieves processing speeds up to 8.7 GB/s

Maintains absolute data fidelity during deduplication

Abstract

Data-intensive applications, ranging from large-scale retrieval systems to advanced data pipelines, are increasingly bottlenecked by the processing of highly redundant text corpora. We present Merlin, a local-first, agnostic, high-throughput deduplication and context optimization engine designed to mitigate these inefficiencies. Utilizing a highly optimized, SIMD-friendly open-addressing flat hash set combined with xxHash3-64, Merlin performs rapid, byte-exact deduplication of text passages and data chunks. While broadly applicable to any text-processing workflow, its impact is particularly pronounced in Large Language Model (LLM) ecosystems, such as Retrieval-Augmented Generation (RAG). Our empirical evaluations demonstrate an input reduction ranging from 13.9% in low-redundancy datasets to over 71% in high-redundancy pipelines, maintaining absolute data fidelity. Furthermore, we detail the system's integration architecture via the Model Context Protocol (MCP), enabling secure, zero-network-interception deployment across major IDEs and autonomous agents. This paper outlines the core algorithmic design, performance benchmarks, and the architectural principles required to process data at sustained speeds of up to 8.7 GB/s.