Semantic Caching and Intent-Driven Context Optimization for Multi-Agent Natural Language to Code Systems

TL;DR

This paper introduces a multi-agent system that efficiently translates natural language queries into Python code for data analytics, achieving high accuracy and cost savings through semantic caching, dynamic prompt assembly, and a dual-threshold decision mechanism.

Contribution

The system combines semantic caching, intent-driven prompt optimization, and a dual-threshold retrieval method, enabling scalable, accurate, and cost-effective natural language to code translation in production.

Findings

Cache hit rate of 67% on production queries

Reduced token consumption by 40-60%

Achieved 94.3% semantic accuracy in enterprise deployment

Abstract

We present a production-optimized multi-agent system designed to translate natural language queries into executable Python code for structured data analytics. Unlike systems that rely on expensive frontier models, our approach achieves high accuracy and cost efficiency through three key innovations: (1) a semantic caching system with LLM-based equivalence detection and structured adaptation hints that provides cache hit rates of 67% on production queries; (2) a dual-threshold decision mechanism that separates exact-match retrieval from reference-guided generation; and (3) an intent-driven dynamic prompt assembly system that reduces token consumption by 40-60% through table-aware context filtering. The system has been deployed in production for enterprise inventory management, processing over 10,000 queries with an average latency of 8.2 seconds and 94.3% semantic accuracy. We describe the architecture, present empirical results from production deployment, and discuss practical considerations for deploying LLM-based analytics systems at scale.