Fine-Tuning and Evaluating Conversational AI for Agricultural Advisory

TL;DR

This paper introduces a hybrid LLM architecture for agricultural advisory that improves factual accuracy and safety by decoupling retrieval from response generation, with a new evaluation framework and open-source tools.

Contribution

It presents a novel hybrid model with separate retrieval and response layers, a new evaluation framework DG-EVAL, and demonstrates cost-effective fine-tuning for agricultural AI.

Findings

Fine-tuning on curated data improves fact recall and relevance.

A smaller fine-tuned model matches or exceeds larger models in factual accuracy.

The stitching layer enhances safety without sacrificing conversational quality.

Abstract

Large Language Models show promise for agricultural advisory, yet vanilla models exhibit unsupported recommendations, generic advice lacking specific, actionable detail, and communication styles misaligned with smallholder farmer needs. In high stakes agricultural contexts, where recommendation accuracy has direct consequences for farmer outcomes, these limitations pose challenges for responsible deployment. We present a hybrid LLM architecture that decouples factual retrieval from conversational delivery: supervised fine-tuning with LoRA on expert-curated GOLDEN FACTS (atomic, verified units of agricultural knowledge) optimizes fact recall, while a separate stitching layer transforms retrieved facts into culturally appropriate, safety-aware responses. Our evaluation framework, DG-EVAL, performs atomic fact verification (measuring recall, precision, and contradiction detection) against expert-curated ground truth rather than Wikipedia or retrieved documents. Experiments across multiple model configurations on crops and queries from Bihar, India show that fine-tuning on curated data substantially improves fact recall and F1, while maintaining high relevance. Using a fine-tuned smaller model achieves comparable or better factual quality at a fraction of the cost of frontier models. A stitching layer further improves safety subscores while maintaining high conversational quality. We release the farmerchat-prompts library to enable reproducible development of domain-specific agricultural AI.