AgriGPT-Omni: A Unified Speech-Vision-Text Framework for Multilingual Agricultural Intelligence

TL;DR

AgriGPT-Omni introduces a unified multilingual speech-vision-text framework for agriculture, creating a large dataset, a novel omni-model, and a comprehensive benchmark to advance multimodal agricultural AI applications.

Contribution

The paper presents the first agricultural omni-model trained on a large synthetic and real multilingual speech dataset, along with a new tri-modal benchmark for agriculture.

Findings

Outperforms general-purpose baselines in multilingual multimodal reasoning

Creates the largest agricultural speech dataset to date

Establishes a new tri-modal benchmark for agricultural AI

Abstract

Despite rapid advances in multimodal large language models, agricultural applications remain constrained by the lack of multilingual speech data, unified multimodal architectures, and comprehensive evaluation benchmarks. To address these challenges, we present AgriGPT-Omni, an agricultural omni-framework that integrates speech, vision, and text in a unified framework. First, we construct a scalable data synthesis and collection pipeline that converts agricultural texts and images into training data, resulting in the largest agricultural speech dataset to date, including 492K synthetic and 1.4K real speech samples across six languages. Second, based on this, we train the first agricultural omni-model via a three-stage paradigm: textual knowledge injection, progressive multimodal alignment, and GRPO-based reinforcement learning, enabling unified reasoning across languages and modalities. Third, we propose AgriBench-Omni-2K, the first tri-modal benchmark for agriculture, covering diverse speech-vision-text tasks and multilingual slices, with standardized protocols and reproducible tools. Experiments show that AgriGPT-Omni significantly outperforms general-purpose baselines on multilingual and multimodal reasoning as well as real-world speech understanding. All models, data, benchmarks, and code will be released to promote reproducible research, inclusive agricultural intelligence, and sustainable AI development for low-resource regions.