Vision-Language Semantic Grounding for Multi-Domain Crop-Weed Segmentation

TL;DR

This paper introduces VL-WS, a vision-language grounded segmentation framework that generalizes across diverse agricultural environments by leveraging semantic alignment and domain-invariant features, significantly improving weed segmentation accuracy.

Contribution

The novel VL-WS framework combines CLIP embeddings with spatial features using FiLM layers, enabling cross-domain generalization and label-efficient weed segmentation in precision agriculture.

Findings

Achieves 91.64% mean Dice score on benchmark datasets.

Outperforms CNN baseline by 4.98% in Dice score.

Improves weed class segmentation, reaching 80.45% Dice score.

Abstract

Fine-grained crop-weed segmentation is essential for enabling targeted herbicide application in precision agriculture. However, existing deep learning models struggle to generalize across heterogeneous agricultural environments due to reliance on dataset-specific visual features. We propose Vision-Language Weed Segmentation (VL-WS), a novel framework that addresses this limitation by grounding pixel-level segmentation in semantically aligned, domain-invariant representations. Our architecture employs a dual-encoder design, where frozen Contrastive Language-Image Pretraining (CLIP) embeddings and task-specific spatial features are fused and modulated via Feature-wise Linear Modulation (FiLM) layers conditioned on natural language captions. This design enables image level textual descriptions to guide channel-wise feature refinement while preserving fine-grained spatial localization. Unlike prior works restricted to training and evaluation on single-source datasets, VL-WS is trained on a unified corpus that includes close-range ground imagery (robotic platforms) and high-altitude UAV imagery, covering diverse crop types, weed species, growth stages, and sensing conditions. Experimental results across four benchmark datasets demonstrate the effectiveness of our framework, with VL-WS achieving a mean Dice score of 91.64% and outperforming the CNN baseline by 4.98%. The largest gains occur on the most challenging weed class, where VL-WS attains 80.45% Dice score compared to 65.03% for the best baseline, representing a 15.42% improvement. VL-WS further maintains stable weed segmentation performance under limited target-domain supervision, indicating improved generalization and data efficiency. These findings highlight the potential of vision-language alignment to enable scalable, label-efficient segmentation models deployable across diverse real-world agricultural domains.