SoilNet: A Multimodal Multitask Model for Hierarchical Classification of Soil Horizons

TL;DR

SoilNet is a multimodal, hierarchical classification model that accurately predicts soil horizons by integrating image data and metadata, matching or surpassing expert performance, and offering transparency in complex soil analysis tasks.

Contribution

This work introduces SoilNet, a structured, transparent multimodal multitask model specifically designed for hierarchical soil horizon classification, addressing challenges of complex label structures and data imbalance.

Findings

SoilNet achieves accuracy comparable to or better than human experts.

The model effectively handles complex hierarchical and imbalanced label structures.

User studies confirm the model's practical utility and reliability.

Abstract

Recent advances in artificial intelligence (AI), in particular foundation models, have improved the state of the art in many application domains including geosciences. Some specific problems, however, could not benefit from this progress yet. Soil horizon classification, for instance, remains challenging because of its multimodal and multitask characteristics and a complex hierarchically structured label taxonomy. Accurate classification of soil horizons is crucial for monitoring soil condition. In this work, we propose \textit{SoilNet} - a multimodal multitask model to tackle this problem through a structured modularized pipeline. In contrast to omnipurpose AI foundation models, our approach is designed to be inherently transparent by following the task structure human experts developed for solving this challenging annotation task. The proposed approach integrates image data and geotemporal metadata to first predict depth markers, segmenting the soil profile into horizon candidates. Each segment is characterized by a set of horizon-specific morphological features. Finally, horizon labels are predicted based on the multimodal concatenated feature vector, leveraging a graph-based label representation to account for the complex hierarchical relationships among soil horizons. Our method is designed to address complex hierarchical classification, where the number of possible labels is very large, imbalanced and non-trivially structured. We demonstrate the effectiveness of our approach on a real-world soil profile dataset and a comprehensive user study with domain experts. Our empirical evaluations demonstrate that SoilNet reliably predicts soil horizons that are plausible and accurate. User study results indicate that SoilNet achieves predictive performance on par with or better than that of human experts. All code can be found at: https://github.com/calgo-lab/BGR/