Intrinsic Explainability of Multimodal Learning for Crop Yield Prediction

TL;DR

This paper explores the intrinsic explainability of Transformer models in multimodal crop yield prediction, demonstrating their superior performance and providing insights into feature and modality contributions using novel attribution methods.

Contribution

It introduces the use of Transformer-based models for interpretable multimodal crop yield prediction and proposes the Weighted Modality Activation method for modality attribution.

Findings

Transformers outperform CNNs and RNNs in yield prediction accuracy.

Attention Rollout provides more reliable temporal attributions.

Modality attributions vary across methods and are interpretable with agronomic knowledge.

Abstract

Multimodal learning enables various machine learning tasks to benefit from diverse data sources, effectively mimicking the interplay of different factors in real-world applications, particularly in agriculture. While the heterogeneous nature of involved data modalities may necessitate the design of complex architectures, the model interpretability is often overlooked. In this study, we leverage the intrinsic explainability of Transformer-based models to explain multimodal learning networks, focusing on the task of crop yield prediction at the subfield level. The large datasets used cover various crops, regions, and years, and include four different input modalities: multispectral satellite and weather time series, terrain elevation maps and soil properties. Based on the self-attention mechanism, we estimate feature attributions using two methods, namely the Attention Rollout (AR) and Generic Attention (GA), and evaluate their performance against Shapley-based model-agnostic estimations, Shapley Value Sampling (SVS). Additionally, we propose the Weighted Modality Activation (WMA) method to assess modality attributions and compare it with SVS attributions. Our findings indicate that Transformer-based models outperform other architectures, specifically convolutional and recurrent networks, achieving R2 scores that are higher by 0.10 and 0.04 at the subfield and field levels, respectively. AR is shown to provide more robust and reliable temporal attributions, as confirmed through qualitative and quantitative evaluation, compared to GA and SVS values. Information about crop phenology stages was leveraged to interpret the explanation results in the light of established agronomic knowledge. Furthermore, modality attributions revealed varying patterns across the two methods compared.[...]