Weakly Supervised Instance Attention for Multisource Fine-Grained Object Recognition with an Application to Tree Species Classification

TL;DR

This paper introduces a weakly supervised deep instance attention model for multisource fine-grained object recognition, demonstrating improved accuracy in tree species classification by fusing spectral, spatial, and structural data at multiple levels.

Contribution

It proposes a novel weakly supervised multi-source attention framework with four fusion levels, enhancing fine-grained classification accuracy over state-of-the-art methods.

Findings

Feature-level fusion achieved 53% accuracy in tree species classification.

All fusion levels outperformed existing methods.

Increased model capacity further improved accuracy by 6.3%.

Abstract

Multisource image analysis that leverages complementary spectral, spatial, and structural information benefits fine-grained object recognition that aims to classify an object into one of many similar subcategories. However, for multisource tasks that involve relatively small objects, even the smallest registration errors can introduce high uncertainty in the classification process. We approach this problem from a weakly supervised learning perspective in which the input images correspond to larger neighborhoods around the expected object locations where an object with a given class label is present in the neighborhood without any knowledge of its exact location. The proposed method uses a single-source deep instance attention model with parallel branches for joint localization and classification of objects, and extends this model into a multisource setting where a reference source that is assumed to have no location uncertainty is used to aid the fusion of multiple sources in four different levels: probability level, logit level, feature level, and pixel level. We show that all levels of fusion provide higher accuracies compared to the state-of-the-art, with the best performing method of feature-level fusion resulting in 53% accuracy for the recognition of 40 different types of trees, corresponding to an improvement of 5.7% over the best performing baseline when RGB, multispectral, and LiDAR data are used. We also provide an in-depth comparison by evaluating each model at various parameter complexity settings, where the increased model capacity results in a further improvement of 6.3% over the default capacity setting.