# Image classification with symbolic hints using limited resources

## TL;DR

This paper explores using text metadata as hints to improve image classification with minimal computational resources.

## Contribution

A late fusion method is proposed to incorporate symbolic hints into image classification with negligible computational cost.

## Key findings

- Late fusion of image and text models performs similarly to mid-level fusion using SVMs.
- Calibration of pre-trained models is essential for effective fusion performance.
- The method is demonstrated on real-world image classification tasks with text metadata as hints.

## Abstract

Typical machine learning classification benchmark problems often ignore the full input data structures present in real-world classification problems. Here we aim to represent additional information as “hints” for classification. We show that under a specific realistic conditional independence assumption, the hint information can be included by late fusion. In two experiments involving image classification with hints taking the form of text metadata, we demonstrate the feasibility and performance of the fusion scheme. We fuse the output of pre-trained image classifiers with the output of pre-trained text models. We show that calibration of the pre-trained models is crucial for the performance of the fused model. We compare the performance of the fusion scheme with a mid-level fusion scheme based on support vector machines and find that these two methods tend to perform quite similarly, albeit the late fusion scheme has only negligible computational costs.