ImageSI: Semantic Interaction for Deep Learning Image Projections

TL;DR

ImageSI introduces a novel semantic interaction method for image dimension reduction that updates underlying image embeddings directly based on user feedback, improving task relevance in visualizations.

Contribution

It proposes a new approach to incorporate user feedback directly into image embeddings, unlike existing methods that modify feature weights, enhancing the relevance of image projections.

Findings

ImageSI effectively captures user-specified relationships in image projections.

The method improves the organization of images based on user interactions.

Comparison shows ImageSI outperforms existing feature-weighting approaches.

Abstract

Semantic interaction (SI) in Dimension Reduction (DR) of images allows users to incorporate feedback through direct manipulation of the 2D positions of images. Through interaction, users specify a set of pairwise relationships that the DR should aim to capture. Existing methods for images incorporate feedback into the DR through feature weights on abstract embedding features. However, if the original embedding features do not suitably capture the users' task then the DR cannot either. We propose ImageSI, an SI method for image DR that incorporates user feedback directly into the image model to update the underlying embeddings, rather than weighting them. In doing so, ImageSI ensures that the embeddings suitably capture the features necessary for the task so that the DR can subsequently organize images using those features. We present two variations of ImageSI using different loss functions - ImageSI_MDS_Inverse, which prioritizes the explicit pairwise relationships from the interaction and ImageSI_Triplet, which prioritizes clustering, using the interaction to define groups of images. Finally, we present a usage scenario and a simulation based evaluation to demonstrate the utility of ImageSI and compare it to current methods.