Human in the Latent Loop (HILL): Interactively Guiding Model Training Through Human Intuition

TL;DR

HILL is an interactive framework that enables users to guide model training by reshaping latent space representations through human intuition, improving performance and offering insights, while also highlighting potential biases.

Contribution

We introduce HILL, a novel human-in-the-loop framework that incorporates human intuition into latent space manipulation during model training using a knowledge distillation-inspired approach.

Findings

Human-guided modifications improve model performance.

The approach maintains model generalization.

User biases can influence training outcomes.

Abstract

Latent space representations are critical for understanding and improving the behavior of machine learning models, yet they often remain obscure and intricate. Understanding and exploring the latent space has the potential to contribute valuable human intuition and expertise about respective domains. In this work, we present HILL, an interactive framework allowing users to incorporate human intuition into the model training by interactively reshaping latent space representations. The modifications are infused into the model training loop via a novel approach inspired by knowledge distillation, treating the user's modifications as a teacher to guide the model in reshaping its intrinsic latent representation. The process allows the model to converge more effectively and overcome inefficiencies, as well as provide beneficial insights to the user. We evaluated HILL in a user study tasking participants to train an optimal model, closely observing the employed strategies. The results demonstrated that human-guided latent space modifications enhance model performance while maintaining generalization, yet also revealing the risks of including user biases. Our work introduces a novel human-AI interaction paradigm that infuses human intuition into model training and critically examines the impact of human intervention on training strategies and potential biases.