DuetML: Human-LLM Collaborative Machine Learning Framework for Non-Expert Users

TL;DR

DuetML is a collaborative framework that leverages multimodal LLMs to assist non-expert users in customizing machine learning models, making ML development more accessible and aligned with user needs.

Contribution

It introduces DuetML, a novel human-LLM collaborative framework that integrates multimodal LLMs as interactive agents to facilitate non-expert ML customization.

Findings

Non-expert users can better define training data with DuetML.

DuetML does not increase cognitive load for users.

Users engage more deeply in ML task formulation.

Abstract

Machine learning (ML) models have significantly impacted various domains in our everyday lives. While large language models (LLMs) offer intuitive interfaces and versatility, task-specific ML models remain valuable for their efficiency and focused performance in specialized tasks. However, developing these models requires technical expertise, making it particularly challenging for non-expert users to customize them for their unique needs. Although interactive machine learning (IML) aims to democratize ML development through user-friendly interfaces, users struggle to translate their requirements into appropriate ML tasks. We propose human-LLM collaborative ML as a new paradigm bridging human-driven IML and machine-driven LLM approaches. To realize this vision, we introduce DuetML, a framework that integrates multimodal LLMs (MLLMs) as interactive agents collaborating with users throughout the ML process. Our system carefully balances MLLM capabilities with user agency by implementing both reactive and proactive interactions between users and MLLM agents. Through a comparative user study, we demonstrate that DuetML enables non-expert users to define training data that better aligns with target tasks without increasing cognitive load, while offering opportunities for deeper engagement with ML task formulation.