InFiConD: Interactive No-code Fine-tuning with Concept-based Knowledge Distillation

TL;DR

InFiConD introduces an interactive, no-code framework that uses visual concepts for knowledge distillation, making model fine-tuning accessible to non-experts through visualization and user interaction.

Contribution

The paper presents a novel visual concept-based knowledge distillation pipeline and an interactive interface for no-code fine-tuning of models.

Findings

Users can effectively create and analyze student models.

The approach enables efficient knowledge transfer understanding.

InFiConD facilitates accessible model fine-tuning for domain-specific tasks.

Abstract

The emergence of large-scale pre-trained models has heightened their application in various downstream tasks, yet deployment is a challenge in environments with limited computational resources. Knowledge distillation has emerged as a solution in such scenarios, whereby knowledge from large teacher models is transferred into smaller student' models, but this is a non-trivial process that traditionally requires technical expertise in AI/ML. To address these challenges, this paper presents InFiConD, a novel framework that leverages visual concepts to implement the knowledge distillation process and enable subsequent no-code fine-tuning of student models. We develop a novel knowledge distillation pipeline based on extracting text-aligned visual concepts from a concept corpus using multimodal models, and construct highly interpretable linear student models based on visual concepts that mimic a teacher model in a response-based manner. InFiConD's interface allows users to interactively fine-tune the student model by manipulating concept influences directly in the user interface. We validate InFiConD via a robust usage scenario and user study. Our findings indicate that InFiConD's human-in-the-loop and visualization-driven approach enables users to effectively create and analyze student models, understand how knowledge is transferred, and efficiently perform fine-tuning operations. We discuss how this work highlights the potential of interactive and visual methods in making knowledge distillation and subsequent no-code fine-tuning more accessible and adaptable to a wider range of users with domain-specific demands.