Visual Editing with LLM-based Tool Chaining: An Efficient Distillation Approach for Real-Time Applications

TL;DR

This paper introduces a distillation method to fine-tune smaller LLMs for real-time visual editing tasks, enabling efficient tool invocation based on natural language requests with reduced cost and latency.

Contribution

A novel distillation approach that fine-tunes smaller LLMs to match larger models' performance for real-time visual editing applications.

Findings

Student models match teacher model performance

Significant reduction in cost and latency

Improved fine-tuning with data augmentation

Abstract

We present a practical distillation approach to fine-tune LLMs for invoking tools in real-time applications. We focus on visual editing tasks; specifically, we modify images and videos by interpreting user stylistic requests, specified in natural language ("golden hour"), using an LLM to select the appropriate tools and their parameters to achieve the desired visual effect. We found that proprietary LLMs such as GPT-3.5-Turbo show potential in this task, but their high cost and latency make them unsuitable for real-time applications. In our approach, we fine-tune a (smaller) student LLM with guidance from a (larger) teacher LLM and behavioral signals. We introduce offline metrics to evaluate student LLMs. Both online and offline experiments show that our student models manage to match the performance of our teacher model (GPT-3.5-Turbo), significantly reducing costs and latency. Lastly, we show that fine-tuning was improved by 25% in low-data regimes using augmentation.