Synthesizing and Adapting Error Correction Data for Mobile Large Language Model Applications

TL;DR

This paper presents a method to synthesize and adapt error correction data for mobile LLM applications, improving their performance through domain-specific data augmentation and reweighting based on live metrics.

Contribution

It introduces a scalable data synthesis pipeline using LLMs, with domain adaptation via reweighting based on live deployment metrics, enhancing mobile error correction models.

Findings

Synthetic data improves offline error correction performance.

Reweighting based on live metrics enhances real-world application results.

Best practices for combining synthetic and real data are identified.

Abstract

Error correction is an important capability when applying large language models (LLMs) to facilitate user typing on mobile devices. In this paper, we use LLMs to synthesize a high-quality dataset of error correction pairs to evaluate and improve LLMs for mobile applications. We first prompt LLMs with error correction domain knowledge to build a scalable and reliable addition to the existing data synthesis pipeline. We then adapt the synthetic data distribution to match the mobile application domain by reweighting the samples. The reweighting model is learnt by predicting (a handful of) live A/B test metrics when deploying LLMs in production, given the LLM performance on offline evaluation data and scores from a small privacy-preserving on-device language model. Finally, we present best practices for mixing our synthetic data with other data sources to improve model performance on error correction in both offline evaluation and production live A/B testing.