Federated In-Context Learning: Iterative Refinement for Improved Answer Quality

TL;DR

This paper introduces Fed-ICL, a federated framework that iteratively refines question-answering responses through multi-round client-server interactions, improving answer quality without transmitting model parameters.

Contribution

Fed-ICL is a novel federated in-context learning approach that enhances answer quality via iterative refinement, reducing communication overhead and leveraging local data.

Findings

Achieves strong QA performance on benchmarks

Maintains low communication costs

Provides theoretical convergence guarantees

Abstract

For question-answering (QA) tasks, in-context learning (ICL) enables language models to generate responses without modifying their parameters by leveraging examples provided in the input. However, the effectiveness of ICL heavily depends on the availability of high-quality examples, which are often scarce due to data privacy constraints, annotation costs, and distribution disparities. A natural solution is to utilize examples stored on client devices, but existing approaches either require transmitting model parameters - incurring significant communication overhead - or fail to fully exploit local datasets, limiting their effectiveness. To address these challenges, we propose Federated In-Context Learning (Fed-ICL), a general framework that enhances ICL through an iterative, collaborative process. Fed-ICL progressively refines responses by leveraging multi-round interactions between clients and a central server, improving answer quality without the need to transmit model parameters. We establish theoretical guarantees for the convergence of Fed-ICL and conduct extensive experiments on standard QA benchmarks, demonstrating that our proposed approach achieves strong performance while maintaining low communication costs.