Prediction-space knowledge markets for communication-efficient federated learning on multimedia tasks

TL;DR

This paper introduces KTA v2, a novel federated learning method that uses prediction-space knowledge trading to improve communication efficiency and accuracy on multimedia tasks, outperforming traditional methods.

Contribution

KTA v2 presents a new two-stage, prediction-space knowledge trading approach for federated learning that reduces communication costs and enhances model performance under data heterogeneity.

Findings

Outperforms FedAvg, FedProx, and FedMD in accuracy and communication efficiency.

Achieves 57.7% accuracy on CIFAR-10 with 1/1100 of FedAvg's communication.

Attains 89.3% accuracy on AG News with 1/300 of FedAvg's traffic.

Abstract

Federated learning (FL) enables collaborative training over distributed multimedia data but suffers acutely from statistical heterogeneity and communication constraints, especially when clients deploy large models. Classic parameter-averaging methods such as FedAvg transmit full model weights and can diverge under nonindependent and identically distributed (non-IID) data. We propose KTA v2, a prediction-space knowledge trading market for FL. Each round, clients locally train on their private data, then share only logits on a small public reference set. The server constructs a client-client similarity graph in prediction space, combines it with reference-set accuracy to form per-client teacher ensembles, and sends back personalized soft targets for a second-stage distillation update. This two-stage procedure can be interpreted as approximate block-coordinate descent on a unified objective with prediction-space regularization. Experiments on FEMNIST, CIFAR-10 and AG News show that, under comparable or much lower communication budgets, KTA v2 consistently outperforms a local-only baseline and strong parameter-based methods (FedAvg, FedProx), and substantially improves over a FedMD-style global teacher. On CIFAR-10 with ResNet-18, KTA v2 reaches 57.7% test accuracy using approximately 1/1100 of FedAvg's communication, while on AG News it attains 89.3% accuracy with approximately 1/300 of FedAvg's traffic.