Federated Inference: Toward Privacy-Preserving Collaborative and Incentivized Model Serving

TL;DR

This paper introduces Federated Inference (FI), a new paradigm for privacy-preserving collaboration among models at inference time, analyzing its core principles, trade-offs, and system behaviors to enable scalable, secure, and effective collaborative inference.

Contribution

It formalizes FI as a distinct collaborative paradigm, analyzes its design trade-offs, and provides empirical insights into its practical challenges and system-level behaviors.

Findings

FI exhibits unique system behaviors not inherited from training-time federation.

Privacy constraints and non-IID data create specific friction points in collaborative inference.

Empirical analysis highlights challenges in privacy, collaboration, and incentive alignment.

Abstract

Federated Inference (FI) studies how independently trained and privately owned models can collaborate at inference time without sharing data or model parameters. While recent work has explored secure and distributed inference from disparate perspectives, a unified abstraction and system-level understanding of FI remain lacking. This paper positions FI as a distinct collaborative paradigm, complementary to federated learning, and identifies two fundamental requirements that govern its feasibility: inference-time privacy preservation and meaningful performance gains through collaboration. We formalize FI as a protected collaborative computation, analyze its core design dimensions, and examine the structural trade-offs that arise when privacy constraints, non-IID data, and limited observability are jointly imposed at inference time. Through a concrete instantiation and empirical analysis, we highlight recurring friction points in privacy-preserving inference, ensemble-based collaboration, and incentive alignment. Our findings suggest that FI exhibits system-level behaviors that cannot be directly inherited from training-time federation or classical ensemble methods. Overall, this work provides a unifying perspective on FI and outlines open challenges that must be addressed to enable practical, scalable, and privacy-preserving collaborative inference systems.