Cloud K-SVD: A Collaborative Dictionary Learning Algorithm for Big, Distributed Data

TL;DR

This paper introduces cloud K-SVD, a distributed algorithm for collaborative dictionary learning that enables multiple sites to learn a shared geometric data structure without sharing raw data, suitable for privacy-sensitive and large-scale applications.

Contribution

It proposes a novel distributed dictionary learning algorithm focused on union of subspaces, with theoretical analysis and practical validation on real and synthetic data.

Findings

Effective in learning shared dictionaries across distributed sites

Maintains data privacy by avoiding raw data exchange

Performs well on real and synthetic datasets

Abstract

This paper studies the problem of data-adaptive representations for big, distributed data. It is assumed that a number of geographically-distributed, interconnected sites have massive local data and they are interested in collaboratively learning a low-dimensional geometric structure underlying these data. In contrast to previous works on subspace-based data representations, this paper focuses on the geometric structure of a union of subspaces (UoS). In this regard, it proposes a distributed algorithm---termed cloud K-SVD---for collaborative learning of a UoS structure underlying distributed data of interest. The goal of cloud K-SVD is to learn a common overcomplete dictionary at each individual site such that every sample in the distributed data can be represented through a small number of atoms of the learned dictionary. Cloud K-SVD accomplishes this goal without requiring exchange of individual samples between sites. This makes it suitable for applications where sharing of raw data is discouraged due to either privacy concerns or large volumes of data. This paper also provides an analysis of cloud K-SVD that gives insights into its properties as well as deviations of the dictionaries learned at individual sites from a centralized solution in terms of different measures of local/global data and topology of interconnections. Finally, the paper numerically illustrates the efficacy of cloud K-SVD on real and synthetic distributed data.