DANIEL: A Distributed and Scalable Approach for Global Representation Learning with EHR Applications

TL;DR

This paper introduces DANIEL, a distributed, scalable, and privacy-preserving framework for learning global representations from large-scale EHR data using an optimized Ising model approach, improving clinical task performance.

Contribution

It develops a novel distributed bi-factored gradient descent method for scalable, privacy-preserving representation learning with Ising models on high-dimensional EHR data.

Findings

Superior performance in clinical tasks like phenotyping and clustering

Effective handling of high-dimensional, multi-institutional EHR data

Enhanced scalability and privacy preservation in representation learning

Abstract

Classical probabilistic graphical models face fundamental challenges in modern data environments, which are characterized by high dimensionality, source heterogeneity, and stringent data-sharing constraints. In this work, we revisit the Ising model, a well-established member of the Markov Random Field (MRF) family, and develop a distributed framework that enables scalable and privacy-preserving representation learning from large-scale binary data with inherent low-rank structure. Our approach optimizes a non-convex surrogate loss function via bi-factored gradient descent, offering substantial computational and communication advantages over conventional convex approaches. We evaluate our algorithm on multi-institutional electronic health record (EHR) datasets from 58,248 patients across the University of Pittsburgh Medical Center (UPMC) and Mass General Brigham (MGB), demonstrating superior performance in global representation learning and downstream clinical tasks, including relationship detection, patient phenotyping, and patient clustering. These results highlight a broader potential for statistical inference in federated, high-dimensional settings while addressing the practical challenges of data complexity and multi-institutional integration.