Uncovering Hidden Subspaces in Video Diffusion Models Using Re-Identification

TL;DR

This paper investigates the limitations of latent video diffusion models in healthcare, revealing they only learn a small fraction of training data and proposing methods to evaluate and improve their fidelity and privacy.

Contribution

It introduces a novel approach using re-identification models trained in latent space to assess the subspace coverage of synthetic videos and improve downstream task performance.

Findings

Only up to 30.8% of training videos are learned in latent models

Latent space training is more efficient and generalizes better

Proposed methods help evaluate and enhance synthetic video fidelity

Abstract

Latent Video Diffusion Models can easily deceive casual observers and domain experts alike thanks to the produced image quality and temporal consistency. Beyond entertainment, this creates opportunities around safe data sharing of fully synthetic datasets, which are crucial in healthcare, as well as other domains relying on sensitive personal information. However, privacy concerns with this approach have not fully been addressed yet, and models trained on synthetic data for specific downstream tasks still perform worse than those trained on real data. This discrepancy may be partly due to the sampling space being a subspace of the training videos, effectively reducing the training data size for downstream models. Additionally, the reduced temporal consistency when generating long videos could be a contributing factor. In this paper, we first show that training privacy-preserving models in latent space is computationally more efficient and generalize better. Furthermore, to investigate downstream degradation factors, we propose to use a re-identification model, previously employed as a privacy preservation filter. We demonstrate that it is sufficient to train this model on the latent space of the video generator. Subsequently, we use these models to evaluate the subspace covered by synthetic video datasets and thus introduce a new way to measure the faithfulness of generative machine learning models. We focus on a specific application in healthcare echocardiography to illustrate the effectiveness of our novel methods. Our findings indicate that only up to 30.8% of the training videos are learned in latent video diffusion models, which could explain the lack of performance when training downstream tasks on synthetic data.