Antagonising explanation and revealing bias directly through sequencing and multimodal inference

TL;DR

This paper explores how deep generative models, especially diffusion models, reflect cultural biases through their reconstruction process, and proposes viewing their outputs as a temporal dialogue with the past, impacting future audiovisual synthesis.

Contribution

It introduces a novel perspective on generative models as a process of cultural and temporal reflection, emphasizing the importance of understanding bias and history in image and video synthesis.

Findings

Generative models encode cultural biases in their reconstruction process.

Viewing diffusion as a backward-in-time process reveals limitations in current synthesis methods.

Historical methodologies can inform and improve modern generative approaches.

Abstract

Deep generative models produce data according to a learned representation, e.g. diffusion models, through a process of approximation computing possible samples. Approximation can be understood as reconstruction and the large datasets used to train models as sets of records in which we represent the physical world with some data structure (photographs, audio recordings, manuscripts). During the process of reconstruction, e.g., image frames develop each timestep towards a textual input description. While moving forward in time, frame sets are shaped according to learned bias and their production, we argue here, can be considered as going back in time; not by inspiration on the backward diffusion process but acknowledging culture is specifically marked in the records. Futures of generative modelling, namely in film and audiovisual arts, can benefit by dealing with diffusion systems as a process to compute the future by inevitably being tied to the past, if acknowledging the records as to capture fields of view at a specific time, and to correlate with our own finite memory ideals. Models generating new data distributions can target video production as signal processors and by developing sequences through timelines we ourselves also go back to decade-old algorithmic and multi-track methodologies revealing the actual predictive failure of contemporary approaches to synthesis in moving image, both as relevant to composition and not explanatory.