Perceptual Kalman Filters: Online State Estimation under a Perfect Perceptual-Quality Constraint

TL;DR

This paper introduces a new class of perceptual Kalman filters designed for online state estimation that balances temporal perceptual quality with estimation accuracy, revealing fundamental trade-offs and novel filtering strategies.

Contribution

It develops a recursive formula for perceptual filters that incorporate perfect perceptual-quality constraints, extending classic Kalman filtering to perceptually-optimized online estimation.

Findings

Perceptual constraints can cause filters to ignore new data, increasing MSE.

Introduction of the unutilized information process as a novel analytical tool.

Demonstration of qualitative effects on video reconstruction under perceptual constraints.

Abstract

Many practical settings call for the reconstruction of temporal signals from corrupted or missing data. Classic examples include decoding, tracking, signal enhancement and denoising. Since the reconstructed signals are ultimately viewed by humans, it is desirable to achieve reconstructions that are pleasing to human perception. Mathematically, perfect perceptual-quality is achieved when the distribution of restored signals is the same as that of natural signals, a requirement which has been heavily researched in static estimation settings (i.e. when a whole signal is processed at once). Here, we study the problem of optimal causal filtering under a perfect perceptual-quality constraint, which is a task of fundamentally different nature. Specifically, we analyze a Gaussian Markov signal observed through a linear noisy transformation. In the absence of perceptual constraints, the Kalman filter is known to be optimal in the MSE sense for this setting. Here, we show that adding the perfect perceptual quality constraint (i.e. the requirement of temporal consistency), introduces a fundamental dilemma whereby the filter may have to "knowingly" ignore new information revealed by the observations in order to conform to its past decisions. This often comes at the cost of a significant increase in the MSE (beyond that encountered in static settings). Our analysis goes beyond the classic innovation process of the Kalman filter, and introduces the novel concept of an unutilized information process. Using this tool, we present a recursive formula for perceptual filters, and demonstrate the qualitative effects of perfect perceptual-quality estimation on a video reconstruction problem.