Learned Single-Pass Multitasking Perceptual Graphics for Immersive Displays

TL;DR

This paper introduces a lightweight, learned multitasking perceptual graphics model that performs multiple perceptual tasks in a single step using text prompts, suitable for immersive displays with limited resources.

Contribution

It presents a novel multitasking perceptual graphics model that uses text prompts for flexible, high-quality effects in a single inference, reducing resource use and management complexity.

Findings

Achieves high perceptual quality on desktop and embedded devices.

Supports varied perceptual effects through text-guided permutations.

Validated by user study confirming perceptual improvements.

Abstract

Emerging immersive display technologies efficiently utilize resources with perceptual graphics methods such as foveated rendering and denoising. Running multiple perceptual graphics methods challenges devices with limited power and computational resources. We propose a computationally-lightweight learned multitasking perceptual graphics model. Given RGB images and text-prompts, our model performs text-described perceptual tasks in a single inference step. Simply daisy-chaining multiple models or training dedicated models can lead to model management issues and exhaust computational resources. In contrast, our flexible method unlocks consistent high quality perceptual effects with reasonable compute, supporting various permutations at varied intensities using adjectives in text prompts (e.g. mildly, lightly). Text-guidance provides ease of use for dynamic requirements such as creative processes. To train our model, we propose a dataset containing source and perceptually enhanced images with corresponding text prompts. We evaluate our model on desktop and embedded platforms and validate perceptual quality through a user study.