Subjective Camera 1.0: Bridging Human Cognition and Visual Reconstruction through Sequence-Aware Sketch-Guided Diffusion

TL;DR

This paper presents Subjective Camera 1.0, a novel framework that reconstructs real-world scenes from subjective descriptions and sketches, bridging human cognition and visual reconstruction without extensive training data.

Contribution

It introduces a sequence-aware diffusion-based method that effectively integrates textual and sketch inputs for scene reconstruction, addressing generalization and concept integration challenges.

Findings

Achieves state-of-the-art image quality and alignment

Outperforms existing methods in spatial and semantic accuracy

User studies favor the proposed approach

Abstract

We introduce the concept of a subjective camera to reconstruct meaningful moments that physical cameras fail to capture. We propose Subjective Camera 1.0, a framework for reconstructing real-world scenes from readily accessible subjective readouts, i.e., textual descriptions and progressively drawn rough sketches. Built on optimization-based alignment of diffusion models, our approach avoids large-scale paired training data and mitigates generalization issues. To address the challenge of integrating multiple abstract concepts in real-world scenarios, we design a Sequence-Aware Sketch-Guided Diffusion framework with three loss terms for concept-wise sequential optimization, following the natural order of subjective readouts. Experiments on two datasets demonstrate that our method achieves state-of-the-art performance in image quality as well as spatial and semantic alignment with target scenes. User studies with 40 participants further confirm that our approach is consistently preferred. Our project page is at: subjective-camera.github.io