MROSS: Multi-Round Region-based Optimization for Scene Sketching

TL;DR

This paper introduces MROSS, a multi-round region-based optimization method for scene sketching that uses semantic and feature losses to generate coherent, high-quality sketches from complex scenes.

Contribution

It proposes a novel multi-round optimization framework with region-specific processing and new loss functions guided by CLIP and VGG to improve scene sketching quality.

Findings

Effective in generating high-quality scene sketches

Outperforms existing methods in sketch realism and coherence

Demonstrates robustness across diverse scene types

Abstract

Scene sketching is to convert a scene into a simplified, abstract representation that captures the essential elements and composition of the original scene. It requires a semantic understanding of the scene and consideration of different regions within the scene. Since scenes often contain diverse visual information across various regions, such as foreground objects, background elements, and spatial divisions, dealing with these different regions poses unique difficulties. In this paper, we define a sketch as some sets of B\'ezier curves because of their smooth and versatile characteristics. We optimize different regions of input scene in multiple rounds. In each optimization round, the strokes sampled from the next region can seamlessly be integrated into the sketch generated in the previous optimization round. We propose an additional stroke initialization method to ensure the integrity of the scene and the convergence of optimization. A novel CLIP-based Semantic Loss and a VGG-based Feature Loss are utilized to guide our multi-round optimization. Extensive experimental results on the quality and quantity of the generated sketches confirm the effectiveness of our method.