Programmatic and Direct Manipulation, Together at Last

TL;DR

Sketch-n-Sketch seamlessly combines programmatic and direct manipulation for SVG editing, enabling users to write code and directly modify outputs with real-time program updates, enhancing flexibility and usability.

Contribution

The paper introduces trace-based program synthesis and heuristics for live synchronization, integrating direct manipulation with programming in SVG editing.

Findings

Effective live synchronization between code and output.

User study indicates improved workflow and usability.

Supports complex SVG programs with real-time updates.

Abstract

Direct manipulation interfaces and programmatic systems have distinct and complementary strengths. The former provide intuitive, immediate visual feedback and enable rapid prototyping, whereas the latter enable complex, reusable abstractions. Unfortunately, existing systems typically force users into just one of these two interaction modes. We present a system called Sketch-n-Sketch that integrates programmatic and direct manipulation for the particular domain of Scalable Vector Graphics (SVG). In Sketch-n-Sketch, the user writes a program to generate an output SVG canvas. Then the user may directly manipulate the canvas while the system immediately infers a program update in order to match the changes to the output, a workflow we call live synchronization. To achieve this, we propose (i) a technique called trace-based program synthesis that takes program execution history into account in order to constrain the search space and (ii) heuristics for dealing with ambiguities. Based on our experience with examples spanning 2,000 lines of code and from the results of a preliminary user study, we believe that Sketch-n-Sketch provides a novel workflow that can augment traditional programming systems. Our approach may serve as the basis for live synchronization in other application domains, as well as a starting point for yet more ambitious ways of combining programmatic and direct manipulation.