Sketch2code: Generating a website from a paper mockup

TL;DR

This paper introduces two automated methods for converting website wireframe sketches into code, leveraging classical vision techniques and deep learning, with the deep learning approach showing superior performance.

Contribution

It presents a novel dataset and evaluation framework for wireframe-to-code conversion, demonstrating the effectiveness of deep semantic segmentation networks over classical methods.

Findings

Deep learning approach outperforms classical vision techniques

A new dataset of website wireframes is released

An evaluation framework using synthetic sketches is proposed

Abstract

An early stage of developing user-facing applications is creating a wireframe to layout the interface. Once a wireframe has been created it is given to a developer to implement in code. Developing boiler plate user interface code is time consuming work but still requires an experienced developer. In this dissertation we present two approaches which automates this process, one using classical computer vision techniques, and another using a novel application of deep semantic segmentation networks. We release a dataset of websites which can be used to train and evaluate these approaches. Further, we have designed a novel evaluation framework which allows empirical evaluation by creating synthetic sketches. Our evaluation illustrates that our deep learning approach outperforms our classical computer vision approach and we conclude that deep learning is the most promising direction for future research.