Sketch2FullStack: Generating Skeleton Code of Full Stack Website and Application from Sketch using Deep Learning and Computer Vision

TL;DR

This paper introduces a deep learning-based method to automatically generate skeleton full-stack web and app code from sketch images, streamlining the development process and reducing resource consumption.

Contribution

It presents a novel multi-part approach that detects UI elements, creates database tables, and generates class files from sketches, which is a new application of deep learning in code generation.

Findings

Successfully detects UI elements from sketches

Generates database schemas from schema designs

Creates class files from class diagrams

Abstract

For a full-stack web or app development, it requires a software firm or more specifically a team of experienced developers to contribute a large portion of their time and resources to design the website and then convert it to code. As a result, the efficiency of the development team is significantly reduced when it comes to converting UI wireframes and database schemas into an actual working system. It would save valuable resources and fasten the overall workflow if the clients or developers can automate this process of converting the pre-made full-stack website design to get a partially working if not fully working code. In this paper, we present a novel approach of generating the skeleton code from sketched images using Deep Learning and Computer Vision approaches. The dataset for training are first-hand sketched images of low fidelity wireframes, database schemas and class diagrams. The approach consists of three parts. First, the front-end or UI elements detection and extraction from custom-made UI wireframes. Second, individual database table creation from schema designs and lastly, creating a class file from class diagrams.