Automatically Generating Codes from Graphical Screenshots Based on Deep Autocoder

TL;DR

PixCoder is a deep learning model that automatically generates platform-specific GUI code from screenshots with over 95% accuracy by using an attention-guided neural network to predict style sheets.

Contribution

This paper introduces PixCoder, a novel deep learning approach employing supervised attention to improve GUI code generation from images, achieving high accuracy.

Findings

GUI code accuracy exceeds 95%

Attention mechanism improves code generation quality

Effective platform-specific GUI code synthesis

Abstract

During software front-end development, the work to convert Graphical User Interface(GUI) image to the corresponding front-end code is an inevitable tedious work. There have been some attempts to make this work to be automatic. However, the GUI code generated by these models is not accurate due to the lack of attention mechanism guidance. To solve this problem, we propose PixCoder based on an artificially supervised attention mechanism. The approach is to train a neural network to predict the style sheets in the input GUI image and then output a vector. PixCoder generate the GUI code targeting specific platform according to the output vector. The experimental results have shown the accuracy of the GUI code generated by PixCoder is over 95%.