Integrating Structural Description of Data Format Information into Programming to Auto-generate File Reading Programs

TL;DR

This paper presents a novel method for automatically generating file reading programs by leveraging structured data format descriptions, significantly reducing manual effort in handling heterogeneous data formats.

Contribution

It introduces a new approach that uses Data Format Markup Language (DFML) to automatically generate file reading programs for various data formats, including binary and text files.

Findings

Effective automatic program generation demonstrated on binary and text files.

DFML-based approach reduces manual coding effort.

Tool DFML Editor facilitates editing and generating data format descriptions.

Abstract

File reading is the basis for data sharing and scientific computing. However, manual programming for file reading is labour-intensive and time-consuming, as data formats are heterogeneous and complex. To address such an issue, this study proposes a novel approach for the automatic generation of file reading programs based on structured and self-described data format information. This approach provides two modes composed of sequentially and randomly reading. The file data format is described by Data Format Markup Language and thus DFML documents are generated. The formation of data type sequences by parsing those DFML documents. The generation of programs for sequential or random reading data with formed data type sequences and general programing rules for specific programming languages. A tool named DFML Editor was developed for generating and editing DFML documents. Case studies on binary files, i.e., ESRI point shapefiles and plain text files, i.e., input files of Storm Water Management Model, were conducted with the software developed for automatic program generation and file reading. Experimental results show that the proposed approach is effective for automatically generating programs for reading files. The idea in this study is also helpful for automatically writing files.