Learning from 25 years of the extensible N-Dimensional Data Format

TL;DR

This paper reviews 25 years of development and application of the N-Dimensional Data Format (NDF), highlighting its role in astronomy data processing and the lessons learned from its hierarchical data model.

Contribution

It provides a historical overview and insights into the design, implementation, and long-term use of the NDF in astronomical data systems.

Findings

NDF has been integral to astronomy data pipelines for over two decades.

Hierarchical data models like NDF facilitate flexible data management.

Lessons learned inform future data format designs in astronomy.

Abstract

The extensible N-Dimensional Data Format (NDF) was designed and developed in the late 1980s to provide a data model suitable for use in a variety of astronomy data processing applications supported by the UK Starlink Project. Starlink applications were used extensively, primarily in the UK astronomical community, and form the basis of a number of advanced data reduction pipelines today. This paper provides an overview of the historical drivers for the development of NDF and the lessons learned from using a defined hierarchical data model for many years in data reduction software, data pipelines and in data acquisition systems.