"Superluminal" FITS File Processing on Multiprocessors: Zero Time Endian Conversion Technique

TL;DR

This paper introduces a 'just-in-time endian conversion' technique for FITS file processing that significantly reduces conversion time, enabling faster handling of large astronomical datasets on multiprocessor systems.

Contribution

The paper presents a novel 'just-in-time endian conversion' method that improves FITS processing speed by delaying conversion until necessary, enhancing performance on modern CPU architectures.

Findings

20% speed increase for single-threaded processing

40% speed increase for multi-threaded processing

Efficiency gain related to CPU instruction pipeline optimization

Abstract

The FITS is the standard file format in astronomy, and it has been extended to agree with astronomical needs of the day. However, astronomical datasets have been inflating year by year. In case of ALMA telescope, a ~ TB scale 4-dimensional data cube may be produced for one target. Considering that typical Internet bandwidth is a few 10 MB/s at most, the original data cubes in FITS format are hosted on a VO server, and the region which a user is interested in should be cut out and transferred to the user (Eguchi et al., 2012). The system will equip a very high-speed disk array to process a TB scale data cube in a few 10 seconds, and disk I/O speed, endian conversion and data processing one will be comparable. Hence to reduce the endian conversion time is one of issues to realize our system. In this paper, I introduce a technique named "just-in-time endian conversion", which delays the endian conversion for each pixel just before it is really needed, to sweep out the endian conversion time; by applying this method, the FITS processing speed increases 20% for single threading, and 40% for multi-threading compared to CFITSIO. The speed-up by the method tightly relates to modern CPU architecture to improve the efficiency of instruction pipelines due to break of "causality", a programmed instruction code sequence.