Enhanced Remote Astronomical Archive System Based on the File-Level Unlimited Sliding-Window Technique
Cong-Ming Shi, Hui Deng, Feng Wang, Ying Mei, Shao-Guang Guo, Chen, Yang, Chen Wu, Shou-Lin Wei, Andreas Wicenec

TL;DR
This paper introduces an enhanced remote data archiving system for astronomy that uses a novel file-level flow-control technique, significantly improving transfer speed and data integrity for large-scale radio telescope data.
Contribution
The paper presents the Unlimited Sliding-Window (USW) flow-control method and its implementation in the ENGAS system, boosting remote data transfer performance for astronomical observations.
Findings
ENGAS is 3-12 times faster than NGAS.
ENGAS fully utilizes network bandwidth.
The system ensures data transfer integrity.
Abstract
Data archiving is one of the most critical issues for modern astronomical observations. With the development of a new generation of radio telescopes, the transfer and archiving of massive remote data have become urgent problems to be solved. Herein, we present a practical and robust file-level flow-control approach, called the Unlimited Sliding-Window (USW), by referring to the classic flow-control method in TCP protocol. Basing on the USW and the Next Generation Archive System (NGAS) developed for the Murchison Widefield Array telescope, we further implemented an enhanced archive system (ENGAS) using ZeroMQ middleware. The ENGAS substantially improves the transfer performance and ensures the integrity of transferred files. In the tests, the ENGAS is approximately three to twelve times faster than the NGAS and can fully utilize the bandwidth of network links. Thus, for archiving radio…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
