NVST data archiving system based on fastbit nosql database
Yingbo Liu, Feng Wang, Kaifan Ji, Hui Deng, Wei Dai, Bo Liang

TL;DR
This paper presents a new data archiving system for the NVST using Fastbit NoSQL database, significantly improving query performance for large-scale solar observation data.
Contribution
The study introduces a Fastbit-based archiving system that outperforms traditional relational databases in indexing and querying large astronomical datasets.
Findings
Fastbit database is 15 times faster in multi-field queries for 40 million records.
The system effectively handles over 20 million FITS files since 2012.
The approach enhances data management for large-scale astronomical observations.
Abstract
The New Vacuum Solar Telescope (NVST) is a 1-meter vacuum solar telescope that aims to observe the fine structures of active regions on the Sun. The main tasks of the NVST are high resolution imaging and spectral observations, including the measurements of the solar magnetic field. The NVST has been collecting more than 20 million FITS files since it began routine observations in 2012 and produces a maximum observational records of 120 thousand files in a day. Given the large amount of files, the effective archiving and retrieval of files becomes a critical and urgent problem. In this study, we implement a new data archiving system for the NVST based on the Fastbit Not Only Structured Query Language (NoSQL) database. Comparing to the relational database (i.e., MySQL; My Structured Query Language), the Fastbit database manifests distinctive advantages on indexing and querying…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Data Storage Technologies · Distributed and Parallel Computing Systems · Parallel Computing and Optimization Techniques
