Beyond Standard Datacubes: Extracting Features from Irregular and Branching Earth System Data

TL;DR

This paper introduces a generalized tree-based data hypercube model for efficiently representing, indexing, and extracting features from complex, irregular Earth science datasets, enabling scalable and flexible data access.

Contribution

It presents a novel compressed tree structure for data hypercubes and integrates it into a feature extraction system within the Polytope framework, enhancing support for complex data spaces.

Findings

Efficient indexing of large, irregular datasets using compressed tree hypercubes.

Improved data access and feature extraction performance on complex Earth science data.

Demonstrated scalability and flexibility of the approach in practical evaluations.

Abstract

Earth science datasets are growing rapidly in both volume and structural complexity. They increasingly contain richly labelled data with heterogeneous metadata and complex internal constraints that impose dependencies between variables and dimensions. Datacubes have become a common abstraction for organising such datasets, but traditional dense and orthogonal datacube models struggle to represent irregular, sparse or branching data spaces efficiently. In this paper, we introduce a generalised data hypercube representation based on compressed tree structures, which enables an accurate and compact description of complex data spaces. We describe the design of this representation and analyse its ability to capture sparsity and conditional relationships while remaining efficient to traverse. Using a concrete implementation, we study the performance characteristics of compressed tree data hypercubes and demonstrate their effectiveness as fast, cache-like indices over large backend data stores. Building on this representation, we present an integrated feature extraction system that operates directly on tree-based data hypercubes within the Polytope framework. By embedding data access strategies into the data hypercube abstraction itself, the system enables precise, sub-field data extraction and supports flexible, user-driven access patterns. We evaluate the performance of the integrated system and show how it enables new ways of interacting with complex datasets that are difficult to support using traditional access models. This work bridges the gap between expressive data hypercube models and efficient data access methods. In particular, it provides a unified framework that combines tree-based data representations with feature extraction capabilities. The proposed approach therefore offers a foundation for scalable and user-centric access to large heterogeneous Earth science datasets.