Protecting Sensitive Tabular Data in Hybrid Clouds

TL;DR

This paper demonstrates how to securely perform large-scale data analytics on sensitive data stored in hybrid cloud environments using encryption techniques that ensure privacy, integrity, and efficiency without vendor lock-in.

Contribution

It introduces a method for running analytics on encrypted data in hybrid clouds using Apache Parquet Modular Encryption, with an innovation to reduce performance overhead from key management calls.

Findings

Secure analytics on encrypted data is feasible with minimal performance loss.

PME provides privacy, integrity, and granular access control in hybrid cloud settings.

The approach prevents vendor lock-in and maintains data security.

Abstract

Regulated industries, such as Healthcare and Finance, are starting to move parts of their data and workloads to the public cloud. However, they are still reluctant to trust the public cloud with their most sensitive records, and hence leave them in their premises, leveraging the hybrid cloud architecture. We address the security and performance challenges of big data analytics using a hybrid cloud in a real-life use case from a hospital. In this use case, the hospital collects sensitive patient data and wants to run analytics on it in order to lower antibiotics resistance, a significant challenge in healthcare. We show that it is possible to run large-scale analytics on data that is securely stored in the public cloud encrypted using Apache Parquet Modular Encryption (PME), without significant performance losses even if the secret encryption keys are stored on-premises. PME is a standard mechanism for data encryption and key management, not specific to any public cloud, and therefore helps prevent vendor lock-in. It also provides privacy and integrity guarantees, and enables granular access control to the data. We also present an innovation in PME for lowering the performance hit incurred by calls to the Key Management Service. Our solution therefore enables protecting large amounts of sensitive data in hybrid clouds and still allows to efficiently gain valuable insights from it.