AMOE: a Tool to Automatically Extract and Assess Organizational Evidence for Continuous Cloud Audit

TL;DR

This paper introduces AMOE, a tool leveraging NLP and QA techniques to automatically extract and assess organizational evidence from policy documents, aiding continuous cloud auditing and increasing transparency.

Contribution

The paper presents a novel NLP-based approach and prototype for automating the extraction and assessment of organizational evidence from textual policies in cloud auditing.

Findings

Prototype retrieves correct answers for over 50% of metrics

Automates auditing of textual policy documents

Reduces time for policy document review

Abstract

The recent spread of cloud services has enabled many companies to take advantage of them. Nevertheless, the main concern about the adoption of cloud services remains the lack of transparency perceived by customers regarding security and privacy. To overcome this issue, Cloud Service Certifications (CSCs) have emerged as an effective solution to increase the level of trust in cloud services, possibly based on continuous auditing to monitor and evaluate the security of cloud services on an ongoing basis. Continuous auditing can be easily implemented for technical aspects, while organizational aspects can be challenging due to their generic nature and varying policies between service providers. In this paper, we propose an approach to facilitate the automatic assessment of organizational evidence, such as that extracted from security policy documents. The evidence extraction process is based on Natural Language Processing (NLP) techniques, in particular on Question Answering (QA). The implemented prototype provides promising results on an annotated dataset, since it is capable to retrieve the correct answer for more than half of the tested metrics. This prototype can be helpful for Cloud Service Providers (CSPs) to automate the auditing of textual policy documents and to help in reducing the time required by auditors to check policy documents.