Semantic Labeling for Third-Party Cybersecurity Risk Assessment: A Semi-Supervised Approach to Intent-Aware Question Retrieval

TL;DR

This paper introduces a semi-supervised, intent-aware question retrieval method for cybersecurity risk assessment that improves efficiency and alignment with assessment scope by using semantic labels instead of direct question similarity.

Contribution

It proposes a novel approach that combines label discovery, large-scale label assignment, and label propagation to enhance question retrieval in cybersecurity assessments.

Findings

Reduces labeling cost and runtime compared to LLM-based annotation.

Achieves better alignment with cybersecurity control domains.

Maintains label quality while improving retrieval efficiency.

Abstract

Third-Party Risk Assessment (TPRA) relies on large repositories of cybersecurity compliance questions used to assess external suppliers against standards such as ISO/IEC 27001 and NIST. In practice, not all questions are relevant for a specific supplier and selecting questions for a given assessment context remains a manual and time-consuming task. Existing question retrieval approaches based on lexical or semantic similarity can identify topically related questions, but they often fail to capture the underlying assessment intent, including control domain and evaluation scope. To address this limitation, we investigate whether an explicit semantic label space can improve intent-aware TPRA question selection. In particular, we separate label space discovery from large-scale label assignment. We start by discovering overlapping clusters of semantically similar questions and then exploit LLMs to assign unique labels for each cluster. Second, we propagate labels through k-nearest neighbors (kNN) for a larger-scale question annotation. Question retrieval is finally achieved by similarity measure of the query with respect to the extracted labels instead of the questions themselves. This reduces repeated LLM calls while preserving label consistency. Experimental results show that the proposed semi-supervised framework reduces labeling cost and runtime compared with per-question LLM annotation while maintaining label quality and improving efficiency. Furthermore, label-based retrieval achieves better alignment with cybersecurity control domains and assessment scope than similarity-based retrieval, highlighting the value of semantic labels as an intermediate representation.