CryptoQA: A Large-scale Question-answering Dataset for AI-assisted Cryptography

TL;DR

CryptoQA introduces a large-scale dataset for evaluating and training language models in cryptography, revealing current limitations and guiding improvements in AI-assisted cryptographic reasoning.

Contribution

The paper presents CryptoQA, the first extensive cryptography-specific QA dataset, and benchmarks LLMs, highlighting their deficiencies and potential for fine-tuning in cryptographic tasks.

Findings

LLMs show significant performance gaps in cryptography tasks

Fine-tuning improves LLMs' cryptographic reasoning abilities

CryptoQA enables targeted evaluation and training of LLMs for cryptography

Abstract

Large language models (LLMs) excel at many general-purpose natural language processing tasks. However, their ability to perform deep reasoning and mathematical analysis, particularly for complex tasks as required in cryptography, remains poorly understood, largely due to the lack of suitable data for evaluation and training. To address this gap, we present CryptoQA, the first large-scale question-answering (QA) dataset specifically designed for cryptography. CryptoQA contains over two million QA pairs drawn from curated academic sources, along with contextual metadata that can be used to test the cryptographic capabilities of LLMs and to train new LLMs on cryptographic tasks. We benchmark 15 state-of-the-art LLMs on CryptoQA, evaluating their factual accuracy, mathematical reasoning, consistency, referencing, backward reasoning, and robustness to adversarial samples. In addition to quantitative metrics, we provide expert reviews that qualitatively assess model outputs and establish a gold-standard baseline. Our results reveal significant performance deficits of LLMs, particularly on tasks that require formal reasoning and precise mathematical knowledge. This shows the urgent need for LLM assistants tailored to cryptography research and development. We demonstrate that, by using CryptoQA, LLMs can be fine-tuned to exhibit better performance on cryptographic tasks.