CAKE: Cloud Architecture Knowledge Evaluation of Large Language Models

TL;DR

The paper introduces CAKE, a comprehensive benchmark to evaluate large language models' understanding of cloud-native software architecture across multiple cognitive levels and topics.

Contribution

It presents a new benchmark with 188 expert-validated questions, evaluating 22 models and analyzing how different formats and augmentations affect model performance.

Findings

MCQ accuracy plateaus above 3B parameters at 99.2%.

Free-response scores increase steadily across cognitive levels.

Different formats reveal different aspects of model knowledge.

Abstract

In today's software architecture, large language models (LLMs) serve as software architecture co-pilots. However, no benchmark currently exists to evaluate large language models' actual understanding of cloud-native software architecture. For this reason we present a benchmark called CAKE, which consists of 188 expert-validated questions covering four cognitive levels of Bloom's revised taxonomy -- recall, analyze, design, and implement -- and five cloud-native topics. Evaluation is conducted on 22 model configurations (0.5B--70B parameters) across four LLM families, using three-run majority voting for multiple-choice questions (MCQs) and LLM-as-a-judge scoring for free-responses (FR). Based on this evaluation, four notable findings were identified. First, MCQ accuracy plateaus above 3B parameters, with the best model reaching 99.2\%. Second, free-response scores scale steadily across all cognitive levels. Third, the two formats capture different facets of knowledge, as the MCQ accuracy approaches a ceiling while free-responses continue to differentiate models. Finally, reasoning augmentation (+think) improves free-response quality, while tool augmentation (+tool) degrades performance for small models. These results suggest that the evaluation format fundamentally shapes how we measure architectural knowledge in LLMs.