# An Exploratory Study on the Usage and Readability of Messages Within Assertion Methods of Test Cases

## TL;DR

This study investigates how assertion messages in Java unit tests are used and their readability, revealing that developers rarely include messages and that message complexity affects understandability based on English proficiency.

## Contribution

It provides the first empirical analysis of assertion message usage and readability in open-source Java tests, highlighting common patterns and challenges.

## Key findings

- Assertion messages are rarely used by developers.
- Messages are mostly string literals, identifiers, or both.
- Readability varies with message content and developer English proficiency.

## Abstract

Unit testing is a vital part of the software development process and involves developers writing code to verify or assert production code. Furthermore, to help comprehend the test case and troubleshoot issues, developers have the option to provide a message that explains the reason for the assertion failure. In this exploratory empirical study, we examine the characteristics of assertion messages contained in the test methods in 20 open-source Java systems. Our findings show that while developers rarely utilize the option of supplying a message, those who do, either compose it of only string literals, identifiers, or a combination of both types. Using standard English readability measuring techniques, we observe that a beginner's knowledge of English is required to understand messages containing only identifiers, while a 4th-grade education level is required to understand messages composed of string literals. We also discuss shortcomings with using such readability measuring techniques and common anti-patterns in assert message construction. We envision our results incorporated into code quality tools that appraise the understandability of assertion messages.