Linting Style and Substance in READMEs

TL;DR

This paper introduces LintMe, a flexible linting tool for READMEs that combines programmatic checks with LLM-based content evaluation to improve style and substance tailored to diverse user needs.

Contribution

It presents a novel approach to README linting using a lightweight DSL and LLM integration, enabling context-specific checks that address style and content issues.

Findings

Approachable and flexible linting tool for READMEs

Effective in diverse contexts and user needs

Demonstrated through user study and extensibility case study

Abstract

READMEs shape first impressions of software projects, yet what constitutes a good README varies across audiences and contexts. Research software needs reproducibility details, while open-source libraries might prioritize quick-start guides. Through a design probe, LintMe, we explore how linting can be used to improve READMEs given these diverse contexts, aiding style and content issues while preserving authorial agency. Users create context-specific checks using a lightweight DSL that uses a novel combination of programmatic operations (e.g., for broken links) with LLM-based content evaluation (e.g., for detecting jargon), yielding checks that would be challenging for prior linters. Through a user study (N=11), comparison with naive LLM usage, and an extensibility case study, we find that our design is approachable, flexible, and well matched with the needs of this domain. This work opens the door for linting more complex documentation and other culturally mediated text-based documents.