Contrastive Learning for API Aspect Analysis

TL;DR

This paper introduces CLAA, a contrastive learning-based transformer model for API aspect detection, demonstrating significant performance improvements and positive developer impact in analyzing API reviews.

Contribution

The paper proposes a novel contrastive learning approach for API aspect detection that outperforms existing models and enhances developer decision-making.

Findings

CLAA achieves 92% accuracy on API review classification.

Contrastive learning improves transformer performance in aspect detection.

Developer study shows increased confidence with CLAA.

Abstract

We present a novel approach - CLAA - for API aspect detection in API reviews that utilizes transformer models trained with a supervised contrastive loss objective function. We evaluate CLAA using performance and impact analysis. For performance analysis, we utilized a benchmark dataset on developer discussions collected from Stack Overflow and compare the results to those obtained using state-of-the-art transformer models. Our experiments show that contrastive learning can significantly improve the performance of transformer models in detecting aspects such as Performance, Security, Usability, and Documentation. For impact analysis, we performed empirical and developer study. On a randomly selected and manually labeled 200 online reviews, CLAA achieved 92% accuracy while the SOTA baseline achieved 81.5%. According to our developer study involving 10 participants, the use of 'Stack Overflow + CLAA' resulted in increased accuracy and confidence during API selection. Replication package: https://github.com/disa-lab/Contrastive-Learning-API-Aspect-ASE2023