Challenges of Using Pre-trained Models: the Practitioners' Perspective

TL;DR

This study investigates the practical challenges faced by practitioners when using pre-trained models by analyzing a large dataset of questions, revealing increasing popularity but significant difficulties and gaps in current techniques.

Contribution

It provides a comprehensive taxonomy of challenges faced in applying pre-trained models, based on analysis of Stack Overflow questions, highlighting practical issues and research gaps.

Findings

PTM-related questions are increasing in popularity.

PTM questions have lower response rates and longer response times.

Identified key challenges like fine-tuning and prompt customization.

Abstract

The challenges associated with using pre-trained models (PTMs) have not been specifically investigated, which hampers their effective utilization. To address this knowledge gap, we collected and analyzed a dataset of 5,896 PTM-related questions on Stack Overflow. We first analyze the popularity and difficulty trends of PTM-related questions. We find that PTM-related questions are becoming more and more popular over time. However, it is noteworthy that PTM-related questions not only have a lower response rate but also exhibit a longer response time compared to many well-researched topics in software engineering. This observation emphasizes the significant difficulty and complexity associated with the practical application of PTMs. To delve into the specific challenges, we manually annotate 430 PTM-related questions, categorizing them into a hierarchical taxonomy of 42 codes (i.e., leaf nodes) and three categories. This taxonomy encompasses many PTM prominent challenges such as fine-tuning, output understanding, and prompt customization, which reflects the gaps between current techniques and practical needs. We discuss the implications of our study for PTM practitioners, vendors, and educators, and suggest possible directions and solutions for future research.