Towards Trustworthy LLMs for Code: A Data-Centric Synergistic Auditing Framework

TL;DR

This paper introduces DataTrust, a data-centric framework for auditing the trustworthiness of large language models for code, integrating training and evaluation data analysis to improve model reliability.

Contribution

It proposes a unified, data-centric auditing framework that links training data quality with evaluation trustworthiness, addressing gaps in existing research.

Findings

Connects model trustworthiness indicators with data quality metrics

Autonomously inspects training data and evaluates models using synthesized data

Engages crowdsourced input for trustworthiness assessment

Abstract

LLM-powered coding and development assistants have become prevalent to programmers' workflows. However, concerns about the trustworthiness of LLMs for code persist despite their widespread use. Much of the existing research focused on either training or evaluation, raising questions about whether stakeholders in training and evaluation align in their understanding of model trustworthiness and whether they can move toward a unified direction. In this paper, we propose a vision for a unified trustworthiness auditing framework, DataTrust, which adopts a data-centric approach that synergistically emphasizes both training and evaluation data and their correlations. DataTrust aims to connect model trustworthiness indicators in evaluation with data quality indicators in training. It autonomously inspects training data and evaluates model trustworthiness using synthesized data, attributing potential causes from specific evaluation data to corresponding training data and refining indicator connections. Additionally, a trustworthiness arena powered by DataTrust will engage crowdsourced input and deliver quantitative outcomes. We outline the benefits that various stakeholders can gain from DataTrust and discuss the challenges and opportunities it presents.