TruthTorchLM: A Comprehensive Library for Predicting Truthfulness in LLM Outputs

TL;DR

TruthTorchLM is an open-source Python library that offers over 30 diverse truthfulness prediction methods for LLM outputs, facilitating research and development in high-stakes AI applications.

Contribution

It provides a comprehensive, extensible collection of truthfulness prediction techniques compatible with popular LLM frameworks, surpassing existing toolkits in scope and flexibility.

Findings

Evaluation on TriviaQA, GSM8K, and FactScore-Bio datasets.

Comparison of diverse truthfulness prediction methods.

Demonstration of library's extensibility and compatibility.

Abstract

Generative Large Language Models (LLMs)inevitably produce untruthful responses. Accurately predicting the truthfulness of these outputs is critical, especially in high-stakes settings. To accelerate research in this domain and make truthfulness prediction methods more accessible, we introduce TruthTorchLM an open-source, comprehensive Python library featuring over 30 truthfulness prediction methods, which we refer to as Truth Methods. Unlike existing toolkits such as Guardrails, which focus solely on document-grounded verification, or LM-Polygraph, which is limited to uncertainty-based methods, TruthTorchLM offers a broad and extensible collection of techniques. These methods span diverse tradeoffs in computational cost, access level (e.g., black-box vs white-box), grounding document requirements, and supervision type (self-supervised or supervised). TruthTorchLM is seamlessly compatible with both HuggingFace and LiteLLM, enabling support for locally hosted and API-based models. It also provides a unified interface for generation, evaluation, calibration, and long-form truthfulness prediction, along with a flexible framework for extending the library with new methods. We conduct an evaluation of representative truth methods on three datasets, TriviaQA, GSM8K, and FactScore-Bio. The code is available at https://github.com/Ybakman/TruthTorchLM