Breaking the Trade-Off Between Faithfulness and Expressiveness for Large Language Models

TL;DR

This paper introduces Collaborative Decoding (CoDe), a novel method that improves the balance between faithfulness and expressiveness in large language models by dynamically integrating external knowledge and internal outputs.

Contribution

It presents CoDe, a new decoding strategy that effectively combines knowledge-based and internal model outputs, along with a reranking mechanism, to enhance faithfulness without losing expressiveness.

Findings

CoDe improves faithfulness in LLM outputs.

The approach maintains or enhances expressiveness.

Experimental results show broad applicability across models.

Abstract

Grounding responses in external knowledge represents an effective strategy for mitigating hallucinations in Large Language Models (LLMs). However, current LLMs struggle to seamlessly integrate knowledge while simultaneously maintaining faithfulness (or fidelity) and expressiveness, capabilities that humans naturally possess. This limitation results in outputs that either lack support from external knowledge, thereby compromising faithfulness, or appear overly verbose and unnatural, thus sacrificing expressiveness. In this work, to break the trade-off between faithfulness and expressiveness, we propose Collaborative Decoding (CoDe), a novel approach that dynamically integrates output probabilities generated with and without external knowledge. This integration is guided by distribution divergence and model confidence, enabling the selective activation of relevant and reliable expressions from the model's internal parameters. Furthermore, we introduce a knowledge-aware reranking mechanism that prevents over-reliance on prior parametric knowledge while ensuring proper utilization of provided external information. Through comprehensive experiments, our plug-and-play CoDe framework demonstrates superior performance in enhancing faithfulness without compromising expressiveness across diverse LLMs and evaluation metrics, validating both its effectiveness and generalizability.