Towards Leveraging News Media to Support Impact Assessment of AI Technologies

TL;DR

This paper investigates how fine-tuning open-source large language models on news media reports can enhance impact assessments of AI by capturing diverse negative impacts across social, cultural, and policy dimensions.

Contribution

It demonstrates that fine-tuned open-source LLMs can generate high-quality, diverse negative impact reports, addressing gaps in traditional impact assessment methods.

Findings

Fine-tuned open-source LLMs support impact assessment with high-quality impact reports.

Small open-source LLMs can capture a wider range of impact categories than GPT-4.

Fine-tuning on news media enhances diversity and relevance of impact assessments.

Abstract

Expert-driven frameworks for impact assessments (IAs) may inadvertently overlook the effects of AI technologies on the public's social behavior, policy, and the cultural and geographical contexts shaping the perception of AI and the impacts around its use. This research explores the potentials of fine-tuning LLMs on negative impacts of AI reported in a diverse sample of articles from 266 news domains spanning 30 countries around the world to incorporate more diversity into IAs. Our findings highlight (1) the potential of fine-tuned open-source LLMs in supporting IA of AI technologies by generating high-quality negative impacts across four qualitative dimensions: coherence, structure, relevance, and plausibility, and (2) the efficacy of small open-source LLM (Mistral-7B) fine-tuned on impacts from news media in capturing a wider range of categories of impacts that GPT-4 had gaps in covering.