Bridging AI and Carbon Capture: A Dataset for LLMs in Ionic Liquids and CBE Research

TL;DR

This paper introduces a new dataset to evaluate and improve the reasoning abilities of small open-source LLMs in the specialized domain of ionic liquids for carbon capture, aiming to support climate change mitigation efforts.

Contribution

We created and released a comprehensive dataset of 5,920 examples for benchmarking LLM reasoning in ionic liquids and CBE, and evaluated the performance of small open-source models in this domain.

Findings

Small LLMs have basic knowledge of ionic liquids.

They lack advanced reasoning skills for CBE applications.

Dataset reveals gaps in current LLM capabilities.

Abstract

Large Language Models (LLMs) have demonstrated exceptional performance in general knowledge and reasoning tasks across various domains. However, their effectiveness in specialized scientific fields like Chemical and Biological Engineering (CBE) remains underexplored. Addressing this gap requires robust evaluation benchmarks that assess both knowledge and reasoning capabilities in these niche areas, which are currently lacking. To bridge this divide, we present a comprehensive empirical analysis of LLM reasoning capabilities in CBE, with a focus on Ionic Liquids (ILs) for carbon sequestration - an emerging solution for mitigating global warming. We develop and release an expert - curated dataset of 5,920 examples designed to benchmark LLMs' reasoning in this domain. The dataset incorporates varying levels of difficulty, balancing linguistic complexity and domain-specific knowledge. Using this dataset, we evaluate three open-source LLMs with fewer than 10 billion parameters. Our findings reveal that while smaller general-purpose LLMs exhibit basic knowledge of ILs, they lack the specialized reasoning skills necessary for advanced applications. Building on these results, we discuss strategies to enhance the utility of LLMs for carbon capture research, particularly using ILs. Given the significant carbon footprint of LLMs, aligning their development with IL research presents a unique opportunity to foster mutual progress in both fields and advance global efforts toward achieving carbon neutrality by 2050.