CRCL at SemEval-2024 Task 2: Simple prompt optimizations

TL;DR

This paper introduces a baseline approach for SemEval-2024 Task 2, utilizing prompt optimization techniques with LLMs to improve inference between clinical trial report sections and statements, highlighting the effectiveness of synthetic CoT prompts.

Contribution

It demonstrates the effectiveness of synthetic Chain-of-Thought prompts in enhancing LLM performance for clinical inference tasks.

Findings

Synthetic CoT prompts outperform manual prompts

Prompt optimization improves inference accuracy

Baseline establishes a reference for future work

Abstract

We present a baseline for the SemEval 2024 task 2 challenge, whose objective is to ascertain the inference relationship between pairs of clinical trial report sections and statements. We apply prompt optimization techniques with LLM Instruct models provided as a Language Model-as-a-Service (LMaaS). We observed, in line with recent findings, that synthetic CoT prompts significantly enhance manually crafted ones.