Complex QA and language models hybrid architectures, Survey

TL;DR

This survey reviews advanced hybrid architectures for large language models tailored to complex question-answering, emphasizing strategies like training, prompting, and agentic architectures to overcome LLM limitations.

Contribution

It provides a comprehensive overview of techniques and solutions for enhancing LLM capabilities in handling complex questions, including hybrid architectures and evaluation metrics.

Findings

Hybrid architectures improve complex reasoning capabilities.

Evaluation metrics like accuracy and explainability are crucial.

Hybrid solutions address LLM limitations effectively.

Abstract

This paper reviews the state-of-the-art of large language models (LLM) architectures and strategies for "complex" question-answering with a focus on hybrid architectures. LLM based chatbot services have allowed anyone to grasp the potential of LLM to solve many common problems, but soon discovered their limitations for complex questions. Addressing more specific, complex questions (e.g., "What is the best mix of power-generation methods to reduce climate change ?") often requires specialized architectures, domain knowledge, new skills, decomposition and multi-step resolution, deep reasoning, sensitive data protection, explainability, and human-in-the-loop processes. Therefore, we review: (1) necessary skills and tasks for handling complex questions and common LLM limits to overcome; (2) dataset, cost functions and evaluation metrics for measuring and improving (e.g. accuracy, explainability, fairness, robustness, groundedness, faithfulness, toxicity...); (3) family of solutions to overcome LLM limitations by (a) training and reinforcement (b) hybridization, (c) prompting, (d) agentic-architectures (agents, tools) and extended reasoning.