REAL: Benchmarking Abilities of Large Language Models for Housing Transactions and Services

TL;DR

This paper introduces REAL, a comprehensive evaluation suite designed to assess large language models' capabilities specifically in housing transactions and services, highlighting current limitations and areas for improvement.

Contribution

The paper presents the first specialized benchmark for evaluating LLMs in real estate scenarios, covering memory, comprehension, reasoning, and hallucination aspects.

Findings

LLMs show significant room for improvement in real estate applications.

REAL includes 5,316 evaluation entries across 14 categories.

Current LLMs are not yet fully reliable for housing transaction tasks.

Abstract

The development of large language models (LLMs) has greatly promoted the progress of chatbot in multiple fields. There is an urgent need to evaluate whether LLMs can play the role of agent in housing transactions and services as well as humans. We present Real Estate Agent Large Language Model Evaluation (REAL), the first evaluation suite designed to assess the abilities of LLMs in the field of housing transactions and services. REAL comprises 5,316 high-quality evaluation entries across 4 topics: memory, comprehension, reasoning and hallucination. All these entries are organized as 14 categories to assess whether LLMs have the knowledge and ability in housing transactions and services scenario. Additionally, the REAL is used to evaluate the performance of most advanced LLMs. The experiment results indicate that LLMs still have significant room for improvement to be applied in the real estate field.