Building AI Agents to Improve Job Referral Requests to Strangers

TL;DR

This paper introduces AI agents that assist job seekers in crafting more effective referral requests by rewriting and evaluating them, significantly increasing predicted success rates for weaker requests using advanced language models and retrieval techniques.

Contribution

The paper presents a novel AI system combining rewriting and evaluation agents with retrieval-augmented generation to improve referral request quality and success prediction.

Findings

Revisions by LLM increase success probability for weaker requests.

RAG enhances LLM edits, preventing deterioration of stronger requests.

Predicted success rates improve by 14% for weaker requests with RAG.

Abstract

This paper develops AI agents that help job seekers write effective requests for job referrals in a professional online community. The basic workflow consists of an improver agent that rewrites the referral request and an evaluator agent that measures the quality of revisions using a model trained to predict the probability of receiving referrals from other users. Revisions suggested by the LLM (large language model) increase predicted success rates for weaker requests while reducing them for stronger requests. Enhancing the LLM with Retrieval-Augmented Generation (RAG) prevents edits that worsen stronger requests while it amplifies improvements for weaker requests. Overall, using LLM revisions with RAG increases the predicted success rate for weaker requests by 14\% without degrading performance on stronger requests. Although improvements in model-predicted success do not guarantee more referrals in the real world, they provide low-cost signals for promising features before running higher-stakes experiments on real users.