Reinforcement Learning for Scalable and Trustworthy Intelligent Systems

TL;DR

This paper discusses advancing reinforcement learning to improve scalability in distributed systems and trustworthiness in language models, focusing on federated optimization and safety alignment.

Contribution

It introduces four key contributions that enhance reinforcement learning's scalability and trustworthiness through federated methods and safety measures.

Findings

Developed communication-efficient federated optimization techniques.

Improved alignment of language models with human preferences.

Reduced inappropriate information disclosure in language-based systems.

Abstract

Reinforcement learning has become a powerful paradigm for improving the capability of intelligent systems, but its practical deployment faces two central challenges. First, reinforcement learning must scale efficiently in distributed environments where communication bandwidth is limited and computation is heterogeneous across agents. Second, as reinforcement learning is increasingly used in post-training large language models and autonomous agents, the optimized policies must also be aligned with human preferences and satisfy safety requirements such as privacy-aware information disclosure. This dissertation addresses both challenges through four complementary contributions spanning federated optimization, preference alignment, and contextual safety. The first part of the dissertation studies scalable reinforcement learning in federated settings. The second part of the dissertation studies trustworthy reinforcement learning for large language models. Together, these contributions advance reinforcement learning along two complementary dimensions. On the one hand, they make reinforcement learning more scalable through communication-efficient and asynchronous federated optimization. On the other hand, they make reinforcement learning more trustworthy by improving alignment with human preferences and by reducing contextually inappropriate information disclosure in language-based intelligent systems. As a whole, this dissertation argues that the next generation of intelligent systems will require both efficient optimization and trustworthy behavior, and that reinforcement learning provides a unifying framework for addressing both goals.