# Truly Batch Apprenticeship Learning with Deep Successor Features

## TL;DR

This paper presents a new batch apprenticeship learning algorithm that learns an expert's reward structure using only observed data, without requiring a model or additional data collection, and demonstrates its effectiveness on benchmarks and clinical tasks.

## Contribution

The paper introduces Deep Successor Feature Networks and a transition-regularized imitation network for off-policy batch apprenticeship learning, enabling reward inference without a dynamics model.

## Key findings

- Achieves superior results on control benchmarks.
- Successfully applied to sepsis management in ICU.
- Outperforms existing methods in batch settings.

## Abstract

We introduce a novel apprenticeship learning algorithm to learn an expert's underlying reward structure in off-policy model-free \emph{batch} settings. Unlike existing methods that require a dynamics model or additional data acquisition for on-policy evaluation, our algorithm requires only the batch data of observed expert behavior. Such settings are common in real-world tasks---health care, finance or industrial processes ---where accurate simulators do not exist or data acquisition is costly. To address challenges in batch settings, we introduce Deep Successor Feature Networks(DSFN) that estimate feature expectations in an off-policy setting and a transition-regularized imitation network that produces a near-expert initial policy and an efficient feature representation. Our algorithm achieves superior results in batch settings on both control benchmarks and a vital clinical task of sepsis management in the Intensive Care Unit.