Distributional Reinforcement Learning

Marc G. Bellemare, Will Dabney, Mark Rowland MIT Press, May 2023 · 384 pages · Adaptive Computation and Machine Learning series

This is the first comprehensive guide to distributional reinforcement learning, providing a new mathematical formalism for thinking about decisions from a probabilistic perspective. Rather than computing expected values, the book focuses on how total reward behaves as a probability distribution — presenting core concepts, mathematical proofs, and algorithmic developments for characterising, computing, estimating, and making decisions based on random returns. Applications span finance, computational neuroscience, psychology, macroeconomics, and robotics.

Praise

“This book shows how reinforcement learning agents can estimate and use full reward distributions to make better decisions. It is a significant enhancement that opens a new range of applications.”

— Andrew Barto, Professor Emeritus, University of Massachusetts Amherst; coauthor of Reinforcement Learning: An Introduction

“This important book extends the machinery of reinforcement learning to encompass full outcome statistics. The brain inevitably got there first, but the authors’ brilliant insights show this, how, and why.”

— Peter Dayan, Director, Max Planck Institute for Biological Cybernetics