Seminar: Safe and efficient off-policy reinforcement learning

SpeakerRemi Munos
AffiliationGoogle DeepMind
DateFriday, 04 Nov 2016
Time13:00 - 14:00
LocationRoberts Building 508
Event seriesMicrosoft Research CSML Seminar Series

In this work, we take a fresh look at some old and new algorithms for off-policy, return-based reinforcement learning. Expressing these in common form, we derive a novel algorithm, Retrace(λ), with three desired properties: (1) low variance; (2) safety, as it safely uses samples collected from any behaviour policy, whatever its degree of "off-policyness"; and (3) efficiency, as it makes the best use of samples collected from near on-policy behaviour policies. We analyse the contractive nature of the related operator under both off-policy policy evaluation and control settings and derive online sample-based algorithms. To our knowledge, this is the first return-based off-policy control algorithm converging a.s. to Q* without the GLIE assumption (Greedy in the Limit with Infinite Exploration). As a corollary, we prove the convergence of Watkins' Q(λ), which was still an open problem. We illustrate the benefits of Retrace(λ) on a standard suite of Atari 2600 games.

Bio: Remi Munos is currently research scientist at Google DeepMind and on leave from Inria. He worked on topics related to reinforcement learning, bandit theory, optimisation, and statistical learning.

iCalendar csml_id_315.ics