Mr. Yingru Li is a Ph.D. student of the Chinese University of Hong Kong, Shenzhen, where he is co-advised by Tong Zhang (HKUST) and Tom Zhi-Quan Luo. He received the Bachelor of Engineering from ACM Honor CS Class advised by Kun He in School of Computer Science at Huazhong University of Science and Technology. He was invited to Cornell University by John E. Hopcroft and for exchange study and research on algorithms design and theoretical analysis for social network problems. He is a member of Shaw College, CUHK, Shenzhen. His research is supported by Presidential Fellowship and Tencent Ph.D. fellowship.
The Chinese University of Hong Kong
B.Eng. (Honor) in Computer Science
Huazhong University of Science and Technology
In deep reinforcement learning, policy optimization methods need to deal with issues such as function approximation and the reuse of off-policy data. Standard policy gradient methods do not handle off-policy data well, leading to premature convergence and instability. This paper introduces a method to stabilize policy optimization when off-policy data are reused. The idea is to include a Bregman divergence between the behavior policy that generates the data and the current policy to ensure small and safe policy updates with off-policy data. The Bregman divergence is calculated between the state distributions of two policies, instead of only on the action probabilities, leading to a divergence augmentation formulation. Empirical experiments on Atari games show that in the data-scarce scenario where the reuse of off-policy data becomes necessary, our method can achieve better performance than other state-of-the-art deep reinforcement learning algorithms.