Experience Replay and Stability in Reinforcement Learning

Experience replay makes reinforcement learning more stable by tackling two instability sources: (1) correlated data and (2) moving learning targets. 1) It decorrelates the training data The lecture highlights that naive online updates come from “highly correlated parts of the trajectory … after the other,” and that this creates unstable learning updates.[‌:cite[1]{ln=1}‌] With experience replay, you randomize the order in which past experience is trained on (“randomize the order in which they arrive”), which “break[s] the correlations” and yields “much more stable updates.”[‌:cite[1]{ln=1}‌] The lecture also states that the benefit is “less correlation between consecutive updates, because minibatches are sampled from memory.”[‌:cite[2]{ln=1}‌] 2) It produces a more “supervised like” training loop The lecture says replay lets you cache experience, then train using “randomly sample from that data set and update in the direction of our random samples,” i.e., turning the loop into something more like supervised learning.[‌:cite[3]{ln=1}‌] It summarizes this as “a more supervised learning like training loop.”[‌:cite[2]{ln=1}‌] Bottom line: experience replay improves stability because it stores past transitions and trains on random minibatches from that replay memory , which reduces correlation between consecutive updates and therefore makes the learning process more stable.[‌:cite[1]{ln=1}‌]