Role Of Experience Replay In DQN
In DQN, experience replay makes training more stable and more data efficient by reusing old experience and breaking up harmful correlations in the data. It stores past transitions as an explicit replay memory, instead...
In DQN, experience replay makes training more stable and more data efficient by reusing old experience and breaking up harmful correlations in the data. It stores past transitions as an explicit replay memory, instead of throwing them away after learning from them once.[:cite[1]{ln=1}], [:cite[2]{ln=1}] During training, DQN samples random minibatches from that memory , and then performs gradient updates using those sampled transitions.[:cite[1]{ln=1}] This “decorrelates the trajectories” : instead of using highly correlated consecutive experience in order, replay randomizes the order transitions are used , which yields more stable updates .[:cite[3]{ln=1}] It also improves the data efficiency , since each experience can be reused many times via replay (and the lecture explicitly calls out “better data efficiency” and “less correlation between consecutive updates”).[:cite[4]{ln=1}], [:cite[5]{ln=1}]