Challenges of Convergence in Reinforcement Learning
The biggest convergence challenges in RL show up when you go beyond the simple “tabular + on policy” setting and combine bootstrapping , off policy data , and function approximation —because then the learning target c...
The biggest convergence challenges in RL show up when you go beyond the simple “tabular + on policy” setting and combine bootstrapping , off policy data , and function approximation —because then the learning target can effectively become moving / coupled to the current model . 1) Bootstrapping makes the target depend on the current approximation TD methods “replace the unknown target” with a bootstrapped estimate, and that target “depends on the current approximation.”[:cite[1]{ln=3}] So the update isn’t just minimizing a fixed supervised loss; the “mathematical problem” changes as the approximation changes.[:cite[2]{ln=6}] 2) Off policy learning can make the moving target chase instability The lecture warns that when you combine bootstrapping + off policy learning + function approximation , “convergence is no longer automatic.”[:cite[1]{ln=3}] It also states that with “off policy data or nonlinear approximators, the update can chase a moving target in unstable ways.”[:cite[2]{ln=4}] 3) With function approximation, the optimization target isn’t the true value regression target In supervised learning, you know the true target; in RL “the true value is not directly given by a supervisor,” so you must “replace it with sampled targets.”[:cite[3]{ln=4}] This means you’re not just doing plain regression to fixed labels; your targets are estimates (e.g., Monte Carlo return vs TD target).[:cite[5]{ln=7}][:cite[4]{ln=1}] 4) Targets are not the same as pure gradient descent on a fixed error The lecture explicitly notes that TD is “not a pure gradient descent update on the mean squared value error,” and that “distinction matters for convergence.”[:cite[4]{ln=2}][:cite[4]{ln=3}] 5) Neural network value approximation introduces additional failure modes (needs stabilization) When going to deep value estimation, the lecture highlights that naive deep Q learning is “unstable” because the same network both “chooses the target and learns from that target,” and DQN stabilizes this with (1) experience replay and (2) fixed/older target parameters via a target network .[:cite[6]{ln=2}][:cite[6]{ln=3}][:cite[7]{ln=2}] Practical takeaway: RL convergence is hard mainly because the learning update uses bootstrapped targets that can shift as your approximator changes, and this gets especially unstable when learning from off policy data and/or using nonlinear function approximators .[:cite[2]{ln=3}][:cite[1]{ln=3}]