What is Double DQN and what problem does it solve?

Double DQN (Double Deep Q-Network) is an improvement of the classic DQN algorithm in reinforcement learning. The main issue that Double DQN addresses is the overestimation of Q-values, which occurs in standard DQN due to using the same neural network estimator for both selecting and evaluating actions.

Double DQN separates these two tasks: one network selects the action with the highest Q-value, and the other evaluates this action. This reduces bias and leads to more stable and accurate training.

Example:

• In DQN: target = reward + gamma * max_a' Q(next_state, a')
• In Double DQN: target = reward + gamma * Q_target(next_state, argmax_a' Q_online(next_state, a'))

Where Q_online is the main network that chooses the action, and Q_target is the target network evaluating the chosen action.