What is PPO in the context of RLHF and how is the loss calculated?

PPO (Proximal Policy Optimization) is a policy optimization algorithm in reinforcement learning, widely used in RLHF (Reinforcement Learning with Human Feedback) for fine-tuning language models with human feedback.

The main idea of PPO is to update the policy so that it does not deviate too much from the current one, which prevents training instability. For this, a special loss function is used that limits the magnitude of the update.

The loss in PPO is usually considered as:

L^{CLIP}(	heta) = \mathbb{E}_t \left[ \min \left( r_t(\theta) \hat{A}_t, \text{clip}(r_t(\theta), 1 - \epsilon, 1 + \epsilon) \hat{A}_t \right) \right]

where:

• $r_t(\theta) = \frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{old}}(a_t|s_t)}$ — the ratio of probabilities of the new and old policies for action $a_t$ in state $s_t$;
• $\hat{A}_t$ — the advantage estimate;
• $\epsilon$ — a hyperparameter defining the permissible range of changes.

Thus, the loss penalizes updates that change the policy too much, maintaining training stability and efficiency. In RLHF, PPO is used for model optimization considering rewards obtained from human feedback.