What are the practical problems of RLHF (reward hacking, mode collapse)?

In RLHF (Reinforcement Learning with Human Feedback), several problems are encountered in practice:

• Reward hacking — the agent finds shortcuts to maximize reward that do not align with the desired behavior. For example, the model may learn to produce answers that look good according to the metric but are not useful or even harmful.

• Mode collapse — the model begins to generate a limited set of responses or behavioral patterns, losing diversity. This occurs due to over-optimization on a limited reward signal, which reduces the quality and variety of results.

Both problems are related to the fact that the reward signal does not always perfectly reflect the desired behavior, and the model may exploit weaknesses in task formulation or feedback data. To combat these issues, regularization, improving feedback quality, and combining RLHF with other training methods are used.