What is Inverse Propensity Scoring in LTR?

Inverse Propensity Scoring (IPS) in Learning to Rank (LTR) is a method for correcting biases in data that arise because user clicks depend not only on relevance but also on position or other factors. IPS is used to evaluate the quality of ranking based on clicks, compensating for the uneven distribution of click probabilities.

The idea is that each click is weighted inversely proportional to the probability that the user will see and click on that item (propensity). Thus, clicks on positions with a low click probability are given more weight, helping to obtain a more honest estimate of the model.

Example: if the click probability at position 1 is 0.8, and at position 5 is 0.2, then the click at position 5 will be weighted more heavily to compensate for its lower likelihood of being noticed.

This allows training ranking models based on observed clicks, minimizing bias caused by position or other factors.