What is sample ratio mismatch?

Sample Ratio Mismatch (SRM) is a situation in A/B testing where the actual distribution of users across test variants significantly differs from the planned (e.g., 50/50).

The causes of SRM can be related to errors in experiment setup, tracking issues, bugs in the code, or external factors affecting the sample.

Why it is important:

• SRM indicates a violation of random distribution, which can lead to biased and unreliable test results.
• When SRM is detected, it is necessary to verify the correctness of the test implementation and possibly stop the analysis until the issue is resolved.

Statistical tests (such as the chi-square test) are usually used to detect SRM to check the conformity of the actual distribution to the expected one.