What is a p-value and how to draw conclusions from it when analyzing an A/B test (rejecting or not rejecting the null hypothesis)?

P-value — this is the probability of obtaining the observed data (or more extreme) if the null hypothesis is true.

In A/B testing:

• The null hypothesis (H0) usually states that there is no difference between the groups.
• If the p-value is less than the chosen significance level (usually 0.05), we reject the null hypothesis — meaning there is a statistically significant difference.
• If the p-value is greater than or equal to the significance level, there is no basis to reject the null hypothesis — the difference may be due to chance.

Example:

If the p-value = 0.03 and the significance level is 0.05, then we reject H0 and conclude that the changes in group B indeed affect the metric.

It is important to remember that the p-value does not indicate the magnitude of the effect or its practical significance, only the statistical significance.