What is statistical significance?

Statistical significance is a measure of how much the results of an experiment (for example, an A/B test) differ from random fluctuations and can be considered reliable.

If the result is statistically significant, it means that the probability of obtaining such data by chance is very low (usually less than 5%). This helps to decide that the change truly affects the metric and is not just random noise.

For example, if the conversion rate in group B is higher in an A/B test, and the p-value is less than 0.05, then we can confidently say that the improvement is not due to chance.