Statistical Significance & Other A/B Test Pitfalls

Last week I tossed a coin a hundred times. 49 heads. Then I changed into a red t-shirt and tossed the same coin another hundred times. 51 heads. From this, I conclude that wearing a red shirt gives a 4.1% increase in conversion in throwing heads.

Put that way, you’d scoff that obviously chance and randomness cause the variation. We should keep that in mind when the variation is small. Especially if the test sample is small.