We set audience criteria where the user account must be created after the test launches, from there its a 50/50 split control/treatment experience (based on user id). The metric we are optimizing for is almost always conversion rate. We will turn the experiment off early if the treatment group is having really poor numbers, otherwise once about 4000 accounts have been entered into the experiment we plug the numbers into a bayesian calculator, and call it a winner of there is a 90%+ probability that the treatment beats the control. https://www.abtestguide.com/bayesian/

Why so high?

Beysian results aren't p-values, a 60-70% probably that treatment beats control is just that, not a pvalue of .4 or .3 (which would say nothing).

So one in ten of your findings is bogus.

But isn’t that better than blindly introducing changes without testing them at all?