Experimentation — Power Before p-values

• · P1 — An experiment is a decision procedure, not a measurement. Before it starts, name what you will do for each possible outcome. If every outcome leads to shipping, you have a launch with extra steps; if no outcome would change anything, you have theatre. Both are common and both cost weeks.
• · P2 — Power comes before p-values, and it is arithmetic rather than opinion. A test that cannot detect the effect you care about will not report "no effect" — it will report noise, and somebody will believe it. Detecting a 10% relative lift on a 3% conversion rate needs roughly fifty thousand users *per arm*. Halve the effect and the requirement quadruples.
• · P3 — Most products do not have the traffic, and that is the finding. Running an underpowered experiment is worse than running none: it converts "we don't know" into a number people will quote. When the arithmetic says no, the honest answer is a different method — see [decision-framework.md](decision-framework.md) — not a smaller threshold.
• · P4 — Decide the metric before you see the data. The single criterion the decision rests on, chosen up front. Picking it afterwards means picking whichever moved, which is a procedure guaranteed to find something.
• · P5 — A metric optimized is a metric distorted. Any measure adopted as a target stops measuring what it did. The decision criterion must be close enough to real value that gaming it *is* delivering value — engagement measures are the classic failure, because the easiest way to raise them is usually to make the product worse.

• · Do the sample-size arithmetic before anything else, including before writing the ticket. It takes two minutes and it cancels most proposed experiments. That is the point.
• · Under a few thousand weekly users into the funnel, you cannot A/B test conversion. Say so plainly. It is a fact about arithmetic, not a lack of ambition.
• · Write down what you'll do for each outcome before you launch. If every branch ships, cancel. If no branch changes anything, cancel. Half of proposed tests die here.
• · Ask "how could this metric go up while the product gets worse?" If the answer is.
• · Declare guardrails even when you're confident. They cost nothing and they are the only thing standing between a win and an unnoticed regression.

• · Ver anti-patrones específicos del dominio en el pack.

• [ ] Required sample size was calculated, not guessed. (EXP2)
• [ ] Expected traffic reaches it within about four weeks. If not, the test was not run and an alternative method was chosen and recorded. (EXP3, EXP38)
• [ ] A decision was written for every possible outcome before launch; no branch where everything ships regardless. (EXP1)
• [ ] No arm withholds safety, accessibility, or security. (EXP42)
• [ ] Experiment data is treated as personal data — inventoried, minimized, retained deliberately, covered by the deletion path. (EXP43)
• [ ] One decision criterion, declared before launch, with the smallest effect worth shipping for. (EXP4)

• Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing

Este pack cita las siguientes fuentes registradas en el índice bibliográfico:

• webTrustworthy Online Controlled Experiments: A Practical Guide to A/B TestingRon Kohavi, Diane Tang, Ya Xu. Cambridge University Press