thinkhard.SI

Prompt repository Product

Plan an A/B test before you run it

You want to test a change and need the hypothesis, metrics and decision rule agreed in writing before anyone looks at results.

Registry no.
PRD-007
Collection
Product
Laid out for
Any chat model
Score
10 out of 10
Blanks to fill
6
Added

ProductAny chat

The prompt

Score 10 of 10, Excellent

Use it when You want to test a change and need the hypothesis, metrics and decision rule agreed in writing before anyone looks at results.

  • {{team}}
  • {{change}}
  • {{rationale}}
  • {{surface_and_audience}}
  • {{baseline_and_traffic}}
  • {{constraints}}
Full prompt and why it works
Prompt
Help me write a one-page plan for an A/B test, so that {{team}} agrees on what success means before the test starts and nobody reinterprets the results afterward.

The change we want to test: {{change}}
Why we think it will work, and the evidence: {{rationale}}
Where it runs and who sees it: {{surface_and_audience}}
Current baseline for the main metric, and typical weekly traffic: {{baseline_and_traffic}}
Constraints, such as launch dates, other tests running, or legal and privacy limits: {{constraints}}

Write the plan with these headings:
1. Hypothesis: "If we [change], then [metric] will [direction] for [audience], because [reason]." One sentence.
2. Primary metric: exactly one, with its precise definition.
3. Guardrail metrics: 2 or 3 that must not get worse, each with the threshold that would stop the test.
4. Design: the variants, the unit of randomization (user, account or session), the traffic split, and who is excluded.
5. Size and duration: the smallest effect worth detecting and the inputs a sample size calculation needs. Give a rough duration only if the baseline and traffic above allow it, show the arithmetic, and tell me to confirm it with a sample size calculator, because rough estimates can be far off.
6. Decision rule: which result means ship, which means do not ship, and which means extend, written before any data comes in.
7. Risks: up to 3, such as novelty effects, seasonality, or a change that also affects users in the control group.

Keep it under 500 words, in plain language a non-analyst can follow. Do not choose a metric because it is likely to move; choose the one that reflects the goal. Avoid adding secondary metrics beyond the guardrails, since every extra metric raises the chance of a false win.

If the change cannot be tested this way, for example because traffic is too low or the change cannot be shown to only some users, say so and suggest a better way to evaluate it.

Variables

{{team}}
Who needs to agree, for example "the growth team and the analytics lead".
{{change}}
What the variant does differently, in one or two sentences.
{{rationale}}
Why you expect it to work: research, past tests, customer feedback.
{{surface_and_audience}}
The page, screen or email, and which users are included.
{{baseline_and_traffic}}
The main metric's current value and the weekly traffic, from your analytics.
{{constraints}}
Dates, other experiments, legal or privacy limits. Write "none" if there are none.

Why it works

  • Who it's for. The plan exists so the team agrees on success before seeing data, which is why the decision rule is written up front.
  • What to avoid. One primary metric, a few guardrails and no extra metrics keep the test from finding a false win among many numbers.
  • If unsure. Duration is estimated only from the baseline and traffic you give, with the arithmetic shown and a reminder to confirm it with a proper calculator.

How to use it

  1. Press Fill in variables and type your details into each blank. What you type stays in this browser.
  2. Copy the filled prompt into the AI tool you already use, such as ChatGPT, Claude or Gemini, or open it in the Grader or the Prompt Tester here.
  3. Before you rely on it, try it on three or four real examples. The Prompt Tester runs them all at once and shows which ones go wrong.