How to Write an Experiment Hypothesis That Actually Teaches You Something

Most experiments fail long before the test goes live. They fail at the sentence that justifies them. "Let's try a green button" is not a hypothesis; it is a hunch with a color preference. A good hypothesis turns a hunch into a prediction you can check, and, more importantly, into a lesson you can reuse whether the test wins, loses or ends flat.

This guide walks through a hypothesis format that small growth teams can apply in minutes, the evidence that should sit behind it, and the habits that keep an experiment backlog honest.

Why the hypothesis matters more than the test

An A/B test is a measuring instrument. It reports whether variant B moved a metric compared to variant A, within some margin of uncertainty. What it cannot do is explain the result. The explanation has to come from you, and the only reliable place to put it is in writing, before you see the numbers.

Writing the reasoning first protects you from the most common trap in experimentation: post-hoc storytelling. When a test wins, it is tempting to invent a tidy narrative. When it loses, it is tempting to blame traffic quality or seasonality. A pre-registered hypothesis forces you to state what you believed and why, so the result either supports that belief or challenges it.

There is a practical benefit too. A team that writes clear hypotheses builds a library of tested beliefs about its users. After twenty experiments you no longer have twenty isolated outcomes; you have a map of what your audience responds to.

A template you can use today

The format below has five parts. Each one answers a question a skeptical teammate would ask.

# Observation
Because we saw [evidence],
# Change
we believe that [specific change] for [audience segment]
# Effect
will cause [expected behavior change].
# Metric
We will know this is true when [primary metric] moves by
at least [minimum detectable effect] over [duration].
# Reasoning
This should work because [mechanism / user psychology].

1. Observation

Start with something you actually saw. Good sources include funnel reports showing a sharp drop between two steps, session recordings where people hesitate or rage-click, support tickets that repeat the same confusion, survey answers to "what almost stopped you from signing up?", and sales call notes. If you cannot name the observation, you are guessing, and that is a signal to do research before building a variant.

2. Change

Describe the change precisely enough that a developer or designer could build it without asking questions. "Improve the onboarding" is not a change. "Replace the five-field signup form with an email-only form and collect company size on the second screen" is. Also state who sees it: all visitors, new visitors from paid search, trial users on the free plan, and so on.

3. Expected effect

Name the behavior you expect to change. More people starting a trial, fewer people abandoning checkout, more invited teammates in the first week. Behavior language keeps you focused on users rather than on page elements.

4. Metric and threshold

Choose one primary metric and decide in advance how big a change would matter to the business. This is your minimum detectable effect. It drives sample size, and it prevents the trap of celebrating a statistically detectable lift that is too small to be worth maintaining. Add one or two guardrail metrics, such as refund requests or support volume, to catch wins that cause damage elsewhere.

5. Reasoning

This is the part most teams skip and the part that creates learning. Why should the change work? Perhaps it reduces perceived effort, clarifies value, adds social proof, removes a fear about pricing, or shortens time to the first useful outcome. Naming the mechanism lets you generalize: if reducing perceived effort worked on signup, it may also work on the invite flow.

A worked example

Imagine a project management SaaS where many visitors land on the pricing page, but few click through to start a trial. Recordings show visitors scrolling back and forth between the plan cards and the feature comparison table, and several support emails ask whether the trial requires a credit card.

Because we saw repeated questions about card requirements and long hesitation on the pricing page, we believe that adding "No credit card required" directly under each trial button for all pricing page visitors will increase trial starts. We will know this is true when the trial-start rate from the pricing page rises by at least 8% relative over three full weeks. This should work because it removes a perceived risk at the exact moment of decision.

Notice what this statement does. It ties the change to evidence, it limits scope to one page, it names a mechanism (risk removal), and it sets a threshold that matters. If the test is flat, the team learns that card anxiety was probably not the main blocker, and the next hypothesis might focus on plan clarity instead.

Prioritizing hypotheses without endless debate

Once a team gets into the habit, the backlog grows quickly. A lightweight scoring method keeps prioritization quick. Many teams use a variant of ICE: impact, confidence and ease, each scored from one to five.

• Impact: how much would the primary metric move if the hypothesis is right, and how many users does the change touch?
• Confidence: how strong is the evidence? Multiple independent sources earn a higher score than a single opinion.
• Ease: how much design, engineering and review time does it take to build and run?

Scores are rough, and that is fine. Their purpose is to make assumptions visible so the team can argue about evidence rather than about whose idea it is.

Common mistakes to avoid

Testing solutions instead of problems

If the hypothesis starts with the change ("We believe a carousel will…"), it is often a solution looking for a problem. Lead with the observation instead.

Moving the goalposts

Switching the primary metric after launch because a secondary one looks better is the quickest way to fool yourself. Secondary metrics are for exploration and new hypotheses, not for declaring victory.

Stopping early

Checking results daily and stopping the moment a variant looks ahead inflates false positives. Decide the duration up front, ideally in full weekly cycles so weekday and weekend behavior are both represented.

Ignoring sample size

Low-traffic pages may need months to detect a modest effect. In those cases, test bolder changes, move the test further up the funnel where traffic is higher, or use qualitative methods instead.

Documenting results so they compound

When the test ends, return to the original hypothesis and add three things: the result with its uncertainty, a plain-language interpretation, and the next question it raises. Store all of it in one searchable place, whether that is a shared doc, a wiki or a simple spreadsheet.

1. Result: what happened to the primary and guardrail metrics.
2. Interpretation: did the evidence support the mechanism you named?
3. Next step: ship, iterate, or retire the idea, plus any follow-up hypothesis.

Over time this log becomes one of the most valuable assets a growth team owns. New teammates can read it to understand the audience, and the team stops re-testing ideas that already failed for well-understood reasons.

Bringing it together

A strong hypothesis is short, specific and honest about its assumptions. It starts with something you observed, describes a change precise enough to build, predicts a behavior shift, commits to a metric and threshold, and explains the mechanism. Write it before launch, revisit it after, and you will learn something from every test you run, not only the winners.