Validate product value: Hypothesis-driven methodology

hypothesis driven

The article is going to introduce you about how to validate product value before we build it. What is hypothesis-driven development in product discovery with clear example on how to appy it.


Key Takeaways

The hypothesis-driven steps to validate product value

  1. Make your own hypothesis (at least you believe in)
  2. Conduct the experiment
  3. Roll the test out (A/B testing and/or user interview)
  4. Evaluate the results of the experiment
  5. Make another experiment round if needed

Why do we need hypothesis-driven approach?

Start with idea: Even the whole product or just one small product feature started with idea(s). While draw-ideas may come from you, your direct manager, CEO’s vision, key customers or users. However, ideas formed up by individual logical assumption which could possibly lead to bias.

The scenario: when you decide to build, it could take months. And your stakeholders put in there high expection, a promissing product could fulfill their business KPI. The rabbit hole here is if product managers didn’t do their work well, these ideas could be easily turned to requirements. As you have already known the result, building somethings with many unanswered questions and bias will lead to a fail product.

Speed is matter! the hypothesis-driven methodology provides you a way to validate idea in less than 1 weeks and move away from these risks.

Hypothesis-driven methodology

Following Barry O’reilly, the steps of the scientific method are to:

  1. Make observations
  2. Formulate a hypothesis
  3. Design an experiment to test the hypothesis
  4. State the indicators to evaluate if the experiment has succeeded
  5. Conduct the experiment
  6. Evaluate the results of the experiment
  7. Accept or reject the hypothesis
  8. If necessary, make and test a new hypothesis

These steps could be shorten to make it more practical. First of all, we start with our own assumption! Why? Starting with somethings that you at least believe in bringing the highest chance of success. Writing down our assumption (user story) as:

  • We believe <this capability>
  • Will result in <this outcome>
  • We will have confidence to proceed when <we see a measurable signal>

Example

My hypothesis (Step 1 – 4)

We believe that having more detail about room facilities in hotel detail page. Will result in improving conversion across channels. We will have confidence to proceed when conversion increases by +3% compared to the current version.

Conduct the experiment (Step 5)

I assumed that user will put their interest in these specific room facilities information from top-down:

  • Bed size, room size (50%)
  • Cost for additional people (30%)
  • Additional cost for 3 hours early checkin and late checkout (10%)
  • The avg. cost per meal at this hotel. (10%)

I did assume that our users have interest in bed size like king-bed, queen-bed, size of room 30 m2, or 50 m2. Then, we asked our UX team and Marketing team to make these changes in Google Optimize (a client-side A / B testing tool) and visualize all these information explicitly on 20 hotel detail page. We also conduct 20 beta-users interview to find out what is their interest.

If conversion rate of these 20 hotels increased by +3%, we will turn this hypotheis to backlog and plan for the next development sprint. Otherwise, the idea will be trashed.

Evaluate the results of the experiment (step 6)

The conversion rate increate by +1% and user inteview showed that users have interest in the early checkin and late checkout cost than other facilities information:

  • Additional cost for 3 hours early checkin and late checkout (40%).
  • Cost for additional people (40%).
  • Bed size, room size (10%, but that 10% are corporate users).
  • The avg. cost per meal at this hotel. (10%).

Another round of hypothesis (step 7 – 8)

The result was not black or white, but a lot useful information is collected. The whole team conduct another round of hypothesis and of course, this time we have more confident.

Conclusion

“Easily said than done!” The Hypothesis-driven is great, but it’s not always applicable. Sometimes I just mindlessly put a draw-idea to development (it was bad) without validate product value because it is my CEO coolest vision. As product managers, we all need to put this methodology in mind and apply it whenever you can. There are serval way to solve the mentioned issue, but it will be in another post: The managing up. You can also find more about Basic A / B Testing in here.

Reference

Barry O’Reilly. (2020, March 16). How to Implement Hypothesis-Driven Development. https://barryoreilly.com/

Lawrence, S. (2019, January 18). Learning is fun: experiments with Hypothesis-Driven Development.

Sylvia Lai. (2018, March 22). 5 steps to a hypothesis-driven design process.

Leave A Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.