Basic A / B Testing for product discovery and release

A / B Testing

Key Takeaways

  • A / B testing is used to test product’s ideas, providing you an empirical data for decision making
  • There are 2 types of A / B testing: server-side and client-side.
  • Server-side is best for product release phase
  • Client-side A / B testing is best for product discovery phase

What is A / B testing

A/B testing is a way to compare two or more versions of something to figure out which performs better. There are 2 A / B testing types: Server-side and client-side testing. Based on your needs at a certain product phase, you can decide which type is better than another.

It is all about data supporting to your decision. Imagine you and your friend had a dicussion about whether Cappucino or Latte (your product) is more prefered by office workers (your target customers). Without data from the empirical testing, both of your conclusion and your friend’s are guesses based on your personal experience.

Testing types

Focusing on tech software product we have 2 type of A / B testings: server-side and client-side.

Server-side: the test is rendered on the web server itself. The page variations (A, B and C …) are then delivered to the user’s browser where no subsequent modifications take place. The pages being tested are fetched randomly from the server and shown to the visitor.

Server side A/B Testing
Server side A/B Testing
  • Best used: when the scope of an experiment big. This will require coding.
  • Pros: Server-side testing is independent from client’s browser, it can be used for mobile app. It is also suitable for running experiments in any channel at different points of time.
  • Cons: Server-side a/b testing tools are multilayered and complex to use. As you need the team of the front end as well as back-end developers.

Client-side: the test happens at your client’s browser. Javascript will decide which data was feed to your customers. Don’t panic! You won’t have to implement this by coding, we have tools like: Google Optimize, VWO, Optimizely, … help you on this.

client side A/B Testing
client side A/B Testing
  • Best used: when you are deploying an internal team of marketers to run experiments without the need for them to touch coding.
  • Pros: Client-side testing is fast, you could start to test your hypothesis and have the result on the next day. It is far less cost than server-side testing.
  • Cons: Fast deployment comes with a price, you could only test simple changes on your sites. It slow down your client’s browser because it required 3rd party software. Most popular applicant is wording in communication, Telling customer A that they will get 20% discount for the next purchase and telling customer B that they will get freeship if they make purchase now, and see how both options work.

How this planned

Planning:

  1. Client-side A / B Testing: The process lifecycle in client-side testing is shorter.
    • You first create a hypothesis
    • Build the different variations using the AB testing tool
    • Analyze the results and determine the winner.
    • Only the winning variation is then built into the main application code and rolled out for all website users.
    • If the test does not generate any increase in conversions, then all variations are removed from the AB testing software and the original page design is served to visitors.
  2. Server-side A / B testing: Comparatively, the process lifecycle takes a longer route with server-side testing.
    • You first create a hypothesis
    • you implement each of your test variations on the server.
    • Analyze the results and determine the winner.
    • Since all variations are already built on the server, then when a winner is determined, it defaults to all users
    • If the test does not generate any increase in conversions, then all variations are removed from the server the original page design is served to visitors.

How to implement and make your site ready for an A / B testing

The following are typical steps followed when setting up and running a client-side a/b test:

  1. Insert code snippet on your website
  2. Create test version using a visual editor
  3. Create your conversion goals- increase in conversion goals, downloading files, clicking on a link etc.
  4. Start the test on your choice of a/b testing tool
  5. Keep a track of website version performance on the a/b testing tool dashboard

A server side a/b test you need to follow the given steps:

  1. Once a user lands on site, a unique visitor ID is generated using the API.
  2. Call the API passing in parameters the unique visitor ID when the test has to be run. Alongside also pass in the test ID.
  3. Responding to your request, the a/b testing tool sends the variation ID.
  4. The server builds its response built on this variation ID.
  5. Data servers are informed once a conversion takes place.

Source from: Invesp by Khalid Saleh. Last access: April 27th, 2020

How A / B testing benefits product

As we went so far, deciding which type of A / B testing highly depends on what you need.

AB-testing-cycle
AB-testing-cycle

In product discovery phase, product team would want to quickly test their ideas and hypotheses, so client-side A / B testing would be the best choice. Time is matter, you test your hypothesis today and have the result on the next day. How cool is it! Product team at Booking.com told me that they performed over 1000 tests a month!

In product release phase, both tech team and product team not only want to see the impact of their new released feature to business KPI, they also want to measure and compare the performance of released feature to the original one: server’s respond time, the different between iOS, Android, website and mobile site … This is when a server-side A / B testing is needed. And remember, in this product release phase, feature was built and you spent your tech team effort to implement this. There are no turning back, if your hypothesis is wrong, it should be proven at product discovery phase.

In this article, I mentioned about product discovery, product release, User testing, hypothesizing. These steps are all in the big picture of product life-cycle. Please check other articles in our Style the product site to learn more about them.

Leave A Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.