A/B testing seems to be the holy graal of start-upers. It might seem easy at the first sight. But actually is a real science which combine marketing, design, statistics and… chance.
What is A/B testing?
Very simple indeed.
Let’s say that you have a product. whatever product it is of course, however A/B testing is mostly used for webdesign and inbound marketing as you can easily test and learn at high speed. So you have a product. You have 2 possibilities regarding this product:
- A version = orange coloured
- B version = blue coloured
However before launching the production you want to know which one will be sold mostly and thus product the one that people wants.
To do that you have an audience Z that you split into two groups:
- Z(A) will be proposed with the A version (orange)
- Z(B) will be proposed with the B version (blue)
And you wait until you know which of A or B version has been sold the most. That’s it you are done.
Of course A/B testing is based on an assumption that nobody will be able to prove. It stays in the fact that people in the Z(A) group will react exactly the same in front of your test and Z(B) people does… So to be sure that your test worth value, you have to achieve a certain amount of visitors to « average » some particular behaviour. So the most people visit your website, the most trusted your A/B test.
A/Z testing for INNOWEO
Imagine now that I want, on INNOWEO, to do some A/B testing. I have around 150 visitors per day. I would like to optimize the people joining my INNOWEO community (more than 100 members, join !!). I have several option to test:
- Put at the bottom of each post a invitation to join (A)
- Put on the side bar an invitation to join (B)
- Have a pop-up that invite you to join after 10 sec on the site (C)
- Have it highlighted in flashy green (D)
Lots of possibilities.
So I could do as the following:
- One third of my audience will see the A version
- One third will see the B version
- One third the C version
That’s A/Z testing where you test multiple variables. I just have to check what produces the most members.
As for an information I have tested A and B version with greater results for B version, the actual one on INNOWEO. I have never tried the other. Should I?
Multiple variable testing
Let’s know imagine that you want to test the shape and colour of a button, sounds and image. You are now entering the world of statistics and combination. 4 buttons, 3 sounds, 3 images give 36 possibilities. You can manage it iteratively or test it statistically to reduce the testing time. Up to you. A good experiment plan is however a valuable help.
The marketing side
Of course if you A/B test your product it is because you want to know which version is the more suited to your goal. That’s marketing.
So before launching a A/B test campaign prepare it to be sure you will reach your goals:
- define precisely what to test (color, size, media, shape…)
- select what kind of testing method you will apply (that’s key because you will have to perform different tasks)
- Select your comparison factor (sales, new members, click…)
This should be in full accordance with your ultimate goal.
Dangers of A/B testing
Lots of dangers in A/B testing. Some are listed here below.
- Testing something not relevant (don’t test everything, focus on UX)
- Stopping too early (A/B test should be statistically significant)
- Stopping it too late (you are loosing time)
- Awaiting an increase in sale (you are not trying to sell more but trying to understand how to communicate)
What to test first
I would say test whatever will give you some more information about your audience and how it reacts to your value proposition and communication channel:
- test a landing page with a video (it is proven that video is highly engaging)
- test testimonials (social proof)
- test writing (is a « join now! » button more efficient that the « join for free » one?)
- test super fast loading of landing page
- test super short sign-up form
- test value proposition (a xx$ coupon or a free shipping?)
Let’s go, just try as I am doing it now. You didn’t notice? 😉
Oh yes I forgot about the chance in that… Well you’ll see…