The Art of A/B testing

THE ART OF A_B TESTINGA/B testing seems to be the holy graal of start-upers. It might seem easy at the first sight. But actually is a real science which combine marketing, design, statistics and… chance.

What is A/B testing?

Very simple indeed.

Let’s say that you have a product. whatever product it is of course, however A/B testing is mostly used for webdesign and inbound marketing as you can easily test and learn at high speed. So you have a product. You have 2 possibilities regarding this product:

  • A version = orange coloured
  • B version = blue coloured

However before launching the production you want to know which one will be sold mostly and thus product the one that people wants.

To do that you have an audience Z that you split into two groups:

  1. Z(A) will be proposed with the A version (orange)
  2. Z(B) will be proposed with the B version (blue)

And you wait until you know which of A or B version has been sold the most. That’s it you are done.

Of course A/B testing is based on an assumption that nobody will be able to prove. It stays in the fact that people in the Z(A) group will react exactly the same in front of your test and Z(B) people does… So to be sure that your test worth value, you have to achieve a certain amount of visitors to « average » some particular behaviour. So the most people visit your website, the most trusted your A/B test.

A/Z testing for INNOWEO

Imagine now that I want, on INNOWEO, to do some A/B testing. I have around 150 visitors per day. I would like to optimize the people joining my INNOWEO community (more than 100 members, join !!). I have several option to test:

  1. Put at the bottom of each post a invitation to join (A)
  2. Put on the side bar an invitation to join (B)
  3. Have a pop-up that invite you to join after 10 sec on the site (C)
  4. Have it highlighted in flashy green (D)
  5. …..

Lots of possibilities.

So I could do as the following:

  • One third of my audience will see the A version
  • One third will see the B version
  • One third the C version

That’s A/Z testing where you test multiple variables. I just have to check what produces the most members.

As for an information I have tested A and B version with greater results for B version, the actual one on INNOWEO. I have never tried the other. Should I?

Multiple variable testing

Let’s know imagine that you want to test the shape and colour of a button, sounds and image. You are now entering the world of statistics and combination. 4 buttons, 3 sounds, 3 images give 36 possibilities. You can manage it iteratively or test it statistically to reduce the testing time. Up to you. A good experiment plan is however a valuable help.

The marketing side

Of course if you A/B test your product it is because you want to know which version is the more suited to your goal. That’s marketing.

So before launching a A/B test campaign prepare it to be sure you will reach your goals:

  • define precisely what to test (color, size, media, shape…)
  • select what kind of testing method you will apply (that’s key because you will have to perform different tasks)
  • Select your comparison factor (sales, new members, click…)

This should be in full accordance with your ultimate goal.

Dangers of A/B testing

Lots of dangers in A/B testing. Some are listed here below.

  1. Testing something not relevant (don’t test everything, focus on UX)
  2. Stopping too early (A/B test should be statistically significant)
  3. Stopping it too late (you are loosing time)
  4. Awaiting an increase in sale (you are not trying to sell more but trying to understand how to communicate)

What to test first

I would say test whatever will give you some more information about your audience and how it reacts to your value proposition and communication channel:

  • test a landing page with a video (it is proven that video is highly engaging)
  • test testimonials (social proof)
  • test writing (is a « join now! » button more efficient that the « join for free » one?)
  • test super fast loading of landing page
  • test super short sign-up form
  • test value proposition (a xx$ coupon or a free shipping?)

Let’s go, just try as I am doing it now. You didn’t notice? 😉

Oh yes I forgot about the chance in that… Well you’ll see…


19 réflexions sur « The Art of A/B testing »

  1. Charlie Daniels has credited the ‘unconditional love’ of fatherhood as the factor that made him become one the first English players to sign up to Juan Mata’s Common Goal initiative. Charlie Daniels says signing up to Manchester United star Juan Mata’s Common Goal and giving 1 of his wages to charity was a ‘no-brainer’ as ‘fatherhood changes everything’

  2. Thank you for all your valuable work on this blog. Betty loves working on investigation and it’s really simple to grasp why. Almost all know all concerning the dynamic ways you deliver efficient tips through your web blog and in addition strongly encourage contribution from visitors on this concern then my child has been discovering a lot. Take pleasure in the remaining portion of the new year. You are carrying out a remarkable job.

  3. I wanted to type a message to thank you for all of the pleasant suggestions you are writing at this site. My rather long internet research has now been paid with good quality facts and strategies to talk about with my companions. I would assume that many of us readers are truly blessed to exist in a notable network with so many wonderful people with beneficial tactics. I feel somewhat fortunate to have seen the site and look forward to so many more exciting moments reading here. Thank you once more for a lot of things.

  4. I would like to show my respect for your generosity in support of individuals that absolutely need assistance with this one topic. Your very own dedication to passing the message along has been extremely beneficial and have frequently encouraged professionals like me to reach their endeavors. Your own warm and helpful hints and tips entails this much to me and much more to my colleagues. Many thanks; from all of us.

  5. I am also writing to make you be aware of of the superb experience my friend’s child developed going through your web page. She figured out a good number of details, including what it is like to possess a wonderful teaching mindset to get many people without difficulty know precisely a variety of advanced things. You really surpassed my desires. Thanks for supplying the informative, trustworthy, revealing and also fun tips about this topic to Lizeth.

  6. I precisely needed to say thanks once more. I am not sure the things that I could possibly have made to happen in the absence of the entire tips discussed by you concerning such a theme. It had been a real intimidating problem in my view, however , considering the specialised manner you treated it made me to leap for fulfillment. Now i’m happy for your service and even wish you realize what a great job that you’re carrying out training people today through your website. I’m certain you have never encountered all of us.

  7. Thank you a lot for giving everyone an exceptionally superb chance to read critical reviews from here. It’s usually very pleasant and also full of fun for me personally and my office mates to visit your site at the very least 3 times in 7 days to learn the fresh tips you will have. And of course, I’m just actually happy with all the striking principles you serve. Some two ideas in this post are really the most suitable we’ve ever had.

  8. I have to express my appreciation to this writer just for rescuing me from this type of crisis. As a result of searching throughout the world wide web and coming across methods which are not pleasant, I was thinking my life was gone. Being alive devoid of the strategies to the problems you have fixed all through your good blog post is a crucial case, as well as the ones which might have badly affected my entire career if I hadn’t discovered your web site. Your training and kindness in touching the whole lot was tremendous. I am not sure what I would’ve done if I had not discovered such a thing like this. It’s possible to at this moment look forward to my future. Thanks very much for your reliable and effective guide. I will not hesitate to propose your web page to any person who needs to have guide on this situation.

  9. I’m just writing to make you know of the notable encounter our princess found reading through yuor web blog. She came to find several things, most notably what it’s like to have an incredible coaching mindset to get men and women easily completely grasp a variety of problematic subject areas. You truly did more than my expectations. Thanks for providing those valuable, trustworthy, revealing and easy tips on that topic to Emily.

  10. My spouse and i got very satisfied that Peter could do his research using the ideas he grabbed when using the blog. It is now and again perplexing to just continually be giving freely thoughts which often the rest may have been selling. And now we already know we have got the writer to be grateful to for that. The illustrations you’ve made, the simple website menu, the relationships you can help to foster – it is mostly exceptional, and it’s making our son in addition to us reason why this subject matter is awesome, which is certainly exceedingly vital. Thanks for the whole lot!

  11. I enjoy you because of your entire effort on this website. My daughter takes pleasure in doing research and it’s really simple to grasp why. A number of us learn all of the powerful manner you create practical secrets by means of this website and as well encourage contribution from other people on the situation plus our favorite simple princess is actually discovering a lot. Take pleasure in the rest of the new year. You are performing a useful job.

  12. I wanted to write you this very little observation to help give thanks once again for the incredible principles you have shown in this case. It is quite remarkably generous with you to make easily all most of us could have advertised as an ebook to make some dough for themselves, precisely considering the fact that you might well have done it in case you wanted. These secrets likewise worked to provide a fantastic way to be sure that other people online have similar keenness really like my very own to grasp a good deal more in respect of this matter. I’m certain there are some more fun sessions up front for people who scan your site.

  13. I precisely needed to appreciate you once again. I’m not certain the things that I might have achieved without the type of points contributed by you concerning this subject. It has been a real difficult matter for me personally, however , considering this skilled style you treated the issue took me to jump with happiness. I am happy for the help and as well , believe you are aware of a great job you are always carrying out teaching many others all through a site. More than likely you haven’t got to know any of us.

  14. I needed to post you a very little observation to help say thank you over again for your beautiful knowledge you have discussed at this time. It has been tremendously open-handed with people like you to offer easily exactly what some people would have distributed for an electronic book to get some profit on their own, primarily since you might have done it if you considered necessary. These concepts also worked to become great way to understand that some people have a similar fervor just as mine to understand many more on the topic of this matter. Certainly there are millions of more pleasurable occasions ahead for individuals who examine your site.

  15. I just wanted to construct a brief word to be able to thank you for all of the fabulous facts you are giving at this website. My time intensive internet investigation has at the end of the day been paid with extremely good ideas to exchange with my guests. I ‘d claim that we readers are definitely lucky to live in a really good website with very many wonderful individuals with helpful secrets. I feel somewhat blessed to have come across the web pages and look forward to many more pleasurable minutes reading here. Thanks a lot once more for everything.

Laisser un commentaire

Votre adresse de messagerie ne sera pas publiée.