How to implement A/B testing?

Asked

Viewed 390 times

8

How can I implement A/B testing on a new system?

I am developing a tool and would like to compare the results of each modification, but I have no practical knowledge of how to implement this type of test.

testing

2 answers

9


How A/B testing works:

A/B testing requires a mass of data already collected to draw conclusions. Therefore, for a new system it is not possible to complete anything immediately.

The process consists of showing slightly differentiated versions of the same thing to different users (randomly), and verifying what action is taken. Not to display both versions at the same time.

Thus a table is constructed in which each record has:

  • The identifier of the variation displayed.

  • The identifier of the action taken by the user.

From there it is only to make an analysis to see if the differentiation has any relevance or not.

Notes:

  1. If the differentiation between the versions is very large it is difficult to know what really caused the behavioral difference, ie, small differentiations make the result more precise.

  2. The greater the mass of data, the more accurate will be the test.

  3. The target audience has to be well-specified, as the variation in behaviour may differ, for example, from one crop to the other.

  4. It is necessary leave well-defined what behaviors are being measured, and obviously have how to obtain and record what was the action of the user... the greater the accuracy of the measure the better. An example is between clicking and not clicking on an advertisement. To register a do not click you need to do this at the time the advertisement is displayed, if the user clicks the record must have its result changed to indicate that the click occurred.

  5. It is possible to change the proportion of random views, but the test result will have to be balanced with the same proportions

  6. It is possible that the results change with the time of day, or else with other seasonal variables, so it is necessary to be careful with this, and determine whether there are external variables, and what they are

  • For this is usually used some kind of framework or everything should be done by hand as required?

  • It is worth noting that the A/B tests deal with functional tests from the user’s point of view. I do not know if the AP knows, but these tests are not effective to evaluate internal behavior of the system (example, if the calculation became faster, accurate, etc in version A or B), unless these purchases translate into something observable in the person using the system (like, that "look, it’s faster now!" rs).

2

The A/B test is used when you have two implementation options that are equally valid. For example: If during the development phase of a web page (e-commerce) your team believes that the placement of the [Buy] button would look as good next to the product image as next to the product description. In this case it would be interesting to use the A/B test to know in which of the two situations the best results are obtained. Two versions would be created and the results collected and compared. In this case you could leave the page during the same time period and check which one resulted in more sales. This test is used when we have two good solutions and we are in doubt on which to adopt. It is usually not done for all system modifications. If your team is in doubt about one modification or another, it’s best to let users decide through this test.

Browser other questions tagged testing

You are not signed in. Login or sign up in order to post.