Why the art testing is necessary and how to do it the right way?

A/B testing (or split testing) allows assessing how the ad creatives affect the attraction of players. Usually, art is tested for two reasons:

  • when the visual component of the game has not yet been determined and the team is searching for the best-fitting artistic style;
  • to determine the cost-effectiveness of the chosen style.

What’s worth attention?

The testing should be aimed at one task. If you need information on several points – run several tests. Never try to shove several tasks into one test at once: you’ll end up with useless results. Art is tested for different parameters; let’s take a look at the main four.

  1. Which setting is best combined with the chosen art style?

When testing the setting, try to ensure that everything depicted in the picture remains within the chosen style. Take a look at the four characters on each of the following images below –  the setting is the same.

The CTR is often very influenced by the pop culture. For example, the premiere of a new Marvel movie can make people more willing to click on the art with superheroes. Unfortunately, something popular today does not necessarily remain popular in a month.

  1. Which art style best fits the chosen setting?

There is a wide range of different art styles: 3D, pixel, low-poly, cartoon and so on.  The style can tell a lot about the game. If someone watches SpongeBob SquarePants for the first time, there is no need to explain that this is a children’s cartoon: the art immediately makes clear that the series is for kids. When choosing the art style for testing it is important to imagine your target audience.

  1. What color palette is best for the game?

The color palette is sometimes also tested, but most often it is determined by the audience of the project and the emotions that the game should trigger.

  1. How commercially popular is the chosen art?

When it comes to the game art, this is perhaps one of the main questions. In general, the trend is as follows: the more casually the game looks, the cheaper the users’ purchase. But the cost of users within the three main categories (casual, midcore and hardcore) still varies greatly! CPI testing is more reliable for determining the commercial demand than the CTR test.

How to measure the results?

The build is ready? If the game already exists (albeit in prototype form), it is possible to measure the CPI, CTR and conversion rate. If the game does not exist, you can measure only CTR.

CTR-test: allows to determine the general interest in the game.

CPI-test: provides more information on the commercial potential of the game. While CTR-test measures the common interest in the game, and the conversion rate – how many people are ready to install the game, the CPI-test reveals how much will cost the users’ purchase.

CTR-test:

  • select one parameter for testing (setting, style, palette, etc.);
  • create 2-4 strongly different ad creatives, but at the same time focus on those elements that you are going to test;
  • add 1 creative from another project with already measured CTR values. It will be used as a reference;
  • create a Facebook ad using selected creatives (use Facebook guides). Make so that when clicking on each of the creatives the user goes to the same application (make sure that no other factors affect the downloadability of the application);
  • ensure that each ad receives the same statistically significant number of impressions.

The most important is confidence in the result! The smaller the difference between the two indicators, the wider the test audience should be.

For example, if one creative has CTR 1.82%, and the other has 2.18%, then it will take 96.2 thousand impressions to find out which is better. If the CTR is 1.70% vs. 2.30%, then only 35 thousand impressions will be required. To calculate how much impressions you need, use a calculator of statistical significance or something alike.

CPI-test:

  • select one parameter for testing (setting, style, palette, etc.);
  • create 2-4 strongly different images, but at the same time focus on those elements that you are going to test;
  • if possible, as a reference simultaneously launch another game in the same countries where the project is being tested. CPI can vary very much from month to month. It is always useful if there is something to compare;
  • create a Facebook ad using selected creatives (use Facebook guides). Make so that when clicking on each of the creatives the user goes to the same application (make sure that no other factors affect the downloadability of the application);
  • ensure that each ad receives the same statistically significant number of clicks.
  • Note: in the case of the CPI test you should count the number of clicks.

Why Facebook?

The answer is simple: targeting! In the campaign settings, you can set both age and gender – so that the art will be displayed only to your target audience. If the testing is performed without a target and shows, for example, a high CTR for some art for preschool age children, despite the fact that the target audience of the game – men 18-35, the results of such test will be useless. If the game is for everyone, then you can test it in any network.

Are the art indicators good or bad?

The market is constantly changing. Therefore, it’s better to use a reference – an image that was used for another game and that can be relied upon when testing a new art, so that you can evaluate the results in comparison.

The test is over. Now what?

  • Ensure that the difference between test results is statistically significant. If it is too small, you may need additional data or images that are even more different from each other.
  • What are the results of the art – are they the same as for the reference image, and better? If not, you may need to make additional changes and re-run the test.

SHARE THIS:
Share on VKShare on Facebook0Share on Google+2Tweet about this on TwitterShare on LinkedIn58

Next Post

Previous Post

Leave a Reply

© 2017 Game Dev Blog by BidOn Games – Everything You Wanted to Know About Making Games