Menu Search
Jump to the content X X
Smashing Conf Barcelona

You know, we use ad-blockers as well. We gotta keep those servers running though. Did you know that we publish useful books and run friendly conferences — crafted for pros like yourself? E.g. our upcoming SmashingConf Barcelona, dedicated to smart front-end techniques and design patterns.

What A/B Testing Taught Us About App Store Optimization

1.5 million apps in Apple’s App Store and another 1.5 million in Google’s Play store. That’s a lot of apps, and for a growing number of mobile users. An average user in the US will download only three new apps per month (at best), according to comScore’s “US Mobile App Report1.”

Competition in the App Store is fierce, and if an indie app developer wants to get noticed, having an amazing product is no longer enough.

Further Reading on SmashingMag: Link2

Lost And Found In The App Store Link

The truth is that if an app doesn’t show up in the App Store’s top rankings or featured lists, its chances of hitting a high number of views and downloads are slim. If your app hasn’t been featured or “noted” by Apple’s editors or has yet to acquire tens of thousands of downloads to rank in the top 10, then the only way to reach users is via so-called “organic discovery,” or search.

In fact, app store optimization (ASO) is an emerging popular term in mobile technology these days. Some call it the new SEO, and many app marketers have jumped on the trend to support their user acquisition goals. To drive organic traffic to your app’s store page and convert views to downloads, you have to be strategic about the app page’s content.

Hacking The App Store’s Search: A/B Testing Link

When planning your app’s page, you’re ultimately deciding on the following elements: title, icon, screenshots, description and keywords.

How does one decide? For one, you can just go with your gut or ask someone you consider an expert for advice. Many app publishers do exactly that: They either come up with an idea in-house or ask a content strategist or marketing consultant. Alternatively, you can generate several of these go-with-your-gut options and then run a couple of tests to see which one works better. Obviously, A/B testing is a way to figure out which of the two variants tested in a controlled experiment gets better results — in the case of the App Store page, it means a better conversion from page views to app downloads.

The outcome of a simple split test can be significant for a business7

A showed a 7.2% conversion against B’s 5%. The outcome of a simple split test can be quite significant for a business. (View large version8)

We all know how it works: to set up an A/B test, first come up with a hypothesis (for example, the icon of a person wearing a hat and sunglasses will convert better than the one of a person without any beach accessories). Then, you’d run a test, splitting your advertising traffic into two equal parts. Drive 50% to your app page with the first icon, while sending the other 50% to an altered landing page that looks and functions exactly like the real one, except for one little detail: the “hatless” character on the icon. The winner gets the glory of having the next 100% of your ad traffic all to itself.

The outcome of a simple split test can be significant for a business. When Google rolled out A/B testing feature for the Play store at the I/O developer conference in San Francisco last month, it shared9 the success story of a pre-launch pilot with Kongregate, a mobile gaming company that increased its conversion rate by 30% with ta simple split testing.

In our own experience, we saw an even higher change in conversion in the iOS App Store — as much as 150% for a travel app. For a mobile gaming company that tested its video cover images, the experiment resulted in an increased number of views — from 25% to 43%. In a recent Skoda Auto app case study10, a simple rearrangement of screenshots resulted in a 50% increase in conversion from views to downloads on the page. In a more recent Splitmetrics’ case study11, Rovio’s Angry Birds improved conversion by a staggering 13% after experimenting with portrait/landscape and caption variations.

Whether it was the hat and sunglasses that added a layer of mystery to the image or the bright cocktail that caught the eye, we can’t say for sure, but the hat-clad character won. The correct answer is A: A showed a 7.2% conversion against B’s 5%.

A/B Testing App Store Pages: Lessons Learned Link

You can run A/B tests to find a winning icon, screenshot or any other element of an App Store page. The results of such tests will vary, because user behavior is often hard to predict, but I thought I’d share some insights that we learned after running a number of A/B tests with various apps.

Plain Icon Always Works Better Link

Imagine a user scrolling through dozens of apps on the search results page, looking for the one. The icon is often the first element of your product that users see. You want it to stand out and be pretty. Our experiments have shown that optimizing the icon can increase an app’s installation rate by 3 to 5 points (for example, from 10% to 13%). Here are some key takeaways from our A/B tests of icons:

  • Avoid cramming elements together in one icon. Less is more.
  • For your background, go with one soft color. Don’t distract users from the main element with bright colors or clutter.
  • For the main element in the icon, choose one simple visual, like the main character of the game or the app’s logo. It should really pop against the background.
Optimizing the icon can increase an app’s installation rate by 3 to 5 points12

Optimizing the icon can increase an app’s installation rate by 3 to 5 points. (View large version13)

For icon ideas and best practices, check out “iPhone Top App Matrix14” on App Annie, which publishes market reports for app developers.

Screenshots: Show Experience, Tell a Story and Personalize by Device Link

A picture is worth a thousand words, and a good picture on an App Store page is worth more than a thousand downloads. Screenshots will sell your appliterally. In a variety of A/B testing experiments, optimized screenshots have shown an increase in conversions from 35% to 49%, from 55% to 66% and from 13% to 24%, respectively (the app publishers that reported results chose to remain anonymous.)

When planning a page for a utility app, for example, consider including one of the utility’s tools in the screenshot to better show the experience, and explain the key features in captions.

Screenshots will sell your app — literally15

Screenshots will sell your app — literally. (View large version16)

For a game app, your main goal is to appeal to emotion (not to show functionality). Choose the game’s most epic scenes and breathtaking graphics to capture the fun of the game. Give a sneak preview so that users get the point of the game before installing it. Use captions to tell a story, and showcase badges and awards to lure gamers in.

Err on the side of novel, crazy and viral to grab attention. Isn’t Bruce Lee amazing?17

Err on the side of novel, crazy and viral to grab attention. Isn’t Bruce Lee amazing? (View large version18)

Landscape orientation screenshots are more difficult for users to scan. Still, they remain the top choice for mobile game developers. Bigger pictures simply look better on the iPad and appear more impressive in search results.

Apple requires developers to submit screenshots that are optimized for different iOS devices. This is your chance to personalize your app’s story for the user through screenshots: You can present a 1136 × 640-pixel screenshot for the iPhone 5s, a 960 × 640-pixel one for the iPhone 4s and so on. The same goes for the Play store: Upload up to eight screenshots for phones, 7-inch tablets, 10-inch tablets and Android TV.

Video Preview: Costly, No Study Confirms Increase in CVR Link

Video previews are a powerful way to show gameplay in action. The interactivity, sound and graphics drive user engagement and, likely, more downloads, right? Not so fast. To our knowledge, no single study has yet shown any impact on conversion. Some developers of utility apps who have run A/B tests claim to have tracked no significant changes after adding a video preview. Making a preview takes time and money, so the question of whether you should have it remains open.

Video preview is costly and no study confirms that it is effective19

Video preview is costly and no study confirms that it is effective. (View large version20)

Here are some quick tips if you decide to go with a video preview:

  • Remember that the “Play” button is right in the center of the preview thumbnail, so don’t put your logo there. Make it bigger or, better yet, move up to be clearly seen.
  • Use action words in the video preview thumbnail to encourage users to press play. Words such as “Start” and “Go” work well.
  • Snap a cool scene from your game or a nice UI from your app for the cover screen. The goal is to get users curious to press play and see what happens next.
  • If you’re offering a bundle with your app, always put captions on top on the screenshot. The lower part will be hidden below the fold, and you want to give users a glimpse of what’s below to capture their interest.
If you’re offering a bundle with your app, always put captions on top on the screenshot21

If you’re offering a bundle with your app, always put captions on top on the screenshot. (View large version22)

Summary: Roll Up Your Sleeves And Get Testing Link

So many apps, so little time. As the number of mobile users grows, new apps pop up daily. To make yours a success, be strategic about how you design the “shop window” for your app — the app’s page. A/B testing and optimization of the icon, screenshots and video preview will give you a better chance of higher conversions, a higher volume of organic downloads and a better return on your investment in user acquisition.

So what experience did you have with designing for the App Store or Google Play? What worked, and what failed, and why? Share your stories in the comments below! —Ed.

(da, ml, al)

Footnotes Link

  1. 1
  2. 2
  3. 3
  4. 4
  5. 5
  6. 6
  7. 7
  8. 8
  9. 9
  10. 10
  11. 11
  12. 12
  13. 13
  14. 14
  15. 15
  16. 16
  17. 17
  18. 18
  19. 19
  20. 20
  21. 21
  22. 22

↑ Back to top Tweet itShare on Facebook

Anya is a marketer based in San Francisco Bay Area. With half a decade of experience in digital marketing, she has worked in mobile app advertising and SaaS. Anya consults startups on growth and content strategy. You can follow her on Twitter @pratsaa

  1. 1

    Thanks for providing us with such a strategic lesson.

  2. 3

    CDN Mobile Solutions

    August 21, 2015 8:45 am

    Thank you so much Anna for providing such an important and detaild information about App store optimization and A/B testing. It is really helpful for me.

  3. 4

    Since Apple doesn’t currently support A/B testing, can you tell me more specifically how A/B testing was done for an iOS app? Is it done through an ad campaign or are you doing something outside of the store itself?

    • 5

      Anna Pratskevich

      August 22, 2015 8:29 am

      Apple indeed doesn’t support A/B testing functionality, but if you regularly buy ad traffic for your iOS app, you can split it 50/50 and run tests with external software tools like Splitmetrics.

  4. 6

    my play story

  5. 7

    Paolo - Chupamobile

    August 24, 2015 12:25 pm

    Nice Article! Some of our publishers make tests of their app icons and screenshots with the landing page. Then they submit the A-B test winners on iTunes, a free easy trick to A-B test the marketing materials of your App!
    We even released a free ASO guide and we should definitely add the A-B testing step there. Anyone is interested can download it for free here:

  6. 8


  7. 9

    We faced some problems when we where creating personalized screenshots. Hiring a freelancer for designing them is pretty expensive for us, also we didn’t had enough budget for that. Lucky for us, nowadays there are a lot of web services that can help you with that. For instance, we have chosen, because they have lots of themes (not only solid colors/gradients), and they don’t upscale one screenshot for different sizes and we could control almost every step. Other wise, we can also recommend “Launchkit screenshot builder”, but if it’s still not good for you, and you’re not in a rush — ol’ & good Fiverr can help.

  8. 10

    Hello and thank you for your article. There is something that I don’t understand with A/B testing screenshots & icon of an app on iOS. From what I understood, Splitmetrics creates a web page with each version. But then, you still need to download the app so my guess is that if you click on the install button you are redirected to the App Store. And there, you can only find the control version (original version) of the app. If I am correct, I am wondering how many users are NOT downloading the application because it looks like a different one. Would love to hear from someone on this. Thanks!

    • 11

      Anna Pratskevich

      November 18, 2015 5:24 am

      Hi Vincent, thank you for your comment. It’s a good question. You understand it correctly: Splitmetrics creates a web page with each version, and when you click install, you’re redirected to the store. Right now, this is the only way you can test and find the variation that converts better and ultimately brings more installs.
      Usually, we run experiments on a fairly low amount of traffic, and the main goal is to identify the best variation. If users install the app while we’re running the experiment – that’s great. Our stats show that more than 50% of users that are redirected to the AppStore proceed to install.
      The simple answer is yes, you do risk losing some installs because of the redirect to the store. And that’s okay, because it’s not the primary goal of the experiment. To be fair, about 30-40% of users drop off after they are asked their iTunes password to install the app. That’s why Apple saves your password for free app installs and authorizes the paid ones via Touch ID.

  9. 12

    Great article. We use for this, the service makes it really easy to run A/B. Love it so far


↑ Back to top