Menu Search
Jump to the content X X
Smashing Conf New York

We use ad-blockers as well, you know. We gotta keep those servers running though. Did you know that we publish useful books and run friendly conferences — crafted for pros like yourself? E.g. upcoming SmashingConf Barcelona, dedicated to smart front-end techniques and design patterns.

The Basics Of Test Automation For Apps, Games And The Mobile Web

Mobile application ecosystems — let’s count Android and iOS here — are unbelievably dynamic, but they also suffer from both software and hardware fragmentation. This is especially true for Android, but fragmentation also exists in the iOS ecosystem1, as experienced with the rollout of iOS 8. As the latest version of iOS was released, many existing apps were made clumsy on updated devices.

Even the new iPhone 6 and iPhone 6 Plus have had not-so-typical issues2 for Apple devices. In addition, a significant proportion of users with older devices have very few options: essentially, buy new hardware (i.e. a new device) to get everything working well.

In the Android world, things are different. As OEMs launch new devices, software updates and customizations for their devices, application and game developers get serious headaches3 trying to keep their latest products up to snuff and fully compatible with all possible device variants. Making a certain app or game work only on high-end devices is out of the question, though. Why would a developer want to miss out on a significant chunk of potential users?

Automation enables simultaneous testing on hundreds of real devices4
Automation enables simultaneous testing on hundreds of real devices. (Image credit: Testdroid6753403733282517135) (View large version6)

Professional automated testing software is a solution to a common problem: how to produce high-quality, robust and reliable software with the ever-growing complexity of technology and under massive competitive pressure. Automated software testing is a cost-effective solution to this problem. Not to mention, it provides three business benefits:

  • increased testing efficiency,
  • increased testing effectiveness,
  • faster time to market.

This article walks through a sample use case for test automation and provides a downloadable example to get you started. Also, we’ll focus on different aspects of mobile test automation and explain how this relatively new yet popular topic can help mobile app and game developers to build better, more robust products for consumers. With the advanced example later in the article, we’ll show how image recognition can be used to test mobile games; specifically, we’ll run Appium’s test automation7 framework against Supercell’s Clash of Clan8 game to illustrate how image recognition can be built into the test-automation process.

Test Automation Is Perfect For Mobile App Development Link

Developing mobile applications is very different from developing PC software or even embedded software. Mobile development is meant to be agile9, and a lot of great tools and practices have been developed for that agility. However, doing something manually — such as testing an app — is never agile, which is why test automation has shown tremendous growth among app and game developers, speeding up their doings and yielding robust and better results.

To achieve compatibility between users, devices and the market, including test automation as a part of the agile development process10 is typical. Fortunately, a lot of tools are available, and test automation is a perfect fit for this process. For example, let’s say your typical development sprint is two weeks. You have daily standups and a lot of scrum activities, and you own internal policies that gear development to the final product. Test automation offers a significant value-add by enabling testing to be done in parallel — for example, as nightly test sessions. By the next morning, the tests will have been finalized and the results of the latest regression will be ready for review. Fixing an issue earlier will save a lot of time and get developers to finalize the product sooner; most importantly, it cumulates to better quality11, with fewer bugs.

Value that test automation brings to agile process.12
The value that test automation brings to the agile process. (Image credit: Testdroid6753403733282517135) (View large version14)

Test automation offers the possibility to test mobile apps instantly and effectively. Once tests have been automated, they can be executed quickly and repeatedly, again and again. In almost all cases, this is the most cost-effective method for regression testing15 of software products that have a long maintenance life. In fact, test automation of any mobile app is the best way to increase the effectiveness, efficiency and coverage of the testing process. The true benefit of automation comes not only from the repeatability of tests, but also from the ability to execute tests that probably could not even be performed manually.

Things To Consider When Automating Mobile App Testing Link

Let’s look at how test automation (as opposed to manual testing) can improve the development process, add value and speed up development.

Costs and Assets Link

Regardless of whether you go for manual or automated testing, you’ll need the following assets and resources (all of which cost money): time, people, infrastructure, tools and training. Depending on the size of the project and the application, test automation will quite obviously provide a good return on investment. For example, once test cases have been created, automated tests can be run over and over again at no additional cost, and these can be more rapidly completed than any manual tests. Automated software testing can reduce the time required to run repetitive tests from weeks to hours. This is a significant time-saving that translates directly into cost-saving.

Continous integration together with mobile app testing16
Continous integration together with mobile app testing. (Image credit: Testdroid6753403733282517135) (View large version18)

Integrated App Development And Testing Cycle Link

Increasing efficiency and productivity with automation actually starts with adopting a new mindset. Software tests have to be repeated often during all development cycles to ensure the best possible quality of an application. Every time source code is modified19, software tests should be repeated. For each release, your software should be tested on all supported variants of operating systems and all configurations of hardware. Manually repeating these tests would be costly and time-consuming. For example, running comprehensive tests manually on all variants of Android and on actual devices would take a lot of time.

Tools and Technology: Test Automation Frameworks Link

To get the most out of your efforts and maximize testing coverage, select the most robust and most cross-platform20 method. That automated methods can be used both to validate requirements and to reduce costs through automated test-case generation is well known. However, the full automatization of large software entities also comes with a cost that many companies haven’t been ready to pay for. Historically, one of the reasons has been a concern over the lack of adequate integration with well-established development life cycles.

Test Coverage and Reusability: Open Standards Mean No Vendor Lock-In Link

Automated testing can increase the depth and scope of tests and significantly improve software quality. Lengthy and thorough tests — often not doable with manual testing — can be run automatically. Ideally, test cases should have full access to an application and test all aspects of it21 — memory contents, data tables, file contents and internal program states — to determine whether the product behaves as expected. Automated tests can easily execute thousands of complex test cases during every test run, providing coverage that is simply not possible with manual testing. Developers, freed from repetitive manual tests, will have more time to create new automated test cases and build more compelling features (and then more test cases).

Time to market for mobile web22

(View large version23)

Improve Effectiveness and Finalize Sooner Link

In a nutshell, professional automated testing software is a solution to a common problem: how to produce high-quality, robust and reliable software with the ever-growing complexity of technology and under massive competitive pressure. Automated testing improves business results in three ways: greater testing efficiency, greater testing effectiveness and a shorter time to market.

Different Ways to Automate Mobile Testing Link

In general, there are three ways to automate the testing of mobile apps.

Handwritten Test Scripts Link

Typically, this is the best choice when you know what you’re doing and when you have programming-capable people doing the test scripts. Plenty of options are available for test automation frameworks, tools and integration — both commercial and open-source offerings. Having your engineers write all of the test scripts will take up some time and tie up resources, but it will get you exactly what you want: well-structured, thorough scripts that test precisely the aspects of your app that you want to test.

Manual versus automated testing for mobile apps24

Manual versus automated testing for mobile apps. (Image credit: Testdroid6753403733282517135) (View large version26)

Record-Playback Approach Link

This approach is less error-prone because nothing needs to be written in code, but it is typically more limited in functionality. Tests can be quickly recorded and then played back again and again against different OS versions and device configurations. These tests focus on user interactions and user-driven activities. Some things might fall beyond a test’s scope (such as integration with other technologies and compatibility with other software).

Automatic Test Exercisers Link

Automatic test exercisers provide a great way to smoke-test applications. No specific tests are needed; rather, the focus is on testing user-interface logic (such as opening menus, clicking buttons, swiping and multi-gesture actions). Automatic test exercisers yield the least exact results but provide quick feedback on each iteration of an app.

Focus Areas In Testing For Mobile Apps And Games Link

Different ways to automate mobile app testing27

Different ways to automate mobile app testing. (Image credit: Testdroid6753403733282517135) (View large version29)

User Interface and Functionality Link

A user interface and its overall functionality will directly affect how successful your app or game30 will be. These two things, which encompass visual appeal and gameplay, are the most important things to get right — and you must ensure that device fragmentation doesn’t break any of these. Various things in the UI need to be tested:

  • UI layouts and elements
    Games especially are typically targeted at a high number of different screen resolutions and screen types. Regression testing should be done each and every time the UI’s layout changes to ensure that the game works.
  • Menu structure and functions
    Testing menu structures, functionality and behavior can be automated with instrumentation and the help of different test-automation frameworks.
  • Screen orientation
    Surprisingly, so many apps and games out there get this wrong. If a screen’s orientation changes during an interaction, for example, what happens? What is supposed to happen? Does the app or game work well in both landscape and portrait modes?
  • Screen resolution
    A lot of screen resolutions exist, especially on Android, and auto-scaling will usually help developers. However, test your game across these resolutions to ensure that the graphics do not stretch.

Graphics Performance Link

Performance needs to be consistent across all device variants31 among your users. Because of this, test on as many real devices as possible. To determine how well your app or game responds to various levels of usage, including performance and battery usage, consider creating tests that last for hours. To determine whether your game runs effectively under a heavy load for a long time, run load (or stress) tests. These performance tests will measure, for example, how responsive your game is on real devices.

Performance testing of CPU load and memory consumption32
Performance testing of CPU load and memory consumption. (Image credit: Testdroid6753403733282517135) (View large version34)

Usability and User Experience (i.e. Good Entertainment) Link

Testing usability, navigation flow and user experience simply cannot be done on a desktop with a mouse and keyboard. So, forget emulators and use only real devices35. And to test how usable and entertaining your app is, consider these two important things:

  • User interaction and responsiveness
    Testing performance is critical because this will make make or break the user experience. Performance lag, for example, is easy to expose with real devices.
  • Background events
    Interruptions, battery consumption and the effect of battery chargers on overall performance and usage all have a significant impact on the user experience — and entertainment value.

Multi-User Features Link

Nowadays, multi-user support is common in both apps and games. Testing multi-player capabilities is important and is naturally more challenging, requiring real users to measure performance. A typical case is a game communicating with the back-end server. In this case, connectivity is essential, to synchronize the back end with devices that need to get information about the gameplay. You should test a ton of different scenarios, many of which could severely affect the game’s experience, resulting in negative feedback and the game being uninstalled by users.

Social Integration Link

Testing social integration in mobile apps36

Testing social integration in mobile apps. (Image credit: Testdroid6753403733282517135) (View large version38)

Integration with social networks is another important factor. Being able to share something across an ecosystem, with friends or just with oneself, is essential in many apps. Test this thoroughly with real Android and iOS devices, with different OS versions and different device configurations, to assess functionality and ease of use.

Security and Liabilities Link

Nearly all developers use some open-source components in their apps. This practice is widely accepted and highly recommended because it offloads the development of code for non-core functionality. However, identifying vulnerabilities and licensing restrictions with third-party code is often neglected by developers.

Breakdown: Android Test Automation Frameworks Link

Comparison of test automation frameworks39
Comparison of test automation frameworks. (Image credit: Testdroid6753403733282517135) (View large version41)

Robotium42 is an Android test automation framework that fully supports native and hybrid applications. Robotium makes it easy to write powerful and robust automatic black-box UI tests for Android applications. With the support of Robotium, test case developers can write function, system and user acceptance test scenarios, spanning multiple Android activities.

UIautomator5043, by Google, provides an efficient way to test UIs. It creates automated functional test cases that can be executed against apps on real Android devices and emulators. It includes a viewer, which is a GUI tool to scan and analyze the UI components of an Android app.

Espresso44, by Google, is a pretty new test automation framework that got open-sourced just last year, making it available for developers and testers to hammer out their UIs. Espresso has an API that is small, predictable, easy to learn and built on top of the Android instrumentation framework45. You can quickly write concise and reliable Android UI tests with it.

Calabash46 is a cross-platform test automation framework for Android and iOS native and hybrid applications. Calabash’s easy-to-understand syntax enables even non-technical people to create and execute automated acceptance tests for apps on both of these mobile platforms.

And then there’s Appium. OK, let’s get into this one.

Appium: Executing Tests On Real Devices On A Cloud Service Link

In a nutshell, Appium47 is a mobile test automation framework (and tool) for native, hybrid and mobile-web apps for iOS and Android. It uses JSONWireProtocol48 internally to interact with iOS and Android apps using Selenium’s WebDriver49.

In its architecture, Appium is an HTTP server written in Node.js that creates and handles multiple WebDriver sessions. Appium starts tests on the device and listens for commands from the main Appium server. It is almost the same as the Selenium server that gets HTTP requests from Selenium client libraries.

In fact, Appium is a pretty good choice for both apps and games because, in many cases, apps and games tend to be identical (or at least very similar) on both platforms, Android and iOS — and so the same test script can be applied to both. Another significant benefit of Appium is that users can write tests using their favorite development tools, environment and programming language, such as Java, Objective-C, JavaScript, PHP, Ruby, Python or C#, among many others.

Appium enables users to execute tests on mobile devices regardless of OS. This is possible because the Appium framework is basically a wrapper that translates Selenium’s WebDriver commands to UIAutomation (iOS), UIautomator5043 (Android, API level 17 or higher) or Selendroid51 (Android, API level 16 or lower) commands, depending on the device’s type.

For Android, this is how Appium compares to other test automation frameworks:

Family tree of Android Test automation frameworks52
Family tree of Android test automation frameworks. (Image credit: Testdroid6753403733282517135) (View large version54)

Android test suites are based on JUnit55. In addition, Android provides an architecture and fully integrated testing capabilities with its standard tools56, which help developers to test at every level, from unit to framework. Android instrumentation57 is a set of control methods in the Android system. These methods control an Android component independently of its normal life cycle.

One of the best things about Appium is that, despite sounding architecturally complex, it actually isn’t — at all. For developers, it provides support for various programming languages, freedom from having to select tools, compatibility across the most important platforms (Android and iOS), freedom from having to install and configure devices to test and more.

If you are familiar with Selenium, then you’ve got Appium covered. An Appium test is pretty much the same as a Selenium test: They use the same WebDriver, and DesiredCapabilities58 is used the same way. Configuring an application to run on Appium has a lot of similarities to Selenium — for example, those DesiredCapabilities. We’ll configure a sample test later in this article.

Also, Appium includes a component called the inspector. This inspector enables a host of functionality — for example, showing all of those UI elements in the application and enabling basic recording and playback.

However, you may not need the inspector because everything can be done in the code. The example later provides all of the scripts and instructions step by step, without using it.

More information about Appium can be found in “Getting Started With Appium59.”

Setting Up The Environment Link

To get started, first download our Appium example60. The example is available in Python, Java and Ruby. Then, depending on which programming language you will be using, select the appropriate client library61. Appium’s documentation62 provides all instructions, help and additional material.

Mac OS X and Linux Link

Ensure that Python 2.7.x or later is installed (it’s preinstalled on Mac OS X):

$ python --version

Install Python if it’s not already (Linux only):

$ sudo apt-get install python2.7

Check whether Python’s “pip” module is already installed:

$ pip --version

If you’re on Mac OS X, install pip if it’s not already:

$ curl https://raw.githubusercontent.com/pypa/pip/master/contrib/get-pip.py > get-pip.py
$ sudo python get-pip.py
$ rm get-pip.py

If you’re on Linux, install pip if it’s not already:

$ sudo apt-get install python-pip

Install the Selenium module for Python:

$ sudo pip install selenium

Verify that Selenium is installed:

$ pip freeze | grep -i selenium

Windows Link

Ensure that Python 2.7.x or later is installed:

$ python --version

If it’s not installed, download and run the setup from Python’s download center63. To add Python environment variables, go to “System properties” → “Advanced System Settings” → “Environment Variables” → “System Variables” → “Edit Path,” and then insert C:\Python27\;C:\Python27\Scripts\ at the end (assuming you have installed Python in the default location). Make sure to restart the command prompt to bring new environment variables into effect.

Check whether Python’s pip module is already installed:

$ pip --version

Install pip if it’s not already:

$ curl https://raw.github.com/pypa/pip/master/contrib/get-pip.py > get-pip.py
$ python get-pip.py
$ del get-pip.py

Install Python’s Selenium module:

$ pip install selenium

Running The First Tests Link

Download the Appium sample test script and sample app64. The sample package includes the APK for Android, the IPA for iOS and one sample test script. In addition, there are three samples:

  • appium_example_ios.py
  • appium_example_android.py (for Android API level 17 and above)
  • appium_example_selendroid.py (for Android API level 16 and below)

Step 1: Create Account and Upload the APK Link

Create an account.65 Note that this service provides a freemium option, and you don’t need a plan to complete these examples using real devices on a cloud service. If you want to access hundreds of device models, then different plans are available.

Before proceeding with running the test script, you’ll need to upload the APK or IPA to Testdroid’s cloud service via HTTP POST. Let’s try that using cURL.

For Mac OS X and Linux:

$ curl -s --user username@example.com:password -F myAppFile=@"/absolute/path/to/app/SampleApp-iOS.ipa" "http://appium.testdroid.com/upload"

For Windows: Link

$ cd C:\path\to\directory\containing\the\application
$ curl -s --user username@example.com:password -F myAppFile=@"SampleApp-iOS.ipa" "http://appium.testdroid.com/upload"

Upon successful upload, you will get a JSON response with status:0 and an unique identifier for the uploaded app:

{
   “status”: 0,
   “sessionId”: “abcdefg-1234-5678-abcd-111122223333”,
   “value”: {
      “message”: “Uploads Successful”,
      “uploadCount”: 1,
      “rejectCount”: 0,
      “expiresIn”: 1800,
      “uploads”: {
         “myAppFile“: “abcdefg-1234-5678-abcd-111122223333/SampleApp-iOS.ipa“
      },
      “rejects”: {}
   }
}

Step 2: Set Credentials and Other Parameters Link

Open the test script, appium_sample_ios.py, in any text editor. Set screenshotDir to the path where you want those screenshots to be saved on your machine. Set your credentials on testdroid_username and testdroid_password in DesiredCapabilities. Set the myAppFile identifier against testdroid_app in DesiredCapabilities. It should look like this:

desired_capabilities_cloud = {
   ‘testdroid_username’:    testdroid_username,
   ‘testdroid_password:     testdroid_password,
   ‘testdroid_project:      ‘Appium iOS Project 1’,
   ‘testdroid_target:       ‘iOS’,
   ‘testdroid_description:  ‘Appium project description’,
   ‘testdroid_testrun:      ‘Test Run 1’,
   ‘testdroid_device:       testdroid_device,
   ‘testdroid_app:          ‘sample/SampleApp-iOS.ipa’,
   ‘testdroid_platformName: ’iOS’,
   ‘testdroid_deviceName:   ‘iPhone device’,
   ‘testdroid_bundleId:     ‘com.bitbar.testdroid.SampleApp-iOS’
}

Step 3: Run the Test Script Link

Execute the following command:

$ python sample-appium_ios.py

The console’s output should look like this:

Console output after test script66
Console’s output after test script. (Image credit: Testdroid6753403733282517135) (View large version68)

Step 4: Get Results From Cloud Link

The screenshots will be available locally on your machine in the directory that you specified in step 2. Log into your cloud service and navigate to the project’s name, as defined in the _project attribute of DesiredCapabilities, to access the following log files:

  • Appium’s server log,
  • Logcat and instruments log.

Advanced Example: Using Appium For Mobile Game Testing With Image Recognition Link

In this example, We’re using Supercell’s popular mobile game Clash of Clans. It’s a fantastic game and I’ll bet many of you have played it, so you should be pretty familiar with its look and gameplay. This example does basic clicks through the game’s tutorial.

## Example script that tests Clash of Clans tutorial first steps
## Works on different resolutions, both iOS and Android

import unittest

from time import sleep
from AppiumCloudTest import AppiumCloudTest, log
from selenium.common.exceptions import WebDriverException

class ClashOfClansTest(AppiumCloudTest):
   def setUp(self):
      # AppiumCloudTest takes settings from environment variables
      super(ClashOfClansTest, self).setUp()

   def test_tutorial(self):
      driver = self.get_driver() # Initialize Appium connection to device

      sleep(10) # Wait for game to load

      # Use this to get detected screen hierarchy
      # print self.driver.page_source

      # Dismiss the in-app purchases dialog if it shows
      okayButton = None
      if self.isAndroid():
         try:
            okayButton = driver.find_element_by_id('button3')
            okayButton.click()
            sleep(5)
         except WebDriverException:
            log("There was no in-app purchases dialog")
      else: # iOS
         self.driver.implicitly_wait(5) # Wait only 5 seconds to find it
         try:
            okayButton = driver.find_element_by_accessibility_id('Okay')
            okayButton.click()
            # No need to sleep here
         except WebDriverException:
            log("There was no in-app purchases dialog")
         self.driver.implicitly_wait(30)

      # Cancel iOS Game Center login
      if self.isIOS():
         #print self.driver.page_source
         try:
            self.driver.implicitly_wait(5)
            cancelButton = driver.find_element_by_accessibility_id('Cancel')
            log("Cancelling iOS Game Center login…")
            cancelButton.click()
            sleep(2)
         except WebDriverException:
            log("The Game Center login was not displayed")
         self.driver.implicitly_wait(30)

      self.screenshot("welcome-chief")

      # Check that a goldmine is on screen
      rect = self.find_image("queryimages/tutorial_goldmine.png",
         screenshot_match="screenshots/goldmine_match.png")
      self.assertIsNotNone(rect,
         "There should be a goldmine on screen at beginning of tutorial")
      log('Gold mine found at %s %s! Continuing…' % (rect[0], rect[1]))

      # Dismiss the bubbles
      self.tap_middle()
      sleep(2) # second blabla
      self.tap_middle()
      sleep(2) # Goblin appears
      self.tap_middle()
      sleep(1)

      # Go to shop
      # NOTE: tap_image does also assert, fails test if target not recognized
      self.tap_image("queryimages/shopbutton.png")
      sleep(1)

      # Buy cannon
      self.screenshot('cannon')
      self.tap_image("queryimages/cannon.png")
      sleep(2)

      # Place the cannon
      self.screenshot('place_the_cannon')
      self.tap_image("queryimages/place_the_cannon.png", width_modifier=0.75)
      sleep(2)
      self.screenshot('finish_now')
      # Use gem to finish right away
      self.tap_image("queryimages/finish_now.png")
      sleep(3)
      # Bring it on!
      self.screenshot('bring_it_on')
      self.tap_image("queryimages/bring_it_on.png", height_modifier=0.75)
      sleep(10)
      self.screenshot('battle')
      sleep(10)
      self.screenshot('end_of_battle')

      # To be continued…

if __name__ == '__main__':
   unittest.main()

Let’s look some of the stages in this test script. The test_tutorial contains the following steps:

  1. It first figures out whether the test executes on Android (self.isAndroid()) or iOS. As you can see, it looks for UI content differently. On Android, it tries to find a button by using an element’s ID (element_name) and on iOS by using an accessibility ID with a description (“Okay”). The same check happens for iOS’ Game Center login.
  2. Screenshots are taken at various steps and stored in files entered as a parameter in a function call.
  3. A check is done to see whether “goldmine” appears on screen by comparing two PNG files using a self.find_image call. If these pictures match (i.e. “goldmine” appears on screen), then the script continues executing the game’s tutorial.
  4. The tutorial proceeds with the following steps:
    1. Go to shop.
    2. Buy cannon.
    3. Place the cannon.

    The information about all three of these items is stored in those PNG files: shopbutton.png, cannon.png and place_the_cannon.png.

  5. Finally, the tutorial finishes and the battle starts. After the battle, the application is closed.

The video below shows how the test executes at each step. For the video, I’ve used three devices: one iOS (iPhone 4S) and two Android phones (Samsung Galaxy S3 Mini and HTC One X). You can also watch the video on YouTube69.

How Is Image Recognition Used Here?

The example shown uses image recognition (i.e. template matching70) to identify which features — basically, pixels and graphic content — are shown on screen and to compare the two pictures to each other. The algorithm built to recognize images was used on real devices and on two different platforms (Android and iOS), and it used a single test script for both platforms. This sort of image comparison is even very handy for recognizing UI elements and graphics that are resized and/or rotated.

Let’s say the template image has some distinctive features, such as text that can be easily abstracted from the background content. In this case, feature-based recognition can be used. In our example, if the “Button” text has been resized or rotated (or otherwise transformed), we can quickly and easily identify this and take further action.

The following functions explain the approach of comparing images:

def find_image(self, queryimage_file, screenshot_match=None):
   log('Trying to find %s in current screen' % queryimage_file)

   fd,screenshot_file = tempfile.mkstemp()
   try:  
      os.close(fd)
      # print screenshot_file
      head, tail = os.path.split(screenshot_file)
      self.screenshot(tail, path=head)
      head,tail = os.path.split(queryimage_file)
      screenshot_match='%s/%s-match.png' %
         (self.screenshot_dir, tail.split('.')[0])
      return ImageFinder.findImages(queryimage_file,
         "%s.png" % screenshot_file,
         scale_tolerance=0.9,
         screenshot_match=screenshot_match)
   finally:
      os.remove(screenshot_file)
def screenshot(self, name, path=None):
   full_path=None
   if path:
      full_path='%s/%s.png' % (path,name)
   else:
      full_path='%s/%s.png' % (self.screenshot_dir, name)

   try:
      self.get_driver().save_screenshot(full_path)
   except WebDriverException: # for iOS, sometimes times out, so retry!
      #log("Failed taking screenshot %s - retrying" % full_path)
      self.get_driver().save_screenshot(full_path)

   #
   width,height = get_image_size(full_path)
   if(height > width):
      if self.platform_name == 'Android':
         #log("Rotating screenshot 270 degrees")
         os.system('sips -r 270 %s >/dev/null 2&>1' % full_path)
      else:
         #log("Rotating screenshot 90 degrees")
         os.system('sips -r 90 %s >/dev/null 2&>1' % full_path)
def tap_image(self, query_image,
   width_modifier=0.5, height_modifier=0.5, retries=2):

   retries_left = retries
   rect = None
   while retries_left > 0 and not rect:
      rect = self.find_image(query_image)
      retries_left = retries_left - 1
   image_name = os.path.split(query_image)[1]
   self.assertIsNotNone(rect, "Image %s is on screen" % image_name)
   x = rect[0]+rect[2]*width_modifier
   y = rect[1]+rect[3]*height_modifier
   log('%s button found at %s %s (%sx%s), tapping at %s %s' %
      (image_name, rect[0], rect[1], rect[2], rect[3], x, y))

   self.tap(x,y)
def tap(self, x, y):
   log('Tapping at %s,%s' % (x,y))
   driver = self.get_driver()
   if self.isAndroid():
      if self.isSelendroid():
         touch_actions = TouchActions(driver)
         touch_actions.tap_and_hold(x, y).release(x,y).perform()
      else:
         action = TouchAction(driver)
         action.press(x=x, y=y).release().perform()
      else: # iOS
         action = TouchAction(driver)
         # TODO: Temporary hack
         # On iPhone 4S = res. 480 x 320 for taps but screenshots are 960 x 640
         action.press(x=int(x/2), y=int(y/2)).release().perform()

Conclusion Link

Testing is crucial to ensuring success in the highly competitive landscape of mobile apps and games71. But even poorly planned testing can take up 20 to 50% of your total development effort, in which case it would also account for the single biggest cost in your budget. To ensure that testing is extremely efficient, covering the breadth of today’s mobile ecosystems and device models, the best option is an online cloud-based service.

If you only start thinking about testing a few days before the app hits the market, it’s too late. You’ll need to test a wealth of elements, data and functionality from day one. Here are some things to consider in making testing a part of your development process:

  • Plan carefully: Automate generic processes as much as possible.
    When you’re building a mobile app, a well thought out strategy is critical, a great user experience and design are paramount, and solid development and testing are fundamental. Many aspects of testing can be automated, and this automation will increase the depth and scope of your testing and significantly improve the app’s quality. Ideally, test cases should have full access to the application and test all aspects of it — memory contents, data tables, file contents and internal program states — to determine whether the product performs as expected.
  • Your app will change during development: The same goes for testing.
    Many things about your app will change as you’re creating it: the user interface, graphics, functionality, language support, the privacy policy, the use of external resources and much more. Even if 10% of your code changes72 or is added to the app, you’ll still need to test 100% of the features. Manual testing can’t keep up with this, so your best option is to build all of your test cases for new features. Then, when a feature is added, all features will be automatically tested. Building your tests to be maintainable over the various development phases of your app is essential.
  • Choose a testing technology and provider you can grow with.
    If you already have an app on the market and are looking to create a similar one, then select a technology73 and vendor that meet your needs. For example, building your test cases according to a certain method or framework will enable you to reuse those test cases for your new application — at least to some extent. So, choose a technology and vendor that is able to handle your needs as your product scales up and as your testing has to cover new geographical areas and even support new platforms (for example, going from Android to iOS).
  • Test automation is available 24/7.
    Automation will reduce the time it takes to test new features and even the app itself by running 24/7.
  • Use a cloud-based platform for truly global reach.
    With an online cloud-based service, you’ll get instant access to hundreds of real Android devices74. Especially with Android, having access to devices that are used in volume holds significant value. Running automated tests on these devices is easy and fast and provides all of the information you’ll need, preprocessed, summarized and in full detail.

What are your thoughts on test automation for mobile? Please let us know in the comments section!

(da, ml, al)

Footnotes Link

  1. 1 http://www.zdnet.com/ios-8-usage-figures-showing-signs-of-stress-fragmentation-after-initial-uptick-7000034389/
  2. 2 http://www.gottabemobile.com/2014/10/22/iphone-6-plus-problems-fixes/
  3. 3 https://www.smashingmagazine.com/2014/10/02/what-every-app-developer-should-know-about-android/
  4. 4 https://www.smashingmagazine.com/wp-content/uploads/2014/12/01-device-rack-opt.jpg
  5. 5 http://www.testdroid.com/
  6. 6 https://www.smashingmagazine.com/wp-content/uploads/2014/12/01-device-rack-opt.jpg
  7. 7 http://appium.io/
  8. 8 http://www.supercell.net/games/view/clash-of-clans
  9. 9 http://www.gartner.com/newsroom/id/2823619
  10. 10 http://testdroid.com/testdroid/5876/test-early-test-often-testing-as-part-of-your-app-development
  11. 11 http://autotestcentral.com/how-test-automation-can-help-to-achieve-5-stars/285
  12. 12 https://www.smashingmagazine.com/wp-content/uploads/2014/12/02-value-diagram-opt.jpg
  13. 13 http://www.testdroid.com/
  14. 14 https://www.smashingmagazine.com/wp-content/uploads/2014/12/02-value-diagram-opt.jpg
  15. 15 http://en.wikipedia.org/wiki/Regression_testing
  16. 16 https://www.smashingmagazine.com/wp-content/uploads/2014/12/03-testing-in-ci-process-opt.png
  17. 17 http://www.testdroid.com/
  18. 18 https://www.smashingmagazine.com/wp-content/uploads/2014/12/03-testing-in-ci-process-opt.png
  19. 19 http://testdroid.com/testdroid/5851/increase-efficiency-and-productivity-with-test-automation
  20. 20 http://en.wikipedia.org/wiki/Cross-platform
  21. 21 http://agiledata.org/essays/tdd.html
  22. 22 https://www.smashingmagazine.com/wp-content/uploads/2014/12/04-time-to-market-opt.jpg
  23. 23 https://www.smashingmagazine.com/wp-content/uploads/2014/12/04-time-to-market-opt.jpg
  24. 24 https://www.smashingmagazine.com/wp-content/uploads/2014/12/05-different-ways-to-automate-opt.jpg
  25. 25 http://www.testdroid.com/
  26. 26 https://www.smashingmagazine.com/wp-content/uploads/2014/12/05-different-ways-to-automate-opt.jpg
  27. 27 https://www.smashingmagazine.com/wp-content/uploads/2014/12/06-different-ways-to-test-opt.jpg
  28. 28 http://www.testdroid.com/
  29. 29 https://www.smashingmagazine.com/wp-content/uploads/2014/12/06-different-ways-to-test-opt.jpg
  30. 30 http://www.appannie.com/indexes/all-stores/rank/overall/
  31. 31 http://www.androidcentral.com/devices
  32. 32 https://www.smashingmagazine.com/wp-content/uploads/2014/12/07-performance-testing-opt.png
  33. 33 http://www.testdroid.com/
  34. 34 https://www.smashingmagazine.com/wp-content/uploads/2014/12/07-performance-testing-opt.png
  35. 35 http://testdroid.com/testdroid/5901/rely-only-on-real-emulators-vs-devices
  36. 36 https://www.smashingmagazine.com/wp-content/uploads/2014/12/08-social-integrations-opt.jpg
  37. 37 http://www.testdroid.com/
  38. 38 https://www.smashingmagazine.com/wp-content/uploads/2014/12/08-social-integrations-opt.jpg
  39. 39 https://www.smashingmagazine.com/wp-content/uploads/2014/12/09-frameworks-opt.png
  40. 40 http://www.testdroid.com/
  41. 41 https://www.smashingmagazine.com/wp-content/uploads/2014/12/09-frameworks-opt.png
  42. 42 https://code.google.com/p/robotium/
  43. 43 http://developer.android.com/tools/help/uiautomator/index.html
  44. 44 https://code.google.com/p/android-test-kit/wiki/Espresso
  45. 45 http://developer.android.com/tools/testing/testing_android.html#Instrumentation
  46. 46 http://calaba.sh/
  47. 47 http://appium.io/
  48. 48 https://code.google.com/p/selenium/wiki/JsonWireProtocol
  49. 49 http://docs.seleniumhq.org/projects/webdriver/
  50. 50 http://developer.android.com/tools/help/uiautomator/index.html
  51. 51 http://selendroid.io/
  52. 52 https://www.smashingmagazine.com/wp-content/uploads/2014/12/10-framework-families-opt.png
  53. 53 http://www.testdroid.com/
  54. 54 https://www.smashingmagazine.com/wp-content/uploads/2014/12/10-framework-families-opt.png
  55. 55 http://developer.android.com/tools/testing/testing_android.html#JUnit
  56. 56 http://developer.android.com/tools/testing/testing_android.html
  57. 57 http://developer.android.com/tools/testing/testing_android.html#Instrumentation
  58. 58 https://code.google.com/p/selenium/wiki/DesiredCapabilities
  59. 59 http://appium.io/slate/en/tutorial/ios.html?ruby#getting-started-with-appium
  60. 60 https://github.com/bitbar/testdroid-samples/tree/master/appium/sample-scripts
  61. 61 http://appium.io/downloads.html
  62. 62 http://appium.io/introduction.html
  63. 63 https://www.python.org/downloads/
  64. 64 https://github.com/bitbar/testdroid-samples/tree/master/appium/sample-scripts/python
  65. 65 http://testdroid.com/sm-appium
  66. 66 https://www.smashingmagazine.com/wp-content/uploads/2014/12/11-console-output-opt.jpg
  67. 67 http://www.testdroid.com/
  68. 68 https://www.smashingmagazine.com/wp-content/uploads/2014/12/11-console-output-opt.jpg
  69. 69 https://www.youtube.com/watch?v=Q5U3HwpD1Ec
  70. 70 http://en.wikipedia.org/wiki/Template_matching
  71. 71 http://www.appannie.com/indexes/all-stores/rank/overall/
  72. 72 http://testdroid.com/testdroid/6186/strive-for-hermetic-but-never-compromise-integrity-of-app
  73. 73 https://www.smashingmagazine.com/2014/01/10/four-ways-to-build-a-mobile-app-part2-native-android/
  74. 74 http://insights.wired.com/profiles/blogs/the-best-advice-for-app-developers-skip-emulators
SmashingConf New York

Hold on, Tiger! Thank you for reading the article. Did you know that we also publish printed books and run friendly conferences – crafted for pros like you? Like SmashingConf Barcelona, on October 25–26, with smart design patterns and front-end techniques.

↑ Back to top Tweet itShare on Facebook

Ville-Veikko Helppi is a Technical Product Manager at Bitbar Technologies and manages the popular Testdroid family of products used in making mobile apps and games super robust. To learn more, follow him at Twitter or visit at website.

  1. 1

    Such a great article ! Thanks you so much for the detailed information and your effort in writing it. I am investigating Appium and this post will help a lot !

    23
  2. 2

    Do you have any suggestions for similar tools for Windows Phone?

    5
  3. 4

    Having both an Android and now an iPhone shortly after, I have noticed the difference when it comes to certain apps that one does perfectly that the other does not. I know when I used to have an Android there were certain apps that had not been created yet on that type of device yet that the Apple device had already had for a long time before. I feel as though Android is always right behind Apple when it comes to the latest and greatest, but Android is working its way up there with all the testing for their apps. I’m not very familiar with creating applications, but I have a few friends that are very passionate about it and just blow my mind with the different types of applications that are on their way to be the next most used app. This article was most definitely intriguing for me since I do not know what it takes to test and create and application, but if i ever would this would be the first place I’d stop and grab tips from!

    1
  4. 5

    Hi Ville-Veikko Helppi,

    Thanks for the article.
    A couple of things to mention:
    1. At the time you published this post the latest Expresso for Android release had already Web features support, as well as latest Android API levels support.
    2. Having the conditional logic in test is not welcomed – will it be better to distinguish on which platform are you before you run the tests and then trigger tests for specific platform? As a bonus you get cleaner and smaller code.

    Actually I am the fan of platform specific test frameworks like Espresso and KIF. Using them you get really fast test execution and much cleaner and compact code. No more sleeps :) Of course in ideal case you have to have the source code access.

    Good luck!

    1
  5. 6

    Fernando Castillo

    February 16, 2015 11:18 am

    Excellent post on business, users requirement and motion sensors. Business strategies are getting much more advanced than using older tactics. Thanks for offering thins kind of informative posts to us.

    1
  6. 7

    Hi Ville-Veikko Helppi,

    I am new to Mobile automation. Planning to use Appium for it.
    I need some idea about triggering test scripts in Appium.
    1. Need to trigger automation scripts when mobile OS ( IOS / Android ) is upgraded.
    2. Need to trigger automation scripts when my app is upgraded.

    1
  7. 8

    Pei Wei-hong

    July 28, 2015 2:00 pm

    Hi Ville-Veikko Helppi,

    Thanks for usefull article. So I have a confusion is which way should I choose to automate mobile testing (Handwritten Test Scripts or Record Playback or Automatic Test Exercisers)? As you wrote about Appium, handwritten test scripts is only choice for automating mobile testing, isn’t it?

    0
  8. 9

    Very informative, covers all aspects that’s what I was looking for Thank You.

    0
  9. 10

    Hello my dear brother
    After greeting

    I want to buy a game like Clash of Clans

    To earn money from them

    Is it possible to settle this work
    ؟

    0

↑ Back to top