In our previous blog, we covered what mobile testing is all about, and a quick checklist to make sure you test all the right things. In this blog, we take a look at how mobile testing can be made easier and faster, i.e. mobile test automation.
A mobile application comprises of two parts:
From a testing perspective, it is relatively straight forward to be testing the APIs individually, as we can directly test the endpoints either manually using REST client or Postman, or by using automation tools like JMeter, SOAPUI etc. However, the key challenge that remains is testing whether the apps that are created and the APIs, work in tandem with the UI.
So what exactly is the challenge that we are talking about here?
Look around you and do a quick sampling. You will see at least 3 to 4 screen sizes, 2 major operating systems (if not 3), and 4-5 operating system versions. A quick calculation reveals approximately 40 possible screen-size-OS-version combinations just from a sample of 10 mobile phones. Now imagine the numbers if you were to create an application which is meant to work on everyone's phone!
The number of possible test combinations can be huge considering just the standard screen-sizes, the OS, and the major updates/releases. With a high frequency of updates to the application, it makes it very difficult for testers to test the application thoroughly in the short time spans available. And this is exactly why we need the mobile test automation, to help reduce the overhead of testing the features manually.
There are various tools that are available to test mobile applications. Let’s take a look at some of the most prevalent mobile test automation tools, their pros and cons, and their best use cases.
Appium: The Appium tool is built over Selenium, with a similar syntax, and has some additional capabilities which allow it to interact with the mobile. With its strong compatibility to Selenium, this is one of the preferred open source tools.
One of the biggest challenges that we come across when automating a mobile application test is finding the correct locators for the element. Appium comes with a built-in tool- UIAutomator, which helps easily find the locators. Appium also has a recorder that helps record the steps and use them in the form of a script.
In terms of support of applications, it has support for Android and iOS applications along with native and hybrid applications.
If you want to take a closer look at the tool, do check out our webinar on “Simplifying Mobile Test Automation with Appium”.
Calabash: The Calabash tool is another well-known tool in the open source community for mobile test automation. The tool is built atop CucumberJS and supports BDD type of automation.
The challenge of locators is handled quite well in Calabash. There is a query language that is used here, which returns all the elements that are present on the screen. This has been very well explained here.
Calabash too has good support for Android and iOS applications. The base scripts here are written in Ruby.
When comparing the two open source tools, it becomes very important to understand when to use a particular tool. With modern day automation, the preferred practice is to have a single suite for testing both the web and the mobile applications. And that becomes a major factor when choosing which mobile test automation tool to use.
Consider a simple example where we have a web application and a corresponding mobile app. If we have done the automation of the web component using Selenium + Java, Appium becomes an automatic choice. That’s because Appium is built over Selenium. However, if we have used Ruby for the web testing automation, we would prefer Calabash.
In terms of features and capabilities, both the tools are equally powerful and capable.
BDD is another factor that does have a role to play in the tool finalisation. With QA professionals moving towards a lot of BDD tests written by the product owners, it has become increasingly important that we perform mobile test automation with the same approach. This is where Calabash has an edge over Appium.
Pro-Tip: Considering the fact that almost all QA professionals have basic Selenium automation experience, I feel it’s best to start off mobile test automation with Appium. With a syntax that’s similar to Selenium, it greatly reduces the learning curve for a team that’s doing mobile testing for the first time.
Perfecto Mobile: This is more of a web tool that allows a user to record a script and play the script across different devices. The users are also allowed to extend the scripts by writing specific code that they feel may be needed.
The advantage of using Perfecto Mobile is the reduction in the cost of procuring actual devices. The devices are present in the Perfecto Mobile lab and each test is run on actual devices where the actions that we wish to perform are performed and streamed online.
Another advantage that comes with Perfecto Mobile is that you can create a test using any of the devices and the same can be played back on any other device. This makes sure that all the tests work fine on actual devices.
And finally, we can simulate interrupts without any additional steps. There are built-in functions available to simulate interrupt operations
SeeTestAutomation: This tool allows you to work on a level lower than Perfecto Mobile. Here, we can actually write code and perform the functions that we want. There is a recorder available to record simulated scenarios. Additionally, there is a SeeTestCloud which allows the tests to be run on actual hosted devices.
Interestingly, both these tools give more or less the same capabilities, including a mobile devices lab, recorders etc. The difference comes in the built-in features that come with the app. The tool of choice here is SeeTestAutomation, especially after looking at the default capabilities that it supports. These include:
Personally speaking, I think it’s best to choose one of the open source tools for mobile test automation. It gives you the liberty to grow and expand the tool as required. Meanwhile, you can use the paid platforms just for device support, and also get services like browserstack to access browsers and app platforms for testing.
But that’s just what I think. Rest is up to the automation testers to think and decide.