Once you’ve decided to automate testing on your project, you need to find the tool for test automation. Automated testing success depends on the right testing tools selection. It takes a lot of time to evaluate variety of automation tools available in the market today. Anyway you need to spend this time to find one or a few tools (for specific projects) to avoid changes (and additional project costs) later.
The scenario below allows you to make a choice for certain project as well as to perform investigation for your company to set up the mobile test automation:
Step 1. Which mobile operating systems are supported?
With the given system requirements specification you have to select tools that support target operating systems (iOS, Android, Windows Phone) and their different versions. It is also important to check the earliest and the latest supported versions of the operating system.
Step 2. Which type of mobile application is supported?
Once you got initial list of automation tools you have to check for the type of application that could be managed by selected tool. In most cases tools do not support native, hybrid and web application at the same time. So, it is quite possible that you’ll need to choose several of them.
Step 3. Is the source code required?
This is a significant point to consider as it is not always possible to get the source code for testing (contract agreement, access is limited due to security limitation, etc.). In some cases the solution is available like app package for iOS which is not the project source but it provides more capabilities for testing than application install file.
Step 4. Is application modification required?
Next obstacle in mobile test automation after source code usage necessity is requirement to modify the application to make it testable by the tool. Depending on the tool it is required to add 3rd party libraries into your project and build the test version of your product or modify the existing app version that is delivered for testing.
Step 5. In which way are the test scripts created?
The easiest Recording/Playback approach is the easiest and fastest way to start test automation on your project. The only one thing that is needed is script parameterization. But this approach is limited by tool capabilities and is not as much powerful as programmatically approach which allows you to use the power coding and capabilities of programming language that is used. The second is not as fast as first way but you are getting more effective and flexible approach instead.
Step 6. Which programming language is utilized?
Step 7. How the object recognition is done?
Application changes and their impact on object recognition in test scripts are always a challenge for automation engineers. The best way is to have unique object identification to reduce the impact of changes and simplify test scripts maintenance. You have to evaluate how object recognition is done using the given tool, if it is possible to get access to the specific objects (standard and complex Cocoa Touch controls, access to Cocoa Touch and Android API classes), check if it is easy to manage recognition properties in collected object library.
Step 8. Does the tool support data driven inputs?
To enable test execution with different set of data you’ll need to manage data sources. The availability of appropriate drivers brings you ability to work with flat files, spreadsheets and database storages.
Step 9. How detailed is the result logging?
To see the general results (whether PASS or FAIL) of testing is good but it is better to get specific information in the reports: the failed step of the scenario, gotten exception, screenshot of the fail. Ability to configure format of report is also a plus.
Step 10. Which options of the integration with other tools are available?
To build strong test automation framework with ability to support team work you need the complete integration of automated testing infrastructure components: IDE, test framework, continuous integration, configuration management, revision control, execution management, test management, defect tracking, report generation and other tools.
Step 11. What is the pricing model?
You will see free open source and proprietary test tools in your evaluation list. But you do not have to rely on the open source tools only. If you plan to use them, check how stable the tool evolution is and how fast those tools are upgraded to support latest changes in technologies. As for the proprietary solutions the price of the tools is one of key points in automation ROI calculation. You have to check the licensing model (per node, per company, period of license validness), find if addons, support and updates add additional cost. It is always possible to use several tools using trial license for evaluation by developing a simple set of automated tests.
And last but not the least – Ease of Use. Complexity matches ability to adopt new tools and programming talent of your QA persons.