Automation ROI.

“We must have an automation strategy”

this sounds familiar to most people involved in software development QA.

But in the Mobile Apps lifecycle does automation truly give a Return On Investment?

In most agile Mobile App projects sprints require that new functional and UI tests are manually tested as soon as the code is built – to maintain the project velocity. During these sprints how should the development of automation solutions be accommodated ?  Typically the QA resources will have moved on to the next sprint without having time to develop and ‘tune’ automation scripts.

True , a  parallel resource pool could be employed to develop the automation scripts as the test case numbers grow on the project through the sprints.

This is an additional loading on the project resources which is never popular.

Add to this the cross platform challenges of automation for Mobile Apps.  Other constraints such as no jailbroken/rooted phones, No instrumented builds make finding a single automation solution harder.

Then consider the skillsets for supporting diverse automation tools and you see the challenges for a mobile apps QA team…

The final factor for consideration is what is the Apps lifecycle – how many builds will the automation be used for over manual regression testing?..When you take this all into consideration is there still a ROI for automation…

“What automation tools do you use for Mobile App testing?”

This is a question most QA/test professionals must have heard at least once a project/day!

The answer is not straightforward for a typical multi-platform project  – having one tool that uses one test script and works across all the platforms is the ultimate goal – to avoid the obvious overheads of having to maintain different tools and different skillsets within the team.

Any tool will require the automation solution to be optimized after development account for dynamic variances in the system under test  and verify that the automation environment produces consistent, reliable result verdicts. When automation is employed in conjunction with a continuous integration build server like Hudson/Jenkins this is especially important. If you have to spend mahours manually retesting because the automation verdicts aren’t solid it defeats the object!

How is mobile testing managed in Agile?

Theres probably not many companies not adopting Agile frameworks to structure their Apps development projects these days. In Agile the traditionally separate test and build functions are merged to compose the ‘development team’. The sprint goals are owned by every member of the development team which includes code development and test items – this cross-functional approach and co-ownership mindset is a big departure from the waterfall days! The testers in the development team must prepare the test cases based on the product requirements/specs. In parallel the developers are coding to the same requirements/specs.  How do you synchronise the sub-tasks optimally so that the test cases are ready in time for the software feature being developed and allow sufficient time for test execution to finish precisely at the sprint end ready for the show-and-tell to the project owner. OK with one task this is feasible, but consider multiple journeys with UI, functional and non-functional test tasks and inevitable blockers, defect retests , test environment issues and youll see the scheduling becomes complex ..all within a 2-3 week sprint ! …Answers on a postcard please

Future ideas..

Android testing challenges.

What OS should I support?

How do I choose my test devices?

Emulator versus Device based testing?

Can Mobile Apps projects skip structured testing and go straight to crowd test?