Crowdsourced Mobile Testing: Doing it Right

Anyone who has ever participated in a team responsible for mobile application quality should be aware of some of the challenges and problems related to this process. Some of them are mentioned below :

• Selecting the group of devices, their systems and versions
  


• Testing under different network conditions (type of network, strength, etc.)
  


• Testing cooperation with the GPS receiver in movement and different locations
  


• Verifying behaviour of the tested object in cooperation with other installed applications and different custom settings
  


• Usability testing - which is extremely important, and where testers must try to play the role of the final user, without any bias and initial knowledge about the SUT (Software Under Test)
  

On which platform should we test our system?

Sometimes you can find this information in the requirements specifications, but most of the time all you know it is just “the most popular Android devices and system versions” or something similar. You will need to have these devices available in your lab as well. But there is a problem– huge expenses, especially if it is your first mobile project.

When there seems to be no way out of the situation, a new possibility appears – the power of people!

Note that when you plan to organize managed beta tests, the problem of selecting the right devices just disappears.

Once we have dealt with the first challenge, a new one arises - selecting the right network. There is an unlimited number of combinations including types of providers(3G, 4G, LTE, etc.). You need to verify your system under a strong as well as a very weak signal. It is finally important to simulate a situation where the network changes automatically (for example from Wi-Fi to 3G). It all adds up to plenty of tests and preparations for you to to handle. There are two solutions – buy as many as different pre-paid starters as possible and also move to distant locations to test various signal strengths, or ask people to do it for you. Once again, when you organize a beta test, everything happens automatically. People just have different network providers, they live in city centers or villages, or they may use high speed LTE or outdated UMTS. All you need to do is to just select any configurations you need (when selecting participants). It seems to be the simple and effective, and, it certainly is.

Most of the recent mobile applications use a GPS receiver for some navigation-related functions. Testing the cooperation with GPS is closely related to the problems described in the previous paragraphs. Besides selecting devices, you usually need to be in constant movement in order to verify your system in a realistic environment.

To solve problems related to this, just ask your beta testers to run the application, then collect their localization data and send it to your servers for further processing. You will certainly receive enough data for evaluation and verification.

There are thousands of applications available in the online stores nowadays. Everyone has their favorites, which means that no two devices are the same. When you are running some tests in a lab, or using outsourced services, your tests are carried out on clear devices using default (factory) settings, or, which is even worse, you can use a platform that was previously used for system development, with many stubs and drivers installed on it. As a result, you have no chance to detect configuration issues related to the interaction with popular applications and widgets.

Besides typical configuration errors, you should also be aware of interoperability issues such as data transfer between applications, import export operations, etc. I am sure you all know the cure for these problems – crowdsourcing. Instead of wondering which applications you need to exchange information with, ask your customers what they would like to use together with your system so that they can report to you if it works this way or not. Of course, you need to ask the right questions and make sure that you finally receive valuable feedback.

Users definitely like to use applications which they really like to look at and feel comfortable with. Very high user experience as a result of complex usability testing seems to be a key process. But usability verification is almost impossible in a test lab unless you have another, independent team for this purpose.

Even if you try to outsource this activity to another test lab, it will still be executed by professional testers who usually have problems with empathizing with the role of end users.

A correctly constructed survey can give you a lot of useful data, but you must be very careful in designing the questions. Too many or too personal questions can discourage users from giving true answers, or even finishing it.

Crowdsourcing sounds not that bad unless you’re into very serious works which require not only expertise but intensive management. I hope you all enjoyed reading my blog and do let me know your thoughts.