Crowdsourcing Provides Answer to App Testing Dilemma

Wed, 02/24/2010 - 12:31pm
Doron Reuveni, CEO of uTest

Of all the challenges that come with launching the next great mobile app, testing is too often an afterthought. But in an industry where iPhone, BlackBerry and Android apps are fighting for media attention, investor dollars and consumer wallets, the pressure to get apps built, tested and launched quickly has never been greater. Even the smallest flaw can ruin a mobile app (and the company behind it). So why do some mobile developers ignore the critical application testing phase?

The answer is comprehensive testing of mobile apps has proven to be difficult and prohibitively expensive using traditional QA methods. The testing matrices for mobile apps are exponentially more complex than for Web and desktop apps, and testing methods have not kept up. No matter what type of mobile application – chat, social networking, games, business or productivity tools – all mobile app developers face the same testing complexity across:

Doron Reuveni• Handset Maker and Model
• Operating System
• Wireless Carrier
• Language (for global apps)
• Location, Location, Location

This quality assurance challenge calls for a new and improved testing model – crowdsourcing. Through crowdsourcing, mobile app companies can use a community of diverse and talented professional testers, capable of testing their app across any and all criteria, and on an on-demand basis. Communities of testers ensure that mobile apps get to market faster, with higher levels of quality and at a lower cost.

Four Alternative Testing Methods: Why They Fall Short in the Mobile World
The four most popular approaches for testing software are outdated and impractical for mobile's "in the wild" testing needs. To illustrate this point, here's a quick summary:

1. In-House: Building a comprehensive in-house testing lab is extremely time consuming and expensive. Imagine the expense of building an in-house team and lab capable of assuring the functionality for iPhone, BlackBerry and Android handsets (of all makes and models) across wireless carriers in the U.S., U.K, Australia, China and Japan. For reasons of cost and coverage, it's no surprise that mobile app companies rarely rely solely on in-house testing resources.

2. Offshoring: While offshoring has proven to be less expensive than hiring an internal team, developers who have contracted with such firms quickly find themselves with no insight into the day-to-day testing activities, losing what little control they had over the process. In addition, even the largest outsourcing firms don't have the resources needed to provide adequate real-world testing coverage across location, handsets, carriers and OS.

3. Emulators/Simulators: One of the biggest challenges for mobile developers is traditional testing is occurring in an environment far removed from the real world. The gap between "in-the-lab" simulation and "in-the-wild" usage is vast and cannot be ignored. The convenience of simulators and emulators has made it easy to be lured into a false sense of security, but they should not be considered a substitute for real-world, on-device testing.

4. Beta Testers: It's rare for a software company to attract a large group of beta testers to test their app. After all, not every company can be Google, with its wildly popular beta versions. But even if you can assemble a large beta group, the method still falls short on its own. First, if a beta goes poorly, most companies can't afford to have it happen in the bright lights of the blogosphere and Twitterati. Beta testers are more often like users in that they will only try to get your app to function properly; real testers will systemically structure their usage to identify weaknesses in your app. In short, users use and testers test.

Crowdsourced Mobile App Testing
The unique demands of mobile application testing fit the crowdsourcing model to a tee. When your users are dispersed around the country (or globe), your testers should be too. When your users experience your app outside the safe confines of the testing lab, so should your testers. Now, any company that develops or markets mobile apps can get the benefits of having world-class testing coverage.

Crowdsourcing Challenges: Quality and Control
The thought of having testers from around the globe may be daunting. That's why it's absolutely essential to choose a reputable community that can point to a track record of its customers' real-world successes.  It's also important to find a community that shares the past performance and reputation of community members. Other common challenges include:

1. Communication: As with any successful project, communication is key. While crowdsourcing doesn't enable face-to-face conversations, it does enable decision-makers to converse with their team in real time through an easy-to-use online platform.

2. Intellectual Property: While it might seem like your IP could be exposed to a community of thousands, top crowdsourcing companies allow customers to carefully select their testers and establish non-disclosure agreements (NDA) with community members. 

3. Quality: It's easy to believe that anyone can test your application. Fortunately, mature communities assign a dedicated project manager to help customers handpick testers by location, handset, carrier, testing experience, past performance ratings, or any other criteria.

Benefits of Crowdsourced Mobile App Testing
There was a time when the testing matrix for mobile apps was too complex, but with the rapid evolution of crowdsourced testing, top mobile app companies – from startups to enterprises – are doing the impossible: maintaining app quality, achieving broad testing coverage, meeting launch dates and staying within budget. Here's a quick look at what to expect when implementing crowdsourced testing:

1. Control Costs: Crowdsourcing allows managers to utilize lower-cost support without being tied down by long-term commitments. The fundamental structure of online communities creates competition, making it more accountable and cost-effective than traditional outsourcing.

2. Tackle Increased Complexity: Significant quality improvement is achieved when development and testing is done across a wider set of handsets, carriers, locations and languages. Crowdsourced app testers can be easily recruited based on a wide variety of criteria.

3. Get to Market Faster: Crowdsourcing enables you to ramp up or down to meet your needs at any given time - eliminating the delays and staffing constraints that often stifle mobile app companies around peak release times.

4. Achieve Higher Quality: Customers expect their mobile apps to be fully functional and bug-free from day one. Mobile app companies that believe "good enough" is good enough when it comes to quality and usability will learn a valuable lesson (the hard way).

5. Retain Users: If your product has problems, or doesn't meet expectations, it's easier than ever for your customers (and prospective customers) to find your competition.

The future of mobile applications remains bright. As the mobile market doubles and triples in size over the next decade, those that have made testing coverage a priority will enjoy their ROI in terms of increased market share, profitability and above all, user loyalty.

Doron Reuveni is CEO of uTest.


Share this Story

You may login with either your assigned username or your e-mail address.
The password field is case sensitive.