Features crowdsourcing testing for the customer

    I noticed that there are practically no notes on the interesting, in my opinion, uTest.com service. I have been working with him for half a year already, tested about a dozen releases, once even received an award as the best tester of the project, took part in the Bug Battle competition, communicate on the forum and with staff members.

    In this article, I will share my thoughts on what features of testing a testing customer contacting uTest may encounter, and what benefits it can derive. If the topic is interesting, I will write later about what the performer gets when joining this community, what types of earnings are available and how best to start a career as a freelance tester in uTest. If you have any questions on the topic, which for some reason cannot be asked in the comments, write to the mail.

    What is crowdsourcing?

    The term was coined in June 2006 by Wired magazine editors Jeff Howe and Mark Robinson in The Rise of Crowdsourcing . According to their definition, crowdsourcing is the transfer by a company or organization of the functions previously performed by its employees or under a contract to a certain community of people in the form of an appeal. Work can be collective, but, as a rule, it is nevertheless performed by individuals. The most important requirements are the use of community appeals and a large audience of potential employees.

    Test Features

    It is quite logical that this type of testing is not permissible for all projects, for example, it is hardly worth the risk of giving something original and unparalleled for testing. Yes, to work on a number of projects, you need to sign an NDA, but as far as I know, the information provided by testers provided by testers is not verified, so it is completely possible to avoid responsibility by indicating incorrect data. Moreover, it is unlikely that it will be possible to identify the source of the information leak, so even no one will sue.

    Another exception is multicomponent projects or requiring complex and expensive equipment. I don’t think that someone at home has a server from Sun or IBM, and it’s quite problematic to do such testing at work. For complex projects, a good understanding of the functioning of individual components, the ways of their interaction with each other, excellent integration testing skills, a tendency to thorough analysis and analysis of the causes of the problem are required. If at work testers can communicate with developers, use debugging tools and emulators specially developed within the company, then specialists hired for one test cycle are deprived of these capabilities.

    Based on the specifics of the testing process in crowdsourcing, we can conclude that for most community members this is a side job, which was confirmed by talking to them in an internal forum. Accordingly, one should not expect one hundred percent concentration of testers on the project and deep, thorough testing.

    I will give an example from my practice. One afternoon, I received a notification about the start of a project in which I had to check the localization of several dynamic web pages. Since I was at work, I managed to start testing only in the evening. By this time, one of the Russian-speaking colleagues had already found about 15 localization-related defects and completed testing. Since the simplest defects were already found, I had to try, but I managed to find about 10 more more complex and unseemly problems. After that, the next day I looked at the list of defects on this project and saw that after me 5 more defects were found on the same pages.

    Thus, on the one hand, the quality of testing improves, since everyone works independently and tries to find the maximum number of defects, maximizing their profits. On the other hand, you cannot be sure of the quality of testing, because the tester can fulfill his task (earn a certain amount by making certain efforts), but fail to fulfill yours — evaluate the real quality of the product.

    Another bottleneck is testing coverage. With this scheme of work, testers are interested in finding the maximum number of defects. If testing takes place inside the company or a team is hired, you can specify the testing depth of each functional area and the criteria for suspending testing. In the case of uTest, I did not encounter such restrictions. The testing itself in this case resembles the ore mining process: an area with a high density of defects is located and is being developed, it is simply unprofitable to study the rest. It would seem that when the vein is depleted, testers will move to areas less saturated with defects, but the time and budget constraints of the project should be taken into account. In addition, where “your” tester analyzes the causes of the malfunction, finds its cause and brings in one defect,

    Here one more hitch arises: the fact is that to enter this community of testers, you do not need to confirm your qualifications. So when giving a release for testing, be prepared for anyone to work on it, including people who have little idea of ​​what testing is or who do not speak English well. Several times I came across the fact that defects are clearly contradictory to what is indicated in the release requirements, in “live” stress tests, some participants behave rather stupidly and rather interfere than help, sometimes even explicitly ignoring the requirements of the product manager who tries to discipline. Of course, there are closed projects where only well-proven testers get involved,

    And finally, one of the rather unpleasant problems, the ways of solving which are now discussed at the internal forum: the correctness of the criticality and type of defect. By increasing the level of criticality and selecting the type, performers can try to increase their profits. As one of the testers wrote on the forum, on some projects up to 35% of the budget was spent precisely due to overstated indicators. Currently, project managers and customer representatives are struggling with this, but this adds to their work and, in addition, sometimes goes unnoticed. On the new version of the platform, which was released about a month ago, these parameters to a lesser extent affect the payment of the accepted defect, however, this does not completely eradicate such manipulations.

    Customer benefit

    The previous section may perhaps suggest that using crowdsourcing is inefficient, but this is not at all the case. The main thing is to pre-evaluate all the positive and negative sides and find the right approach. In addition, uTest employees are interested in long-term cooperation, so that they will help to develop a strategy that maximally takes into account the needs of the customer in the first place.

    Turning to the community, you have at your disposal a whole army of testers, ready to work around the clock and possessing almost all possible devices and systems available to the average user.

    Checking the localization of the project now does not present the slightest difficulty, because you have access to native speakers of all languages ​​who will appreciate the correctness of the translation, taking into account the cultural characteristics of the particular country whose market you plan to enter.

    Quality control can be performed in the shortest possible time, because these testers are ready to work at night and seven days a week (in fact, they usually work at this time), if only to get ahead of their colleagues when introducing defects. Taking into account the fact that all time zones are represented, the customer receives really continuous testing, so giving the release for testing on Friday evening you will receive a report on the work done on Monday morning, while the number of man-hours can be almost any.

    It's no secret that working for a long time in one company, testers, like other specialists, get used to certain methodologies and approaches, while testing the product is somewhat one-sided. In the case of attracting specialists from all over the world, the company can see its project through the prism of, perhaps, all existing schools and methodologies and synthesize the most reliable quality assessment. It will also allow you to take a fresh look and see those problems that your specialists are used to and do not document.

    The customer receives feedback from the end consumer even before entering the market, which can significantly reduce the costs associated with modernization after a commercial launch. In addition, unlike ordinary users who may not know what they really want or what will be convenient for them, or will not be able to explain their needs, reviews will be written by professionals who do this every day and know the features of the implementation of existing systems. It may well turn out that a competitor tester will work on product testing, and it will help to make a new product, devoid of the shortcomings of an existing one!

    Crowdsourcing is very convenient for companies developing products for mobile devices. Since many such devices have their own unique characteristics, for example, screen size and resolution, memory size, processor frequency, platform features, testing on all devices declared to be supported can bring a lot of headache. But what if you need to verify the quality of the multicomponent application in the networks of different mobile operators? Perhaps crowdsourcing is definitely not enough. It is not necessary to resort to it to test each assembly, but it will not hurt to check before release.

    The remaining benefits for the customer will flow from the specifics of his business, but I would like to mention such well-known brands as Google, Microsoft, ICQ, and Babylon among uTest customers. A more complete list can be found on their website - www.utest.com/customers

    Also popular now: