FAQ

Below you will find a list of the most asked questions and their answers.

Please note that the answers can be given in different ways and can evolve over time, below I will give you my version of the answers to the most popular questions from our customers.

Feel free to add new questions or improve my answers.

• How many testers do you test with? Are you sure that that number is enough?
  • Easy for clients to assume that given our large crowd number (250K) that we would test with at least 1000. In reality, we typically use 15/20 people for functional testing and 10/15 for exploratory testing. We usually go higher (around 35 testers) if we need to test cross device and cross country. Clients will ask if so "little" are enough and our answer is yes because the qualitative delta between using little vs many is virtually non existent. If you use 000s of testers you will receive loads of duplicate reports, more overhead and effort but the real impact on the quality is small. UX side is also the same thing, many publications reveal how a number as low as "8" already find roughly 80% of all improvements to be made.
• How do you ensure that testers do not disclose confidential information?
  • One of the more negative reputation aspects of crowd testing surrounds the topic of confidentiality, as it's not unusual for clients to test for situations that are yet to be made public. All of our crowd testers which are part of our crowd, during the signup process sign an NDA which legally prevents them from disclosing confidential information.

TBD:

• How is the community segmented?
• How do you distribute the digital product to the outside world?
• When is it best to test? Pre-post prod etc? What are the timing of your tests?
• What does your test include? What are the costing criteria and test perimeters?
• Are UX experts also part of the crowd? How do you know if a UX is valuable or not? What are the KPIs?