WinterMilestone 1 MMM

From crowdresearch
Jump to: navigation, search

This is the page for milestone submission page by Team MMM.

Experience the life of a Worker on Mechanical Turk

Instructions: Reflect on your experience as a worker on Mechanical Turk. What did you like? What did you dislike? If you're from outside the USA and unable to access MTurk, try try the worker sandbox, or other sites like CrowdFlower or Microworkers or Clickworker.

Experience the life of a Requester on Mechanical Turk

Instructions: Reflect on your experience as a requester on Mechanical Turk. What did you like? What did you dislike? Also attach the CSV file generated when you download the HIT results. If you're from outside the USA and unable to access MTurk, you can try the MTurk requester sandbox, or CrowdFlower or Microworkers or Clickworker

Explore alternative crowd-labor markets

Compare and contrast the crowd-labor market you just explored (TaskRabbit/oDesk/GalaxyZoo) to Mechanical Turk.



  • What do you like about the system / what are its strengths?
    • Ability to utilize the workforce in less developed countries
      • According to the research, while Amazon’s Mechanical Turk consists of 56% workers from United States and 36% workers from India. We can possibly utilized and expand the workforce in less developed countries for these tasks
    • Flexibility of Mobile
      • Many Indians lack access to a desktop computer. Having a platform on mobile allows them to fulfill the tasks anywhere, anytime, expanding the market of the workforce
      • Such mobile platforms can expand the type of tasks beyond just SMS based tasks that other platforms used, such as TxtEagle
    • Accuracy
      • The overall accuracy of the workers is very acceptable. Without considering multiple entry error detection, the accuracy was about 89%. Dual entry accuracy was 98.79% and triple entry 99.89%. Using multiple entries, the platform offered from Mobile Works has a very satisfying accuracy level compared with desktop platforms.
  • What do you think can be improved about the system?
    • Expanding the type of tasks
      • Apart from image recognition, there are also other types of low-level tasks: such as those that require users to visit certain websites to gain information or fill out surveys. Mobile Works can consider building platforms that support these tasks. For example, for tasks that require users to go to specific websites, Mobile Works can preload the text from those websites so that users do not have to press the URLs directly.


  • What do you like about the system / what are its strengths?
    • Ability to share responsibility on the quality of tasks
      • For traditional crowdsourcing platforms, the worker bears most of the responsibility in the quality of the tasks performed. However, in many cases, the tasks might not have been well-design or clear for the workers. By sharing the responsibility using prototype tasks, this system is a lot fairer
    • Aligns incentives between worker and requester
      • The boomerang mechanism ensures that workers who have been rated higher by a requester have a higher chance in working with the same requester in the future. This reduces issues with rating inflation and provides incentives for requesters to fairly rate workers
    • Rating systems that are more aligned to the actual performance of worker
    • Increased task quality
      • With better alignment-incentive systems and the existence of prototype tasks, the tasks are better designed and workers have greater incentives to perform the tasks better
  • What do you think can be improved about the system?
    • May not have enough incentives for workers and requesters to rate
      • Can suggest requesters to rate at least 10% of the workers
    • Not every user will be incentivized to rate carefully with Boomerang.
      • Requesters may feel that rating future workers increases their workload. Those who only have one task to do may care a lot less about rating.
    • Feedback mechanism
      • The current Daemo feedbank system is static. There can be further experiments with synchronous chat so that requesters can have continuous iterations and this can speed up the prototype design process.
    • Additional operational costs for the requester:
      • When requesters are required to iterate, there may be additional time cost as well as the monetary cost from the premium given to prototype feedback workers/
      • However, these costs may be traded off with better quality of tasks and increased payment for prototype tasks to reduce wait and iteration time.
    • Workers may not be incentivized to do prototype tasks
      • Workers often only look for small tasks for larger quantity returns. We can consider paying even more for prototype tasks or offer other privileges to workers who complete prototype tasks: eg. faster access to certain tasks

Flash Teams

  • What do you like about the system / what are its strengths?
    • Ability to handle complex tasks
      • The flash teams system allows requesters to request complex tasks that require expert level or deep domain knowledge that spans across different areas. For other platforms available, the system makes it difficult to decompose them into independent micro tasks that anyone can complete.
    • Saves half the time compared with simply assigning teams
      • According to the research, the flash team took approximately half as many work hours as the traditional team. By calculating active work time, even the slowest flash team completed the task faster than the fastest team in the control traditional condition. On average, the control teams spent 2.4x the hours on Design, 1.9x the hours on UX Research and 1.4x the hours on Development. This results in an extra 10hr 44min in cumulative work.
    • Better coordination and work experience
      • According to the results of the research, the teams under the flash structure followed the design process more closely and required less coordination.
    • Better at emergency handling
      • When workers from both types of teams disappeared due to personal reasons or other commitments, the flash teams were quick in making rearrangements and could reach out to the crowd for replacement quickly. For the traditional control team, they usually grew too frustrated with the experience and many other members left because of this reason.
    • Can take advantage of timezone differences
      • Flash teams can make use of differences in timezone that may allow them to carry on tasks uninterruptedly for days or even weeks.
  • What do you think can be improved about the system?
    • Less of a camaraderie experience
      • According to the research, certain flash team members said that they wanted a better experience in building camaraderie — which may be more difficult to achieve in flash teams as tasks were divided and some members would just finish the task and leave.
      • Solution:
        • The Flash team system can consider making slightly longer blocks or group several blocks into larger “processes” or categories so that members will feel that they are more welcomed or obliged to engage / provide feedback in processes with several building blocks even when they might not have the expertise that certain blocks of a process.
    • Can consider providing review mechanisms so that members who previously worked with each other can rate their members to optimize team formation
      • The flash system can consider asking members to rate their other members so that when a new task is formed, the system could arrange those people who liked working with each other together in the same team, and those who did lot like the experience of working with each other in separate teams. This is because different people may have different personality traits even if they have the same expertise and different personality traits may work better with each other. More organizational behavior research can be done in this area.

Milestone Contributors

Slack usernames of all who helped create this wiki page submission: @

@mengnan @manojpandey @michellechan