WinterMilestone 1 Team BK

From crowdresearch
Jump to: navigation, search

Experience the life of a Worker on Mechanical Turk

Starting with the registration process for a Worker, Mechanical Turk (MTurk) can be confusing and ambiguous. Although the feature to view HITs before an account is approved is a useful incentive for Workers to continue their registration, it becomes unclear when an account is still pending approval because it is difficult to check the registration status. Navigating the site when completing my registration led me in circles; I was asked to fill out the same form multiple times and was therefore confused about how to proceed with the registration. Perhaps a breadcrumbs design on the account registration pages could track the progress of registration for new Workers and reduce the amount of confusion in this process.

As a Worker trying to earn $1.00, I find it rare to be able to earn more than $0.20 on any given task. Many tasks are published as microtasks, which will take me between a few seconds and five minutes to complete, but will pay me less than $0.10. Although this means I can quickly breeze through tasks to earn money, it is tedious to continuously browse the MTurk Worker site for small tasks. The ability to search for slightly longer tasks that pay more would be desirable in this case. It would be helpful to have a function that allows Workers to set parameters such as “up to 5 tasks” and “earn $1.00”, so Workers could choose from a combination of fewer tasks that would ultimately pay them the amount they intended to earn. However, I am uncertain whether it is a common need amongst Workers that they are looking to earn a specific amount of money on a given day.

The interface used for searching for tasks is simple and easy to navigate. It is not the most aesthetically pleasing interface, with its brownish and greenish colors, but the content is mostly clear. Occasionally there are tasks highlighted in a different color though it is not immediately clear why it is a different color or there are tasks where I do not meet the prerequisites but did not see the requirements at the beginning, but the overall experience is smooth.

My overall impression of MTurk as a Worker is that, although it offers an easy way of earning money in the comfort of my own home or anywhere else I prefer to work, the time it takes to browse through tasks and search for “deals” where the time spent on a task is worth the amount earned would leave me as a displeased Worker. It is my impression that earning a decent amount of money on MTurk is incredibly time consuming, even if there is not much effort required in each task. It may actually be favorable to limit task choices for Workers so they will be able to spend less time browsing the tasks and more time completing them.

Experience the life of a Requester on Mechanical Turk

As a Requester, it was fairly easy to navigate the Mechanical Turk Requester page and discover how to create projects. When starting a new project, it was easy to view the different types of tasks and interfaces available in order to decide what type of project template would best fit the project I was trying to create. This makes creating projects accessible to a wide spectrum of people, from Requesters who are quite tech savvy to those who would have preferred not to use a computer at all.

After creating the project, I waited for Workers to complete the task but realized that the incentive may have been too low. Upon trying to edit the task to pay Workers more, I realized that it could not be edited. After some exploration of the MTurk Requester pages, I was able to edit the payment to a higher rate and save the project. However, I realized that this required me to re-publish the same project, which was unclear to me when I began the edit. It is understandable why a project cannot be edited while in progress, but it would be helpful if it were explicitly stated that editing the project creates an edited version of the project that can be published only as a new batch, not as an edit of the existing one. Another thing I found unclear was the “Time Allotted” section. I estimated the task to take Workers no longer than two minutes, but allowed them an hour to complete the task since the description asked for a generous amount of time. I believed there would be another section to write the estimated task duration, but there was no such section. These misunderstandings caused me to create a low-paying task that seems to require much more time than actually needed.

After publishing the project, obtaining Workers was smooth and painless. I was about to reach 15 HITs within a few hours. When viewing the results, it is not immediately clear which HITs are approved, rejected, or pending approval. My HITs were set to automatically be approved within a certain time frame, but HITs were not clearly marked as “approved” after the time passed and the option to approve those HITs were still available. The interface for Requesters can be improved to avoid confusion by disabling the option to approve a HIT once it has already been approved and clearly marking the HIT as “approved”.

Overall, my impression of MTurk as a Requester was positive. Task creation is simplified for Requesters and Workers quickly complete the task even with very low payment. I believe that using MTurk as a Requester is more rewarding than using it as a Worker, because Requesters are able to get results from Workers quickly, whereas Workers browse through and complete many microtasks at low-paying rates, which is time-consuming and inefficient.

Explore alternative crowd-labor markets

Compare and contrast the crowd-labor market you just explored (TaskRabbit/oDesk/GalaxyZoo) to Mechanical Turk.



  • What do you like about the system / what are its strengths?

MobileWorks is an innovative idea for providing short and simple microtasks to people in need of work. The strengths of MobileWorks lie in the minimalism and simplicity both in the user interface and its overall goal. The minimalistic user interface is a strength because it presents the task in a straightforward manner and reduces extraneous clutter that may complicate the task or distract users from the task. Another strength is that MobileWorks only focuses on one task. Since it only creates one type of task, optical character recognition (OCR), it decreases and nearly eliminates the confusion of what the user must do in each task, since every task follows the same ideas and instructions. By doing this, users who continually perform these microtasks will easily become accustomed to the format and purpose of the task, leading them to perform the task faster.

  • What do you think can be improved about the system?

One potential improvement of MobileWorks is the reduction in the amount of time each microtask takes a user to do. While the study mentions high efficiency, 120 tasks per hour only reduces to two tasks per minute. Since the task itself should take users only a few seconds to complete, based on the length of the tasks as shown in Figure 2, it seems that users are either getting heavily distracted during the task, or an intervening event prevents faster task completions per hour. It is possible that poor internet connections or slow loading of the task are preventing the users from performing at an even higher efficiency. If MobileWorks is able to develop a method for presenting each task to the user faster, it will presumably increase the efficiency of the system.


  • What do you like about the system / what are its strengths?

One of the greatest strengths of Daemo is its implementation of prototype tasks. Small iterations of a task to discover faults or areas for improvement are an essential aspect of the design process, which, when implemented in this crowdsource platform, may be promising for the functionality and success of each Daemo task. The prototype tasks allow workers to explicitly tell requesters where or how the task can be improved and also allow requesters to realize improvements by obtaining a small sample of completed work.

  • What do you think can be improved about the system?

Although the idea of having a task released initially only to “check-plus” workers, this could cause problems for tasks which hope to receive data from tasks that require a wide range of skills or a diverse population. For example, many university psychology studies are now conducted through MTurk, but specifically targeting highly-rated workers will bias the population sampled. For cases like these, it would be best to develop a way of identifying a diverse set of workers who also have high ratings and recommending that the task be launched to these workers as a way of surveying an unbiased population and receiving completed tasks that are likely to be Accepted.

Flash Teams

  • What do you like about the system / what are its strengths?

Flash Teams act as a useful and fast way to find experts for specific tasks in order to optimize individual pieces of a project. Blocks are a strength of Flash Teams because they explicitly outline the tasks needed and divide them accordingly so the correct experts can assess problems and complete the task. Furthermore, blocks allow people to pass on tasks that they cannot perform themselves to different groups of people who are experts at the task. Another strength of Flash Teams is that it enables a crowd to continually work on a project and complete it faster by having people from different time zones complete different tasks. Whether the tasks are completed in parallel or in serial, the advantage of having teams that can work on a block at any hour helps it get its name of “Flash” Team.

  • What do you think can be improved about the system?

Flash Teams could be improved by implementing a way for better communication amongst the crowds that work on each block. Due to the setup of the blocks, it is possible that one crowd working on one block may misinterpret the input from another block, thus producing the wrong or unintended output. Flash Teams’ strengths are also their weaknesses because allowing blocks to be completed in parallel increase productivity but also increase the chance of miscommunication, which could consequently decrease productivity by causing crowds from certain blocks to restart their tasks. A better communication system or method for tracking progress and making sure that the output is indeed the intended output would greatly augment the power of Flash Teams.

Milestone Contributors

Slack usernames of all who helped create this wiki page submission: @bchinh, @gvkarthik93