Milestone 1 spontaneous

From crowdresearch
Jump to: navigation, search

Since we are not from US/Canada, we could not work on Mechanical Turk and our account was rejected. We worked on Sandbox and completed several HITs for a combined earning of $3.

Experience the life of a Worker on Mechanical Turk

On Sandbox, signing in was easy, although we did later get a mail that our account was rejected, regardless we were allowed to complete HITs.

What did we like?

  • Each HIT comes with a time limit. This makes sure that the worker takes the assigned work seriously.
  • A worker can search for hits according to the money he wishes to earn.
  • Ability to view a HIT before working on it.
  • Account Dashboard allowing us to view our pending and accepted HITs and earnings.

What did we dislike?

  • No fair payment protocols are followed.
  • No pre- defined categories and tags are present to allow the user to search for an appropriate work.
  • The website is not responsive.
  • Tasks can be quite boring if you don't find one according to your interests.
  • All requesters do not make their tasks clear for workers to understand and complete.
  • Payment delay and uncertainty.
  • Not much incentive for workers.
  • Tasks don't always lead to corresponding compensations

Experience the life of a Requester on Mechanical Turk

What did we like?

  • The sample questions available were a great help to understand the process.
  • Examples and tutorial for CSV creation made it easier.
  • Instantaneous ability to view results.

What did we dislike?

  • While signing in as a requester, only United States residents were allowed to become a requester on Sandbox.
  • The work submitted couldn't be tested for quality submission.
  • Previous knowledge of HTML is a must to edit the predefined templates.
  • Navigating between worker and requester accounts is confusing.
  • Making tasks and the navigation of designing them according to requirements is not clear and gets confusing.

Explore alternative crowd-labor markets

We explored Crowd Flower and Microworkers crowd sourcing markets and these were our observations.

[1] Crowd Flower

  • To be a requester, it was necessary to have a work email.

[2] MicroWorkers

  • It was easy to register on the site.
  • There was a limit on the number of tasks a worker can perform.
  • Many tasks we came across were repeated.



What do you like about the system / what are its strengths?

  • Good use of high penetration of mobile phones and low cost of internet
  • Works on basic phones since smartphones are relatively expensive
  • Use of mobile phone allows being able to work from anywhere
  • Benefits the financially needy, since mostly everybody owns a mobile phone but the same cannot be said for a laptop/desktop

What do you think can be improved about the system?

  • More complex tasks for people using smartphones like image classification
  • Consequently compensate accordingly
  • Allow native language tasks depending on region


What do you like about the system / what are its strengths?

  • Prototype Tasks that would help identify genuine workers for a requester
  • Prototype Tasks would also help the worker understand the task that the requester expects to be completed and his ability to complete those tasks
  • Task feed ranking helps the worker in sifting through the tasks faster and being able to work on tasks related to his/her interests.
  • Drag and drop interface would help attracting more clients since it makes life a lot easier
  • Worker's ability to provide feedback to the requester on how the task can be improved, allowing interaction between the two
  • Boomerang rating system helps the requester and the worker mutually benefit by ensuring quality work for the requester and assured wages for the worker

What do you think can be improved about the system?

  • As of now, no clear method for how to make tasks clearer by requesters so that workers are able to understand and hence deliver as per the requester's requirements
  • New workers would be at a disadvantage
  • No compensation for completing prototype tasks for the requester if worker is rejected

Flash Teams

What do you like about the system / what are its strengths?

  • Sequences of linked tasks allows coordination and effective collaboration between the workers.
  • Crowdsourcing complex work by making experts from a field work collaboratively.
  • 'Flash team' is assembled according to the task to be completed.
  • Directly responsible individuals(DRIs) handling coordination and management of tasks.
  • Foundry recalculates time for completing tasks every time a task is completed early/late.
  • Pipelining allows streaming intermediate results, removing the mandatory need for waiting for a task to be completed.

What do you think can be improved about the system?

  • Combining several flash teams may cause conflict or confusion for larger tasks and may lead to incomplete delivery.
  • Assumption that end users know how to create teams but most people are not able to handle management.
  • No system for interaction between the flash team and requester for continuous feedback and discussion of expectations.

Research Engineering Milestone

Daemo Local.png

Milestone Contributors

Slack usernames of all who helped create this wiki page submission: @amitoj and @diksha