WinterMilestone 1 DineshDhakal

From crowdresearch
Jump to: navigation, search

Submission for Winter Milestone 1 by Dinesh Dhakal

Experience the life of a Worker on Mechanical Turk

My request to sign-up on Amazon Mechanical was rejected. The list of tasks was visible but I was unable to proceed to see the HITs. Meanwhile, I tried the Amazon Mechanical Turk sandbox and it was a different experience than what I had expected.

What did you like?

  • The sheer number of HITs available. The popularity of the platform ensures the huge number of HITs available. At any point there were over 2 Million HITs available.
  • Clear List Items with Reward and Time Allotted mentioned for each HIT. This should help the worker judge the estimated effort required
  • A directory of qualifications setup by the requesters.
  • The sorting options are quite relevant and helpful
  • The check-box to quickly view only the jobs you are eligible for
  • A good dashboard that reflects all your earnings and also the ability to view detailed statement of HITs solved on a per day basis

Dashboard dineshd.png

Status detail.png

  • The ability to quickly contact the requester and can be used to convey with issues, suggestions or requests.

Contact requester.png

What did you dislike?

  • Country restrictions - This restriction is simply discouraging for me as a worker. The internet is a very powerful tool that spans across boundaries, use it well.
  • The rewards for a lot of assignments are extremely low.
  • Some of the requests with utterly low payments make it comparable to a cheap labor camp exploiting workers.
  • Opaque decision making in terms of why worker account application is accepted or rejected.
  • Outdated legacy-looking UI elements.
  • No engine to match requesters to probable workers
  • No transparency to the worker as to why their submissions were rejected.
  • Some approvals take too long.

Experience the life of a Requester on Mechanical Turk

Reflect on your experience as a requester on Mechanical Turk.

What did you like?

  • Templates to help create most widely submitted tasks quickly and with minimal effort.
  • Ability to set the level of qualification required to maintain the quality of results.
  • Freedom to create batches of HITs with fairly well manageable parameters.
  • Ability to provide bonus to workers for each assignment.
  • Well defined APIs to interface with AWS based systems.

What did you dislike?

  • Outdated legacy-looking UI elements.
  • No way to target workers based to their skill sets
  • No universally defined skill sets for workers. This would help in targeting workers that have a good track record in a certain domain of work, e.g. image labelling.
  • Interoperability with external systems - should provide well defined APIs to interact with any external system that the requester might have. (non AWS based).
  • Though well defined, limited number of templates for creation of new HITs
  • No Quick-start guide or tutorial to get started speedily.
  • Small but annoying bugs that were rather easy to hit. Eg:- Going to workers tab under Manage and selecting workers directly somehow did not bind the worker to an Assignment ID, while clicking on worker ID through Results page did.

Dineshd Bonus no assign id.png


CSV file containing the results: Media: DineshDhakal_mturkSB_results.csv

Explore alternative crowd-labor markets

Microworkers is another platform that aims to connect the workers and requesters in the micro-tasks domain.

A few differences

  • First and foremost, I am surprised how easy the sign-up process was. It was a breeze. A simple sign-up form and an email confirmation. That's it! Workers are not required to wait for approval unlike Amazon Turk.

Dineshd microworkers signup.png


  • Better categorized than Amazon turk
  • Very few tasks available at any point as compared to Amazon Turk
  • Payments options like Paypal and Skrill available for the workers
  • Better set of RESTful APIs that can be used with external systems as desired by the requester


This is how the home page looks like after signing in.

Dineshd microworkers home.png

Similarities

Comparing this with MTurk, both the platforms are similar in a lot of aspects.

  • The sorting options are same as that on Amazon Mechanical Turk.
  • Rewards are more or less in the same ranges. Still underpaid according to me.
  • The power resides with the requester and rejection is still not explained to the worker.
  • Very limited targeting of Worker groups based on a rating system for each category.

Readings

MobileWorks

What do you like about the system / what are its strengths?

  • It can run on a mobile phone browser
  • The reach of the system (mobile-first approach)
  • Simple to use; practically no learning curve
  • The accuracy achieved by serving the same image to multiple workers until the submissions match
  • Simple tasks that can be understood with limited qualifications and language requirements
  • How it helps the weaker sections of the society to make a fair livelihood

What do you think can be improved about the system?

  • A native app that would allow the common UI elements to remain constant and just download the images
  • Batch mode - let a worker with consistent quality sets download a batch of images and submit tasks in batch so as to reduce dependence on network connectivity
  • Compression - use of image compression to reduce data costs for the workers

Daemo

What do you like about the system / what are its strengths?

The two obvious strengths of the system are Boomerang and Prototype tasks

  • Boomerang - A feedback mechanism one of the most important parts in any process design. In an efficient system, feedback must make a meaningful difference in the output. Boomerang achieves this by manipulating the availability of tasks for workers and similarly affecting the availability of workers for the requester.
  • The ability to review and return the submission to worker for improvements.
  • The idea of having a Feed of tasks and supply workers based on mutual "trust" between them.
  • Prototype tasks - Create a very effective review mechanism for the task created.
  • Incremental improvement in task designing and authoring helps workers delver expected results


What do you think can be improved about the system?

As the paper already points out there are certain limitations that apply in the real world. A couple of improvements that I can suggest are

  • Auto-suggestions - Based on the tasks that the requester has already approved and rejected, certain suggestions (or Don'ts) can be generated for workers who are still picking up a task from the same project. For example : If a requester is consistently rejecting very short answers which are below 50 words, a tip can be generated that informs this pattern.
  • Classification of tasks into various domains (like image labelling, survey, online marketing etc.); Workers might be better at one or more types of tasks. Including domain specific stats in computing probabilities to populate the task feed would also be beneficial to get quick and high quality results.
  • Gamification of the system by showing highest rated workers, highest paid, most number of tasks completed etc.

Flash Teams

What do you like about the system / what are its strengths

  • Facilitates hiring a team of verified experts through oDesk
  • Can achieve multi-faceted goals like Software Development, Animation etc. effectively
  • Modular goals that can be linked together by passing structured hand-offs as deliverables for each block to achieve a final output
  • Computational management and the ability to grow, complement or shrink team as and when required; can pipeline results to complete tasks faster
  • Can create composite teams comprising of multiple flash teams that can themselves be treated as one block
  • Automated notifications to each block regarding start time, end time, delays, changes, new inputs etc. which help in optimizing the project timelines

What do you think can be improved about the system?

  • Have a pool of constant modular teams (eg: A set of android developers) known to work well together
  • Allow the requesting user to recruit specific team members based on past interactions
  • Intelligent team selection coupled with scheduling so as to align working hours in different time-zones with project timelines
  • Implement a chain of feedbacks or ratings that run against the chain of hand-offs (each block provides feedback/ratings to the block preceding them)

Milestone Contributors

Slack usernames of all who helped create this wiki page submission: @dineshd