WinterMilestone 1 @arjunpp

From crowdresearch
Jump to: navigation, search

Experience the life of a Worker on Mechanical Turk

I couldn't get approval from Mechanical Turk so I did tasks on Microworkers and Amazon Mechanical Turk Sandbox.

Microworkers

I did HITs primarily on Microworkers. I did 10 very simple tasks out of which 9 were accepted and 1 is still pending. Many requesters were fast with their approval and managed to earn 0.96$. The following is my feedback regarding MicroWorkers:

Cons

  • Poor Introduction: Their introduction page is very confusing and if someone new to HITs joins this website, he is not introduced to the site and how things work. Maybe something like a showcase would help a lot.
  • No multiple filtering: I wanted to look for jobs with multiple criteria selected and it is not possible. A worker should be able to see which jobs are highest paying of multiple types(Sign up/Vote/Click).
  • Poor relevancy: You can only select one method of sorting and it just naively sorted the jobs according to the type you select. It would be better if they use some ranking algorithm based on recency, payment and Time to finish.
  • No Search: Even though the filters help in finding jobs, a search would be preferred if I am trying to find a particular job.
  • No notifications: It doesn't notify me if some work I did was approved. However, they do have a notification for Hire Group tasks.
  • Urls not hyperlinked: Why aren't urls in the tasks hyperlinked? Is it to prevent crawlers and bots?
  • No categorization of workers: Nothing to differentiate from a good worker and a bad worker apart.

Pros

  • Easy to Join: Unlike MTurk, anyone can join and do tasks on Microworkers. MTurk rejected most of the applicants on this project and most of us non-US citizens had to go for alternatives.
  • Hire Group Tasks: A requester can invite specific workers to do the job. I was invited to do a sign up job and these tasks were almost twice as much paying as a normal job.
  • Time to finish: It is an estimate to how much time is required to finish the task. It is not a constraint and a worker can take as much time as he wants.
  • Time to rate: The time which the requester takes to rate the task. I cannot comment on whether it is a constraint as I could not request.
  • Categorization of requester: They have bronze, silver and gold medals for requesters based on the approval of tasks.
  • Fast approval by requsters
  • Submitting tasks is simple and fast.
  • Has worker ratings

Jamesalcatraz microwoker.png

Amazon MTurk Sandbox

I used the Sandbox for doing a few HITS which were submitted by fellow participants at the Milestone HITs page.

Cons

  • No Categorization/Filters: MTurk seriously needs some categorization for the tasks. It has keywords which were very random and confusing.
  • Poor sorting structure: They have a dropdown regarding what criteria to sort from. From a UX point of view, it can be improved a lot by having sortable tables.
  • No Introduction/Get Started
  • The time constraint is unnecessary.
  • No "Time to rate" category similar to Microworkers
  • No notifications

Pros

  • Search function
  • Easy and fast to submit tasks
  • Has a master category based on qualifications.

Jamesalcatraz mturk.png

Experience the life of a Requester on Mechanical Turk

I couldn't use the money I earned from Microworker to post a real job/campaign, so I used the Amazon MTurk Sandbox for posting a job.

Cons

  • Only two types of workers.
  • Some types of tasks require a Master qualification but they are not distinguishable in the list of categories. You have to click on create project and read the conditions to find out.
  • No access to requester's profile: I wanted to see how authentic and rewarding the requester was but I could not find a way to check out his details.
  • The process of creating a job needs to be simpler, cleaner and involve a lesser number of steps.
  • I could not find the url of my job which was posted and I had to login to the worker sandbox to find the url by searching.
  • Tasks are not Editable: I guess it is to prevent scamming, but I think requesters should have an initial allotted time before workers can see the HIT during which they can edit.
  • There was a bug in the HTML creation of Tagging the Image HIT.

Pros

  • Has predefined templates for a multiple category of projects to choose from.
  • Has a nice job management tab for seeing the progress of your jobs.
  • Has a Master qualification and we can also create custom qualifications.
  • Can publish tasks in batches.

Link to CSV: File:Batch 119501 batch results.csv

Jamesalcatraz mturk req1.png

Explore alternative crowd-labor markets

I have explored MicroWorkers and have reflected about my experience previously in WinterMilestone_1_@arjunpp#Microworkers

Readings

MobileWorks

  • What do you like about the system / what are its strengths?
    1. It has been more than 4 years since the paper has been published. The number of mobile users in India has increased a lot and as smartphones have become very cheap, the rural population is gradually shifting to smartphones. With cheap access to internet, the platform can help the rural worker make some money by doing micro jobs.
    2. The Human OCR process involves scanning of documents and the platform creates the application UI, so the requester doesn't have to go through designing the task.
    3. The entire process seems simple and accurate. By using the accuracy history of the worker and giving same tasks to different workers, they ensure that the process is foolproof.
  • What do you think can be improved about the system?
    1. Would work only for OCR tasks: How do users do tasks which are very difficult or tedious using a mobile phone? For eg. many sign up tasks use websites but using those websites on a cheap phone would take a lot of time and data.
    2. Some tasks take up a lot of internet data: Internet is cheap but it isn't free. Wired connection is cheaper than Wireless mobile internet, so people doing tasks on a computer can afford to do any task. But mobile phone users will have to pay for each mb they use and they will be picky about the tasks they do.
    3. No scope for employer-worker interaction. There is only a feedback and no other form of rating of a requester or job.

Daemo

  • What do you like about the system / what are its strengths?
    1. The platform solves the essential loopholes in the existing products which have very little employer-worker interaction.
    2. The Boomerang rating system is a very impressive way of rating which reduces the inflation of reputation score.
    3. Prototype tasks are incorporated to maintain high quality of tasks.
    4. Task Feed  : The tasks would be ranked by combining a variety of scores.
  • What do you think can be improved about the system?
    1. Surveys on small number of people aren't always true for a crowd: How are you sure the crowd would be same as the tests you have conducted?. Especially for the prototype tasks, the results maybe very different. So try testing on a larger crowd.
    2. No task recommendations based on history: Why isn't Daemo recommending tasks based on the tasks which a worker has done before.
    3. Boomerang tasks require a lot of rating: While you cannot force them to rate, you sure can persuade them to. Like e-commerce websites keep bugging users with rate the product and comment.

Flash Teams

  • What do you like about the system / what are its strengths?
    1. Working in groups has mostly proved to be more productive in working alone. Flash groups can dominate the crowd market by utilizing expert worker's potential to do complex tasks.
    2. Flash teams use a model that can take advantage of computation’s ability to abstract, scale, and visualize progress.
    3. It aims to coordinate the experts efficiently to perform tasks faster.
    4. Most of it is automated and inspired from organizational behavioral research.
  • What do you think can be improved about the system?
    1. Their study is only based on napkin sketch design teams. They need to conduct more tests and verify the results.
    2. The initial recruitment still requires human intervention. The flash group's hiring can be automated once the system has begun.
    3. Flash teams still faced some lack of coordination which needs to be resolved. Why not include flash teams to coordinate flash teams, and use the human intelligence to strengthen the existing AI. It should improve the results in my opinion.

Milestone Contributors

Slack usernames of all who helped create this wiki page submission: @arjunpp