Milestone 2 APES

From crowdresearch
Jump to: navigation, search

Attend a Panel to Hear from Workers and Requesters


  • There’s a requirement to validate the IDs of people participating as workers on Mturk. Various social science researches require that demographics of the workers are known. There’d been cases where a survey is to be filled by young mothers but various males fill it up messing up with the data.
  • During early morning and late night hours, there are less hits and more workers, proportionally. This can also be verified by the mturk tracker. But during the normal hours academic researchers become more active online and hits increase for the workers interested in the same.
  • Get the right people to get the task done. This can be achieved by seeing the person’s qualifications and checking whether they are in coherence with the skills required. Assigning smaller tasks first instead has both its pros and cons. Downside being you end up spending more time deciding the goal points, but you also get to learn more about your own task.
  • Its always preferable to divide the work in subtasks as much as possible. It boosts up the work quality.
  • The rejection system on Mturk is “fundamentally broken”. Most requesters don’t reject the workers. There are many reasons behind it. Firstly it looks bad. And rejected workers repeatedly plead the requesters to remove it, thus wasting a lot of time on both the sides.
  • Monetary incentives matter, in fact it is the only reason that binds many turkers. Although the some workers think of the free tasks as voluntary online work.
  • Amount of money made by the turkers vary over the week. The academic researches slow down during weekends and holiday seasons. But the surveys remain more or less in same number.
  • Introducing better tags would help the turkers find the work they are interested in. For example subjectwise tags such as philosophy, psychology, etc. can be added to the tasks or surveys.

Reading Others' Insights

Worker perspective: Being a Turker

Observations about Workers:

1. They condemn unfair rejection of work,slow payment,low wage and lack of communication with requesters.

2.The turkers are interested in comparision to gain information and knowledge,set themselves targets and importance of their AMT income varies depending on earning ability and other life circumstances.

3.The workers who do the AMT work even for the extra money do so because they do not have enough money from other sources.

4.For some workers who live in difficult circumstances where their AMT work is vital enough to make their ends meet.For such workers AMT can form a so-called "safety net".

5.Some workers appreciate the fact that no transpotation costs are required,no fixed hours are required,and judgements are restricted to the work they submit rather than the personal appreance and the way they present themselves.

6.The turkers are offended when requesters reject their work for reasons they donot understand as this deprives them of the money they think they deserve and also it has a damaging effect in their approval.

7.They are fair as observed in the TurkerNation. If a rejection is fair from their understanding, or in the explanation they are given, they will generally admit fault.They do not suspend everyday moral standards.

8. Turkers fall broadly into two categories: (i)Novice:They have concentrate more on both getting their approval ratings up and also their HIT count. (ii)Experienced:They are not concerned much about their HIT count but the approval rating is of key importance to them as higer approval ratings are crucial in getting them access to wider selection of and better paying HITs.

9.As observed they donot want the US government to intervene and regulate AMT as they think catering to the diversity in opinion is not an easy task.

10.They worry that interference by journalists and academics will actually just lead to closing AMT which leave those for whom AMT is a safety nets on the brink of being broke.They orient towards the idea that they can regulate the AMT marketplace with their actions.

11.For most of the workers AMT is clearly a labour market and the notion that they do it for the fun rather than the money is rejected by many as money is the motivating factor for them.

12.Their biggest concerns are having enough information to make good decisions on selecting jobs, having good relationships with requesters, and how to act collectively.

Observations about Requesters :

1.Some requesters are fair and look to turker nation to discuss their hits and aprrove hit's quickly and fairly.

2.There are some bad requesters too who look to take advantage of the low wage paying rate and disapprove wrongly in order to save money.

(Many points cannot be deduced about the requesters as the article was focused on the worker's perpective.)

Worker perspective: Turkopticon

Observations about Workers:

1. The reason for using AMT varies from person to person. For some it is the sole source of income and for some its just a way to use free time to earn some extra buck.

2. Workers are not cared about much by AMT as there are hundreds of thousands of workers and some leaving the service is not a concern, thus AMT focuses on requesters.

3. Workers are thus ignored and they need a platform to speak up for their good as well as of fellow workers.

4. The quality of work is subjective to the requester resulting in sometimes rejection of income inspite of putting a lot of work into it.

5. The honest(who work sincerely and donot mess around for fast money) need to have some kind of assurance of accpetance.

Observations about Requesters :

1. They benefit from this service as they get to do cost cutting by paying only for the required work and need to hire any labor full time. Thus they can pay way below minimum wage rate and get the work done.

2. They use labor as a service rather than as human resource. Thus they pay only for the service they use and are not liable for any other expenses.

3. They are the main focus of AMT and thus enjoy many services and benefits.

Requester perspective: Crowdsourcing User Studies with Mechanical Turk

Observations about Workers:

1.Micro-tasks markets are easy to use promising source to use your skills to earn money from anywhere and everywhere.

2.if user solve tasks seriously and spend some time on that then his submission will not flag as invalid and his ratings will improve over those who does gaming to take advantages of system maliciously.

3.So users who really sincere towards it will have an edge over those who just do it for advantages by answering randomly only thing is requester should design the task in this fashion as done in experiment 2.

Observations about Requesters :

1.User studies are important for many aspects of the design process as user input and feedback can improve requester design significantly.

2.Important factors for planning user evaluations are economic cost and time required. Also Collecting input from only a small set of users is problematic.

3.Micro-tasks markets are promising platforms for user study tasks but extra care must be taken in design of the task, such as user measurements that are subjective or qualitative.

4.M-turks have advantages like can be used for rapid iterative prototyping by asking users a number of verifiable questions regarding content.

5.But requester have some limitations on usage of m-turks on issues such as different browser experience or distractions in physical environment.

6.In designing tasks it is extremely important to have explicitly verifiable questions as part of the task and it is advantageous to design the task such that completing it accurately and in good faith requires as much or less effort than non-obvious random or malicious completion.

Requester perspective: The Need for Standardization in Crowdsourcing

  • Workers have a lot of flexibility, in the sense that they can easily find the work they are interested in. E.g. based on the diffculty or skill of the task, a worker can make an informed choice based on his qualifications.
  • We have a queue of standardized tasks that need to be completed, and workers can complete them at any time, without having to think about the reputation of the requester or to refamiliarize themselves with the task. This should lead to much more efficient task execution.
  • There is no need for requesters to think on how to create the user interfaces and best practices for simple tasks.
  • If requesters want to prioritize their own tasks, they can simply price them higher than the current market price. This corresponds to an increase in demand, which moves up the market price. On the other hand, if no requesters post tasks then, once the tasks with the highest prices get completed, then we automatically move to the tasks that have lower price associated with them.

Both perspectives: A Plea to Amazon: Fix Mechanical Turk

Observations about Workers

1. Most good workers leave the market because due to absence of reputation system the wages of good and bad workers is same.With so low wages, good workers leave the market. At the end, the only Turkers that remain in the market are the bad ones(or the crazy good ones willing to work for the same payment as the bad workers.)

2. Workers need a trustworthiness guarantee for requesters in order to work on big requests and more efficiently as they have a fear of mass rejection.The requesters should not be able to reject good work , they should be able to reject only spams.

3. Currently the workers are highly restricted by the current interface, in their ability to find tasks. Workers cannot search for a requester.Every worker is not the same in term of abilities so he should be able to search the tasks according to their abilities and some recomendation system to recommend tasks on the basis of past history.

Observations about Requesters

1. There are few big requesters and a long list of small requesters that are posting tasks because in order to get good results, a requester needs to:

  (a)build a quality assurance system from scratch, 
  (b) ensure proper allocation of qualifications, 
  (c) learn to break tasks properly into a workflow, 
  (d) stratify workers according to quality, 
  This needs a lot of time and effort for even small requests. Hence requesters need a better interface to post tasks .

2. Requesters need a reputation system for workers because then they cannot differentiate good from bad workers, they tend to assume that every worker is bad. Hence absence of this effects the quality.

Do Needfinding by Browsing MTurk-related forums, blogs, Reddit, etc

1. The requesters need an interfeace which could be flexible to all kind of people and tasks i.e even not so qualified person could easily post difficult tasks.


2. Workers needs a better reputation rating system. It should not be based on "Number of completed HITs" and "approval rate". It should take into account

  a) Public qualification tests.
  b) Keep track of working history.
  c) Allow rating of workers.
  d) Disconnect payment from rating.
  e) Separate HITs and ratings by type.

3. Workers like to work for old and trust worthy requester than to new requesters due to the fear of rejection and to avoid the fraud requesters.


Synthesize the Needs You Found

Worker Needs

  • Money is definitely a motivating factor.Evidence:As pointed out it in the article:Being a Turker in Example 1:"larak56:I agree with most everyone here. While I do find some of the HITS fun and actually learn an incredible amount by doing HITS, I do it for the cash." Interpretation:Even though he finds it fun money is definitely one of the major factors at play here.
  • They appreciate communication with requester but not over-communication.Evidence:Buffy:Evidently he learned from earlier experience. I have been doing work for him since last month. He is just super nice, normally online while the work is going on and answers any questions right away.The hits are now paying .15 each. My favorite requester.Interpretation:This worker enjoys that the requester responds when he/she asks questions and approves the work fast.
  • The honest(who work sincerely and donot mess around for fast money) need to have some kind of assurance of accpetance.
  • Workers need a better reputation rating system. It should not be based on "Number of completed HITs" and "approval rate".

Requester Needs

  • There’s a requirement to validate the IDs of people participating as workers on Mturk. Various social science researches require that demographics of the workers are known. There’d been cases where a survey is to be filled by young mothers but various males fill it up messing up with the data.(Panel 2)Interpretation:The requesters need the work done to be verified as this can lead to them getting a wrong data for their research or their work.

  • Requesters need a better interface to post tasks.Evidence:In the blog:A Plea to Amazon: Fix Mechanical Turk,it clearly pointed out in the section of Requester's needs to have a better interface to post tasks.