Milestone 2 ABC

From crowdresearch
Jump to: navigation, search

Panel meeting with workers and requestors


1.Varying working hours for workers on mturk facilitates them.
2.Fluctuations in availability of work on mturk.
3.Invitation to specific people can be sen for performing specific tasks on both mturk and odesk.
4.Requestors usually dont know who their workers are and what sjills they possess.
5. Techniques for screening tasks and workers according to skills are not standardized.
6. Time taken to create test tasks is enormous but can significantly contribute to requestors skill of task creation in a structured and effective manner.
7. People on forums are always ready to share experiences and read those of others to work better.
8. Mturk has severe security restrictions which often make it extremely tough for users feel comfortable especially when it comes to payments.

Reading others’ insights

Observation from the worker’s perspective: Being A Turker

1.Studies prior to this the current paper, show that around 80% of the tasks are carried out by the 20% most active.
2.One third of the respondents have a median income <$10,000.
3.Stemming from the low wages, there has been a large on-going debate talking about why turkers turk, since the wages are extremely low that a full time turker can earn maximum $15k, but surprisingly money was revealed as the primary motivation for a turker in previous studies.
4.In the current paper, the question “why do turkers turk?”, exposed several interesting observations, including:

 4.1. Some turkers said they did the tasks because they found it fun, educational or interesting, which was invariably linked to comments about the HITs paying well.
 4.2 Some explicitly mentioned they had chosen several low paying tasks ust because they found the task enjoyable.
 4.3 Some turkers criticize the idea of ‘HITS for fun’, since it would convey a wrong message to requestor, that they don’t need to pay good wages.

5.One Popular theme of discussion on Turker Nation, the forum studied in this paper, is the role of turkers in promoting fair pay.
6.Talking about “How much do the turkers make?” following results were found:

 6.1.Highest earning around $15k, were made by experienced turkers, this was evident through their posts and high status on the forum.
 6.2.These experienced workers said they take up only well paying and more professional AMT tasks.
 6.3.The lower end of the annual earnings was $50.

7.Apart from the above two central findings, following interesting observations were made:

 7.1. Turkers are interested in gaining knowledge about how to turk better, how much can one earn on AMT annually, how quickly can they be more proficient at turking, how much time they can invest to make a good amount of money and also where do they stand in comparison to other turkers.
 7.2. They set annual targets for themselves.
 7.3. How important their AMT income is to them lies basically on the following two facts:
      7.3.1.Earning ability – if the worker can work elsewhere as a full time professional, or if AMT  is his/ her primary source of income.
      7.3.2.Other circumstances – some people work on AMT to gain some money for specific purposes like buying a gift,etc.

8.Answering “why turkers choose this wage?”, following points were noted:

 8.1. Best workers are aware that they can make an equivalent yearly as working 40 hours a week at minimum wage. But we are unaware of the time spent by these turkers earning this amount. The overall income of an experienced turker satisfies the federal minimum wage criteria, but an important aspect left unconsidered is the time spent by the turker in making that money.
 8.2. Highest money earned at the federal minimum wage of $7.25 per hour, 40 hours per week for 52 weeks is $15,080 annually, which is a nominal amount for a turker in US , while turkers in countries like India might have a different stance on this income.
 8.3. People in full time jobs, with imposed salary cuts, may take up turking as a prt time job. Some are even authorized by their employers to take turking as a side job.
 8.4. What was shocking was that many on AMT make a significantly less amount than $15k a year, but still turk as means of helping the live ‘hand-to-mouth’.

9.Seeking answers to ‘ how do turkers relate to requestors?’, the following observations were made:

 9.1. The prime purpose of turker nations was to find good requestors and avoid bad ones.
 9.2. In most cases, people simply describe their experiences with the requestors.
 9.3. In some, people provide complete history and key components of why they feel a requestor is good or bad.
 9.4. People sometimes provide contrasting assessment. 
 9.5. People discuss the requestors with other turkers.
 9.6. Turkers are offended the most on rejection of their HIT submission by the requestor for two main reasons:
      9.6.1. They are ripped off a fair amount of money they were sure to earn.
      9.6.2. It significantly affects their approval rating, having less than 90% of which won’t qualify them for high paying tasks.
      9.6.3. Hasty judgments by requestors have adverse impact on turkers motivation, earning and reputation.
      9.6.4. Tools like blocking leave sole power into the requestors’ hand, tools like these need to be used with care and concern.

10. Many conversations between the turker and the requestor show that it’s a lot more than sum process of turking and paying for the job. More importantly it was observed, that open, polite direct and respectful conversations are highly valued.
11. “Majority rules” is a common way to try to achieve good quality in tasks like tagging. Turkers consider these types of tasks risky, especially when they are subjective.
12. Turkers may be more likely to come to negative conclusions when Requesters do not play by the ‘rules’ of good practice that Turkers orient to such as: designing tasks well; pricing them fairly; providing rapid approval and payment; responding to communication; and being polite. Lack of information seems to increase the adversarial tension between Turkers and their Requesters.
13. Turkers are use the discussion to work out whether a Requester is good or bad.
14. Turker workplace ethics:

   14.1. Turkers arethought of as having a significant amount of untrustworthy members amongst their ranks.Turkers complain about being unfairly labeled as bots, spammers, and so forth
   14.2. The discourse within Turker Nation around cheating generally works as follows: cheating is generally frowned upon; if a Requester pays an insulting amount per HIT it is no shame if they fall prey to cheats or crooks; if a Requester designs their task badly and leaves it open to scams, that is their problem. All of this seems pretty fair, particularly as Turkers are also on the other end of nefarious activity – fraudster Requesters, badly designed HITs, unjust blocking and suspensions, and insulting pay and comments.
   14.3. Rejections hurt, but a lot of (often cooperative) activity where Turkers seek to make sense of why things have gone wrong and self-critical work is a part of this. They also look for causes that are technical or with HIT design, and Turkers are more tolerant of genuine mistakes, particularly when the Requester seeks to sort them out.

Observation from the worker’s perspective: Turkoption

1. Need a mechanism so that they can communicate with the requesters.
2. They need to share their experiences, ratings and reviews with one another so that they can engage in mutual aid.
3. They need to have a balance between anonymity and reputation
4. They should know about the requesters' qualities, i.e. communicativity, generosity, fairness, and promptness, so that they know under which employer it would be worthwhile to work.
5. They should be able to express their views openly without having to worry about the consequences.
6. They must be respected by the employers and not taken for granted as the crowd-sourcing world is nothing without the humans working as a resource.
7. They require quicker response and reply from the requesters regarding queries, complaints and questions.
8. They need comment moderation from the platform.
9. Workers’ jobs are temporary and poor work culture.
10. Wages barely adhere to the federal minimum wage requirement and hence treat workers in an unfair manner.
11. Added to late payments, was the problem of usual work rejection without genuine reasons. In such cases the time spent by both the worker and requestor resolving the dispute and clarifying the reasons for rejection was much more than the time spent on designing the HITs and achieving the HITs.
12. The workers look for building relationships with the requestors, relationships beyond just payment and turking, which would help the workers understand the requestors need better while himself building faith for the requestor he/ she serves.

Observation from the requester’s perspective: Turkoption

1. They need an efficient mechanism through which they can reply to the questions, queries and complaints of the workers. This is because it is in-feasible for them to have one to one communication with the large community of workers.
2. It requires them to process the ratings and reviews given by the workers carefully in order to improve the jobs and the workforce.
3. Very often they exploit and misuse workers’ conditions and the powers given to them as a requestor.
4. AMT’s design, allowing requestors’ to reject worker’s submission without a justified reason allows a large number of requestors to simply escape payments to the workers without giving any clarifications.
5. AMT’s legal framework gives no rights to workers whose submissions are rejected. There is a severe need to regulate this legal issue.
6. The entire status and image of a requestor is overpowered and hence allows requestors to develop an unethical attitude towards workers and the tasks they perform. There is a need to change the way these communities look at workers and requestors.

Observation from the requester’s perspective: Crowdsourcing User Studies with Mechanical Turk

1. The paper talks about how Mechanical Turk, uses small and low wage tasks and provides employment to large communities. There for the authors discuss the platform on a broad basis talking about its pros and cons, followed by two experiments. The following observations were made regarding the tasks on AMT:

  1.1. Microtasks are a great way to harness human intelligence, and particularly perform tasks like surveys, tagging, rating, etc. which computers can’t perform as efficiently till date.
  1.2. The labor involved in these tasks is very cheap making the full completion of tasks a reasonable deal for requestors.
  1.3. The platform doesn’t provide any  help to the requestor to evaluate the task, hence there is no way to either assure authenticity of work or proper evaluation of work.

2. In general the following inferences can be made about workers from the above mentioned paper:

  2.1.  Workers usually target HITs with high pays and obviously expect most of their submissions to be approved, while some of them perform tasks randomly just in order to finish off a large number of tasks. This can be attributed to the low wage problem, which can only be compensated by performing more number of tasks.
  2.2.  Also there is no way to authenticate workers owing to small amount of information related ot the user, this sometimes leading genuine one to be regarded as bots.
  2.3. In general it was observed that an interesting HIT had as much probability of being chosen to be done by the worker as an easy HIT. 

3. The following inferences can be made about requestors from the above mentioned paper:

    3.1. Submissions are not evaluated and feedback isn’t given to the worker in a way that would be beneficial for the worker as well.
    3.2. Questions should be framed by the requestors keeping the following things in mind:
          3.2.1. workers are not experts hence don’t expect perfect answers, instead try to work upon better questions yourself.
          3.2.2. Questions should be framed with the motive of providing incremental knowledge to the worker, so that he becomes more efficient in the domain he works.

Observation from the requester’s perspective: The need for standardization in crowdsourcing

1. Observation about workers

  1.1. Workers are free to choose any task usually from online crowdsourcing platforms based on the amount of time they can spend on the task, their skill set, money they are aiming for, etc.
  1.2. No legal actions or proceedings take place against the user in case of wrong, incomplete, unsatisfactory submission, but their payments are at stake.
  1.3. High demand tasks usually require low level skills.
  1.4. Current methods of task suggestions to workers based on their skillset and preferences are highly inefficient. This area remains untouched by innovation.
  1.5. Also the existing rating framework is not reliable enough to evaluate workers submissions in a fair and justified manner.
  1.6. There is no existing standardization of user interfaces across platforms, making it extremely difficult for beginners to comprehend new interfaces.
  1.7. A new worker is simply thrown into the market to market himself, unavailability of any beginners tutorial or referral material makes the worker take a long time to get understand how things work. 

2.Observation about requestors

  2.1. The problem of no standardized user interface spans across requestors, taking them a long time to get familiarized with the platform.
  2.2. Also there is not specific format for requestors to submit their HITs in.
  2.3. No standardized payment scale is used. Different requestors may offer different wages for the same task.
  2.4. Requestors are usually unaware of the workers skillset hence having no idea about the quality of work that can be expected.

Both perspectives: A Plea to Amazon: Fix Mechanical Turk

1. Observations about workers

       1.1.Trustworthiness: strong need for the workers to find how much a particular requestor can be trusted. Hence giving them an assurance that they will be paid on time, their work wont be rejected for unfair reasons, the rejection will be justified,etc.
       1.2.Development of a better user interface: often workers are lost in the large number of tasks and end up spending a large amount of their time just seeking perfect tasks among the clutter. This same time can be used by them to perform a larger number of tasks, but only of the user interface could be made more friendly, comprehensible, organized and interactive.
       1.3.Standardization of wages: same task offered by different requestors are paid differently, same task across countries with significantly different economic standards are paid the same. There is a need to bring about standardization in wages.

2. Observations about workers

  2.1.	Development of a better user interface
  2.2.	 A better review system for assessing the workers based on their skillset, in order to facilitate the requestor choose an appropriate set of workers for his tasks.

Do Needfinding by Browsing MTurk-related forums, blogs, Reddit, etc

We have observed several forums including MTurk Grind, Turker Nation but particularly reddit. We were successful in figuring out the following results:
1. People love to talk and share: whether its the turker are requester, there is always an extensive amount of communication going on in the forum.
2. Respondents are extremely active, my first question on reddit received a comment within minutes. I believe this gives a lot of motivation to the new turkers.
3. In general conflicts(particularly those related to payment) can be easily resolved through communication, and these forums provide a lot of that.

Need Synthesis

1. Understanding: The wokrers need to clearly understand what the platform is and what it can be used for and how it works.
2. Knowing: Worker need to know how to make money on such platforms, what amount of money can be made in a given amount of time, what are the chances of work being rejected. Requestors on the other hand need to know how a worker can be keep motivate and inspired towards better performance, and what is the fair amount that should be paid for what quality of work.
3. Evalauating: The worker needs to evaluate which kind of HITs his skills are best for suited for, and the requestors need to evaluate what are the target skills needed for the task he wishes to assign also learn practicing a fair evaluation scheme on the submissions submitted by workers and pay his workers appropriately.
4. Communicating/ Building trust: There is a strong need for requestor and worker to establish communication which further helps establishing a strong trustful bond among the two communities, which are ultimately benefit symbiotically.