Difference between revisions of "Milestone 2 cder"

From crowdresearch
Jump to: navigation, search
(Created page with "Template for your submission for Milestone 2. Do not edit this directly - instead, make a new page at Milestone 2 YourTeamName or whatever your team name is, and copy...")
 
Line 42: Line 42:
  
 
List out the observations you made while doing your fieldwork. Links to examples (posts / threads) would be extremely helpful.
 
List out the observations you made while doing your fieldwork. Links to examples (posts / threads) would be extremely helpful.
 +
 +
 +
**Worker's Needs:**
 +
 +
http://www.mturkgrind.com/threads/lack-of-searching-filtering-and-tracking-options-on-mturk.19824/
 +
 +
Workers need an MTurk functionality which would help them subscribe to preferred requesters and get notifications for the same.
 +
 +
Workers need an MTurk feature to enable searching the website  to find out a specific requester or the hit count.
 +
 +
Workers need to be able to separate hits based on different categories .
 +
 +
http://www.reddit.com/r/mturk/comments/2yj04z/the_account_has_been_unsuspended_so_that_you_may/
 +
 +
Workers need to be patient and develop trust in terms of payment transactions.
 +
 +
Workers need to learn the intricacies of the interface for each  requester.
 +
 +
**Requester's Needs:**
 +
 +
http://www.mturkgrind.com/threads/guideline-for-academic-requesters-on-mturk.26327/
 +
 +
Requester  needs to set the 'Time Allotted' limit for HITs to an amount of time much longer than the expected amount of time.
 +
 +
http://www.mturkgrind.com/threads/guideline-for-academic-requesters-on-mturk.26327/
 +
Requesters need to be clear about bonuses.
 +
 +
A requester needs to use a consent/intro page or paragraphs for his/her HIT request.
 +
 +
http://mturkforum.com/showthread.php?12918-Newbie-Requestor-Question
 +
 +
A requester needs to implement a custom qualification to grade workers and assign complexity to the HIT.
 +
 +
This avoids time wastage as unnecessary extra HITS don’t need to be done.Plus, one can avoid mass rejections which can kill a requester’s reputation.
 +
 
 +
http://www.behind-the-enemy-lines.com/2012/02/need-for-standardization-in.html
 +
 +
Requesters need to implement the “best practices” for each type of work. For example, the designing part)
 +
 +
Requesters need to understand the “Lifetime Approval Rate”
 +
 +
Requesters need to understand rejections and 'hardblocks.'
 +
:http://www.mturkgrind.com/threads/guideline-for-academic-requesters-on-mturk.26327/ (master’s reply)
 +
 +
Every Requester needs to price its work unit without knowing the conditions of the market .
 +
 +
  
 
== Synthesize the Needs You Found ==
 
== Synthesize the Needs You Found ==
Line 49: Line 96:
 
=== Worker Needs ===
 
=== Worker Needs ===
  
A set of bullet points summarizing the needs of workers.
+
Example: Workers need an MTurk functionality which would help them subscribe to preferred requesters and get notifications for the same.
 +
Interpretation: this will increase the efficient on the worker’s side (efforts) as they get to choose whom they wish to work for and It will also save time .
 +
Evidence:Many people upvoted for this suggestion on the link provided in the previous section.
 +
 +
Workers need an MTurk feature to enable searching the website  to find out a specific requester or the hit count.
 +
Interpretation:This feature would save time and effort, it would help a worker choose a specific kind of task and help him keep a record of his/her HITS.
 +
Evidence:Many people upvoted for this suggestion on the link provided in the previous section.
  
* Example: Workers need to be respected by their employers. Evidence: Sanjay said in the worker panel that he wrote an angry email to a requester who mass-rejected his work. Interpretation: this wasn't actually about the money; it was about the disregard for Sanjay's work ethic.
+
Workers need to be able to separate hits based on different categories .
 +
Interpretation: Category based HITs help a worker work efficiently and maximize his output.
 +
Evidence:In many forums it was a suggestion offered by many workers.
 +
 +
Workers need to be patient and develop trust in terms of payment transactions.
 +
Interpretation: The lack of trust arises due to the payment suspension of many Mturk workers. Since MTurk is quite a big and reputed crowd-sourcing platform it would definitely deal with this payment error and rectify it ASAP.
 +
Evidence: In this thread(http://www.reddit.com/r/mturk/comments/2yj04z/the_account_has_been_unsuspended_so_that_you_may/ ) the requester is  having payment issues since his account has been suspended.
 +
 
 +
 
 +
Workers need to adapt to the requirements of each requester and handle the intricacies.
 +
Interpretation: For ease of gaining maximum HITs the worker needs to comply with the working criteria and standards or the requester.
 +
Evidence: Since we know that experienced MTurks are likely to earn better wages on Mturk than the inexperienced , it highlights the fact that those people are aware of how to interact with their each requester and are comfortable working in their environment.
  
 
=== Requester Needs ===
 
=== Requester Needs ===
Line 57: Line 121:
 
A set of bullet points summarizing the needs of requesters.
 
A set of bullet points summarizing the needs of requesters.
  
* Example: requesters need to trust the results they get from workers. Evidence: In this thread on Reddit (linked), a requester is struggling to know which results to use and which ones to reject or re-post for more data. Interpretation: it's actually quite difficult for requesters to know whether 1) a worker tried hard but the question was unclear or very difficult or an edge case, or 2) a worker wasn't really putting in a best effort.
+
Requester  needs to set the 'Time Allotted' limit for HITs to an amount of time much longer than the expected amount of time.
 +
Interpretation: Time duration provided is an important factor in determining the number of workers one wants to get within a specific interval of time. It attracts more workers and the requester earns points.
 +
Evidence:
 +
 
 +
Requesters need to understand the “Lifetime Approval Rate”
 +
Interpretation:When the workers made a  HIT for this requester, and it wasn’t approved for some duration , the requester claimed  it to be "0% (0/0)"  which was much less than the targeted HIT rate margin (90%) set by the requester.
 +
Evidence: http://www.mturkgrind.com/threads/lifetime-approval-rate-misunderstood-by-requesters.27458/
 +
With MTurk support the requester s were able to resolve the problems caused to the workers due to rejection.
 +
 
 +
Requesters need to be  clear about bonuses.
 +
Interpretation: If a bonus is offered,the potential amount (or range of possible amounts) and how to earn it, and how soon workers should expect it to be paid must be mentioned .
 +
 
 +
A requester needs to implement a custom qualification to grade workers and assign complexity to the HIT.
 +
Interpretation: This avoids time wastage as unnecessary extra HITS don’t need to be done.Plus, one can avoid mass rejections which can kill a requester’s reputation.
 +
 +
Requesters need to implement the “best practices” for each type of work. For example, the designing part)
 +
Interpretation: To attract more workers the designing part chosen by the requester should be well enough and legible.
 +
 
 +
Every Requester needs to price its work unit without knowing the conditions of the market .
 +
Interpretation: In order to get the work done the price limit to fetch workers needs to be set.
 +
 
 +
Requesters need to understand rejections and 'hardblocks.'
 +
Interpretation: Requesters  reject a worker as they never intended to pay. Or even when the requester set up the hit incorrectly. The workers are even hardblocked without a  reason and are sent  an Amazon letter stating that another attempt may lead to jeopardy, nothing much can be done about it.
 +
Evidence:http://www.mturkgrind.com/threads/guideline-for-academic-requesters-on-mturk.26327/ (master’s reply)

Revision as of 10:45, 11 March 2015

Template for your submission for Milestone 2. Do not edit this directly - instead, make a new page at Milestone 2 YourTeamName or whatever your team name is, and copy this template over. You can view the source of this page by clicking the Edit button at the top-right of this page, or by clicking here.

Attend a Panel to Hear from Workers and Requesters

Deliverable

Report on some of the observations you gathered during the panel.

Reading Others' Insights

Worker perspective: Being a Turker

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

Worker perspective: Turkopticon

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

Requester perspective: Crowdsourcing User Studies with Mechanical Turk

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

Requester perspective: The Need for Standardization in Crowdsourcing

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

Both perspectives: A Plea to Amazon: Fix Mechanical Turk

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

Do Needfinding by Browsing MTurk-related forums, blogs, Reddit, etc

List out the observations you made while doing your fieldwork. Links to examples (posts / threads) would be extremely helpful.


    • Worker's Needs:**

http://www.mturkgrind.com/threads/lack-of-searching-filtering-and-tracking-options-on-mturk.19824/

Workers need an MTurk functionality which would help them subscribe to preferred requesters and get notifications for the same.

Workers need an MTurk feature to enable searching the website to find out a specific requester or the hit count.

Workers need to be able to separate hits based on different categories .

http://www.reddit.com/r/mturk/comments/2yj04z/the_account_has_been_unsuspended_so_that_you_may/

Workers need to be patient and develop trust in terms of payment transactions.

Workers need to learn the intricacies of the interface for each requester.

    • Requester's Needs:**

http://www.mturkgrind.com/threads/guideline-for-academic-requesters-on-mturk.26327/

Requester needs to set the 'Time Allotted' limit for HITs to an amount of time much longer than the expected amount of time.

http://www.mturkgrind.com/threads/guideline-for-academic-requesters-on-mturk.26327/ Requesters need to be clear about bonuses.

A requester needs to use a consent/intro page or paragraphs for his/her HIT request.

http://mturkforum.com/showthread.php?12918-Newbie-Requestor-Question

A requester needs to implement a custom qualification to grade workers and assign complexity to the HIT.

This avoids time wastage as unnecessary extra HITS don’t need to be done.Plus, one can avoid mass rejections which can kill a requester’s reputation.

http://www.behind-the-enemy-lines.com/2012/02/need-for-standardization-in.html

Requesters need to implement the “best practices” for each type of work. For example, the designing part)

Requesters need to understand the “Lifetime Approval Rate”

Requesters need to understand rejections and 'hardblocks.'

http://www.mturkgrind.com/threads/guideline-for-academic-requesters-on-mturk.26327/ (master’s reply)

Every Requester needs to price its work unit without knowing the conditions of the market .


Synthesize the Needs You Found

List out your most salient and interesting needs for workers, and for requesters. Please back up each one with evidence: at least one observation, and ideally an interpretation as well.

Worker Needs

Example: Workers need an MTurk functionality which would help them subscribe to preferred requesters and get notifications for the same. Interpretation: this will increase the efficient on the worker’s side (efforts) as they get to choose whom they wish to work for and It will also save time . Evidence:Many people upvoted for this suggestion on the link provided in the previous section.

Workers need an MTurk feature to enable searching the website to find out a specific requester or the hit count. Interpretation:This feature would save time and effort, it would help a worker choose a specific kind of task and help him keep a record of his/her HITS. Evidence:Many people upvoted for this suggestion on the link provided in the previous section.

Workers need to be able to separate hits based on different categories . Interpretation: Category based HITs help a worker work efficiently and maximize his output. Evidence:In many forums it was a suggestion offered by many workers.

Workers need to be patient and develop trust in terms of payment transactions. Interpretation: The lack of trust arises due to the payment suspension of many Mturk workers. Since MTurk is quite a big and reputed crowd-sourcing platform it would definitely deal with this payment error and rectify it ASAP. Evidence: In this thread(http://www.reddit.com/r/mturk/comments/2yj04z/the_account_has_been_unsuspended_so_that_you_may/ ) the requester is having payment issues since his account has been suspended.


Workers need to adapt to the requirements of each requester and handle the intricacies. Interpretation: For ease of gaining maximum HITs the worker needs to comply with the working criteria and standards or the requester. Evidence: Since we know that experienced MTurks are likely to earn better wages on Mturk than the inexperienced , it highlights the fact that those people are aware of how to interact with their each requester and are comfortable working in their environment.

Requester Needs

A set of bullet points summarizing the needs of requesters.

Requester needs to set the 'Time Allotted' limit for HITs to an amount of time much longer than the expected amount of time. Interpretation: Time duration provided is an important factor in determining the number of workers one wants to get within a specific interval of time. It attracts more workers and the requester earns points. Evidence:

Requesters need to understand the “Lifetime Approval Rate” Interpretation:When the workers made a HIT for this requester, and it wasn’t approved for some duration , the requester claimed it to be "0% (0/0)" which was much less than the targeted HIT rate margin (90%) set by the requester. Evidence: http://www.mturkgrind.com/threads/lifetime-approval-rate-misunderstood-by-requesters.27458/ With MTurk support the requester s were able to resolve the problems caused to the workers due to rejection.

Requesters need to be clear about bonuses. Interpretation: If a bonus is offered,the potential amount (or range of possible amounts) and how to earn it, and how soon workers should expect it to be paid must be mentioned .

A requester needs to implement a custom qualification to grade workers and assign complexity to the HIT. Interpretation: This avoids time wastage as unnecessary extra HITS don’t need to be done.Plus, one can avoid mass rejections which can kill a requester’s reputation.

Requesters need to implement the “best practices” for each type of work. For example, the designing part) Interpretation: To attract more workers the designing part chosen by the requester should be well enough and legible.

Every Requester needs to price its work unit without knowing the conditions of the market . Interpretation: In order to get the work done the price limit to fetch workers needs to be set.

Requesters need to understand rejections and 'hardblocks.' Interpretation: Requesters reject a worker as they never intended to pay. Or even when the requester set up the hit incorrectly. The workers are even hardblocked without a reason and are sent an Amazon letter stating that another attempt may lead to jeopardy, nothing much can be done about it. Evidence:http://www.mturkgrind.com/threads/guideline-for-academic-requesters-on-mturk.26327/ (master’s reply)