View source for Milestone 2 cder

Members
*Naman Gupta
*Aditi Mithal



== Attend a Panel to Hear from Workers and Requesters ==

=== Deliverable ===

Report on some of the observations you gathered during the panel.

== Reading Others' Insights ==

=== Worker perspective: Being a Turker  ===

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

=== Worker perspective: Turkopticon ===

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

=== Requester perspective: Crowdsourcing User Studies with Mechanical Turk ===

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

=== Requester perspective: The Need for Standardization in Crowdsourcing ===

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

=== Both perspectives: A Plea to Amazon: Fix Mechanical Turk ===

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

MTurk does not guarantee trustworthiness of requesters and they are free to reject good work and may even refuse to pay.When a new requester comes he has to maintain some reputation and earn trust of the workers.
The worker analyses about any new requester by checking if the requester is a scammer or not and if the requester pays on time. This observation is done in order to avoid trust issues in future. The workers generally avoid newcomer requesters who post big batch of HITs in order to avoid mass rejection so it's better for the newcomer requester to post smaller batch HITs to prove his/her legibility.

A worker's behaviour on seeing a new requester is as follows:
The worker checks the objective characteristics of  a requester including :
a) Speed of payment made by the requester.
b) Rejection rate of the requester based on reports.
c)Appeal rate for requesters.
d) Disallowing ability to reject work which is not a spam : rejecting a work by a requester should be done only when it's considered a spam.
e) API to make the above things accessible.

Workers need a better User Interface: To gain efficiency, minimize the cost involved in a task and reduce the transaction overhead a good UI is important indirectly.

Functionalities a worker needs on MTurk in order to make his/her search for the requester and  a particular category of a job easier can be provided by adding some features(good UI). Some valuess adeed points for  a good UI can be :
a) Having a browsing system with tasks posted based on categories. b) Improving the search engine c) Using a system to recommend HITs.
Everything depends on the requester-worker relation. One needs to trust the other to establish a conducive working environment.



2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

A requester faces a lot of problems in the beginning like:
a) Scaling up.
b) Managing the API and execution time.
c)Ensuring Quality

Requesters need a better Interface that would help them post HIT in an effective manner.
The command-line used earlier proved to be non-ser friendly as it increased complexity so the UI was of a core concern. It was of utmost importance for requesters to a)build a system that ensures quality b) to ensure proper allocation of qualifications and c) ensure proper breakage of tasks into a workflow.

Requesters also need a good Worker review system and reputation system to analyse each worker. Someone quoted : "A market without a reputation mechanism turns quickly into a market for lemons."
It's important for requesters to be aware of the quality of workers to assign and accept HITs accordingy. To have this system:
a) Qualification tests are conducted.
b) Working history of workers is recorded.
c) Workers are rated on their performance.
d)HITs and ratings are separated on different types/categories .
e) Providing payment based on rating should be discontinued.

The question arose was that why not have a two-sided reputation/rating system?
The answer given was indeed true that it's not much needed and it might also create many mutual-admiration schemes. So, this idea of having two-sided rating was rejected.



== Do Needfinding by Browsing MTurk-related forums, blogs, Reddit, etc ==

List out the observations you made while doing your fieldwork. Links to examples (posts / threads) would be extremely helpful.

Many interesting observations were made regarding the needs of the workers and requesters :

MTurk forums are really great for observation, they provide good insights into the workers and requesters need. The workers and requesters get a platform to interact on the forums which proves beneficial.

1)Worker Needs:

a) Workers need an MTurk functionality which would help them subscribe to preferred requesters, categorize HITs ,get notifications for the same and be able to search the site to find a specific requester.
 
b) Workers need to be patient and develop trust in terms of payment transactions.

c) Workers need to trust the requesters.

d)Workers need to trust the requesters and the HITs posted by them.

e) Workers need to have a a better user interface and need to learn the intricacies of the interface for each  requester.


2) Requester Needs:


a) Requester  needs to set the 'Time Allotted' limit for HITs to an amount of time much longer than the expected amount of time.

b) Requesters need to be clear about bonuses and be interactive with the Workers.

c) A requester needs to use a consent/intro page or paragraphs for his/her HIT request.
 
d) A requester needs to implement a custom qualification to grade workers and assign complexity to the HIT. 

e)Requesters need to implement the “best practices” for each type of work. For example, the designing part)

f) Requesters need to understand the “Lifetime Approval Rate”

g) Requesters need to understand rejections and 'hardblocks.'

h) Every Requester needs to price its work unit without knowing the conditions of the market .



== Synthesize the Needs You Found ==

List out your most salient and interesting needs for workers, and for requesters. Please back up each one with evidence: at least one observation, and ideally an interpretation as well.

=== Worker Needs ===

1. Example: Workers need an MTurk functionality which would help them subscribe to preferred requesters and get notifications for the same.
Interpretation: this will increase the efficient on the worker’s side (efforts) as they get to choose whom they wish to work for and It will also save time .
Evidence:Many people upvoted for this suggestion on the link provided in the previous section.
 
2. Workers need an MTurk feature to enable searching the website  to find out a specific requester or the hit count.
Interpretation:This feature would save time and effort, it would help a worker choose a specific kind of task and help him keep a record of his/her HITS.
Evidence:Many people upvoted for this suggestion on the link provided in the previous section.

3. Workers need to be able to separate hits based on different categories .
Interpretation: Category based HITs help a worker work efficiently and maximize his output.
Evidence:In many forums it was a suggestion offered by many workers.
 
4. Workers need to be patient and develop trust in terms of payment transactions.
Interpretation: The lack of trust arises due to the payment suspension of many Mturk workers. Since MTurk is quite a big and reputed crowd-sourcing platform it would definitely deal with this payment error and rectify it ASAP.
Evidence: In this thread(http://www.reddit.com/r/mturk/comments/2yj04z/the_account_has_been_unsuspended_so_that_you_may/ ) the requester is  having payment issues since his account has been suspended.

  
5. Workers need to adapt to the requirements of each requester and handle the intricacies.
Interpretation: For ease of gaining maximum HITs the worker needs to comply with the working criteria and standards or the requester.
Evidence: Since we know that experienced MTurks are likely to earn better wages on Mturk than the inexperienced , it highlights the fact that those people are aware of how to interact with their each requester and are comfortable working in their environment.

6. Workers need to trust the requesters. Interpretation : Once the trust between a requester- worker develops it will be easy to work and get paid off for hard work done. Evidence: http://www.behind-the-enemy-lines.com/2010/10/plea-to-amazon-fix-mechanical-turk.html

=== Requester Needs ===

A set of bullet points summarizing the needs of requesters.

1. Requester  needs to set the 'Time Allotted' limit for HITs to an amount of time much longer than the expected amount of time.
Interpretation: Time duration provided is an important factor in determining the number of workers one wants to get within a specific interval of time. It attracts more workers and the requester earns points.Evidence:http://www.mturkgrind.com/threads/lifetime-approval-rate-misunderstood-by-requesters.27458/
With MTurk support the requester s were able to resolve the problems caused to the workers due to rejection.

2. Requesters need to understand the “Lifetime Approval Rate”
Interpretation:When the workers made a  HIT for this requester, and it wasn’t approved for some duration , the requester claimed  it to be "0% (0/0)"  which was much less than the targeted HIT rate margin (90%) set by the requester.
Evidence: http://www.mturkgrind.com/threads/lifetime-approval-rate-misunderstood-by-requesters.27458/
With MTurk support the requester s were able to resolve the problems caused to the workers due to rejection.

3. Requesters need to be  clear about bonuses. 
Interpretation: If a bonus is offered,the potential amount (or range of possible amounts) and how to earn it, and how soon workers should expect it to be paid must be mentioned . 

4. A requester needs to implement a custom qualification to grade workers and assign complexity to the HIT. 
Interpretation: This avoids time wastage as unnecessary extra HITS don’t need to be done.Plus, one can avoid mass rejections which can kill a requester’s reputation.
 
5. Requesters need to implement the “best practices” for each type of work. For example, the designing part)
Interpretation: To attract more workers the designing part chosen by the requester should be well enough and legible.

6. Every Requester needs to price its work unit without knowing the conditions of the market .
Interpretation: In order to get the work done the price limit to fetch workers needs to be set.

7. Requesters need to understand rejections and 'hardblocks.'
Interpretation: Requesters  reject a worker as they never intended to pay. Or even when the requester set up the hit incorrectly. The workers are even hardblocked without a  reason and are sent  an Amazon letter stating that another attempt may lead to jeopardy, nothing much can be done about it.
Evidence:http://www.mturkgrind.com/threads/guideline-for-academic-requesters-on-mturk.26327/ (master’s reply)