Milestone 2 bb

From crowdresearch
Jump to: navigation, search

Attend a Panel to Hear from Workers and Requesters

Deliverable

Notes from the panel:

- Hard to find a single worker to work on something, they don't want to send the request out to all the turkers, there's no easy way to find a specific worker (kind of hacking mturk into desk system)

- microtasks are good, give out a few tasks that you already know the answer to, to test the experience and knowledge of the worker

- workers should be able to work on tasks from anywhere

- communication methods should be better between workers and requesters. Workers email requesters for tasks and questions.

- requesters and workers need to find a way to work well with each other so that the quality of the work is good.

requesters have to learn to understand from the workers' side and give interesting tasks and appropriate time periods.

Reading Others' Insights

Worker perspective: Being a Turker

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

- The primary reason for turking is to earn money.

- Although it may seem like workers are performing tasks for lower prices because they are innovative, creative, and fun, the key motivation for these tasks is still money.

- Being a full-time or part-time worker can influence the amount of money a worker can earn.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

- Requestors have better information on the Turkers than vice versa and hence have more power

- Requestors also have the power to reject HIT submissions and to block turkers, creating a damaging effect for the approval rating of a turker. When they fail to acknowledge the relationship and the emotional and moral involvement turkers have, they are criticized on forums.

- Requestors can fall for scams and cheats if they pay less per HIT and design their tasks horribly and lack organization

- Requestors are often assessed on how well they communicate with workers their willingness to work with Turkers. Only nurturing and supporting relationships will help them get their tasks finished efficiently and properly (turkers will be willing to work for you)

Worker perspective: Turkopticon

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

- allows workers to create and use reviews of employers when choosing employers on AMT.

- Turkopticon supports a thriving collective of workers engaged in mutual aid.

- Once a worker submits work, the employers have the decision of whether they want to pay for it or not.

- workers have no legal recourse against employers who reject work and then go on to use it

- workers' dispute messages become signals to employers

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

- requestors have a lot of power including choosing which workers they want (based on criteria), the structure of the data that is inputted, and whether or not they want to pay them or not. As a result, some requestors misuse their power and cause wage theft , taking advantage of workers. Though requestors may not pay the workers, they have full authority over the submissions of the workers

- though workers can contact requestors to resolve dispute, Amazon does not require requesters to respond, so many do not. They can sustain the loss of workers because there are so many of them

- Communicativity, generosity ,fairness and promptness are qualities that requestors need in order to have happy workers willing to work for them

Requester perspective: Crowdsourcing User Studies with Mechanical Turk

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

- the diversity and unknown nature of a worker is both a benefit and a drawback

- the inspiration of the system is to have the workers complete tasks that are hard for computers to perform.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

- design of the task can determine whether workers will try to look for loopholes in order to cheat and get more money or whether they will give accurate and good quality answers

- the introduction of quantitative and verifiable questions in experiment 2 resulted in less invalid comment responses compared to experiment 1 -- experiment 2 matched up with expert ratings more closely!

Requester perspective: The Need for Standardization in Crowdsourcing

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

- workers recruited from online labor markets

- poor performance by workers can cause them to lose their jobs

- workers within firms are those who have been screened, trained for the job, and have incentives for good performance

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

- Employers that have been in the crowdsourcing business for a long learn from their mistakes and fix the design problems, whereas newcomers have to learn the lessons of bad design only after experience

- Requestors have to place a set price on a task without knowing how conditions of the market is

- there is no incentive to fix the negative externalities

Both perspectives: A Plea to Amazon: Fix Mechanical Turk

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

- requesters are free to reject and not pay for the work they get to keep.

-requesters don't have to pay on time

-workers can find tasks they want to work on

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

- In order for requestors to have better designs of tasks, there needs to be change on the interface itself. Some people are unable to create tasks themselves because it is not that user-friendly. Due to workers willing to work for employers that have good design, those who do not have experience with the interface are unable to get big tasks done

- Some requestors can have the tendency to use work for free without paying workers -- there needs to be a new system implemented to create trust in requestors

Do Needfinding by Browsing MTurk-related forums, blogs, Reddit, etc

- people are looking for legitimacy

- a better rating system for requesters and workers

- accurate ratings specific to

- way to know if a requester is underpaying/ unfair

- open communication between workers and company

Synthesize the Needs You Found

List out your most salient and interesting needs for workers, and for requesters. Please back up each one with evidence: at least one observation, and ideally an interpretation as well.

Worker Needs

- ensure legitimate work can't be rejected by a requestor

- if requestor is underpaying or being unfair, they should have the power to rate them, once they reach a certain rating, they should be removed

- open communication between workers and the company, an open forum for a task, not have that much of a disconnect

- workers think that the limited access between the company and worker is a big reason why projects fail

- accuracy scores should be tied to HIT's

- if a worker is good at one thing but not another, each should have it's own score, not have it combined, so requestors can view specifically what they want

- better rating system because it's hard to reach 99% accuracy (as requested by the workers), rejections are sometimes too harsh

Requester Needs

- mark a HIT done as incorrectly, but still pay for the situation (human error)

- need a better system for rejections, many requesters don't reject, and if you do, they are labeled as bad requesters.

- list what demographic the worker is looking for in the hit description as well as a small eligibility survey, so requesters don't need to find out that they're ineligible later

-pay at least minimum federal wage ($0.13/minute)

Source: http://rh.reddit.com/r/mturk/comments/2clggw/i_am_a_requester_on_mturk_what_suggestions_do/