Difference between revisions of "WinterMilestone 2 Team-UXCrowd"

From crowdresearch
Jump to: navigation, search
(Milestone Contributors)
(Do Needfinding by Browsing MTurk-related forums, blogs, Reddit, etc)
Line 147: Line 147:
  
 
===Deliverables===
 
===Deliverables===
 +
 +
'''Observations''
 +
 +
Reddit, Mturk grind
 +
 +
- Mturk Relies too much on automation
 +
 +
-  There should be care for workers's concerns, morality needs to be there, workers are imperfect and not mind readers
 +
 +
- Workers end up working without compensation
 +
 +
- There is no standard quality in the way approvals and rejections work. Requesters reject on whim
 +
 +
- All Hits are mixed and finding a hit with appropriate interest levels are too high especially when user wanted to work on academic hits
 +
 +
- There is no ethics board for workers to reach out to
 +
 +
- Unreliable pay
 +
 +
[http://mturkforum.com/showthread.php?33351-Francis-Smart-and-the-1000s-of-Free-Hits-you-should-feel-Privileged-to-Complete UnreliablePay]
 +
 +
- The site is not mobile responsive
 +
 +
- Rating, block or flag requesters
 +
 +
- Sort by rating
 +
 +
- Better power distribution, requesters have too much power
 +
 +
- Reduce repeated tasks
 +
 +
- Separate hits to work in the area of interest
 +
 +
- Hit with a defined pay range
 +
 +
- Instructional videos, support, tutorials and interface
 +
 +
[https://www.reddit.com/r/mturk/comments/2wkrdu/what_would_your_ideal_crowd_work_platform_be_like/ Reddit discussion]
 +
 +
- Require requesters to respond via email
 +
 +
- Be more transparent with worker standards... for example, I recently got Masters. Why? What aspect of my work prompted Amazon to do that? And what is Amazon specifically looking for when they decide to either allow me to keep it or take it away?
 +
 +
 +
[http://www.mturkgrind.com/threads/my-mturk-wish-list-for-amazon.19821/ Mturkwishlist]
 +
 +
 +
- Display more hits per page
 +
 +
-The search engine to be upgraded to allow for searching multiple terms.
 +
 +
-A star or checkbox could be added to HITs on the search page allowing people to mark and save their preferred work.
 +
 +
-Functionality could be added to allow workers to subscribe to preferred requesters and receive notifications upon posting.
 +
 +
-The option could be added for workers to be able to block HITs or requesters from showing up in their search results.
 +
 +
 +
[http://www.mturkgrind.com/threads/lack-of-searching-filtering-and-tracking-options-on-mturk.19824/ Filtering]
 +
 +
 +
 +
 +
 +
- workers should pay attention
 +
 +
- workers should understand the work properly
 +
 +
- Being able to change amount paid to workers on the same hit ( preferably make it higher only)
 +
 +
- Want to view stats properly
 +
 +
- High fees charged and money not reaching the workers, amazon has raised their fees
 +
 +
 +
 +
 +
[http://www.mturkgrind.com/threads/lifetime-approval-rate-misunderstood-by-requesters.27458/ Approvals]
 +
 +
 +
  
 
'''Worker's needs'''
 
'''Worker's needs'''

Revision as of 09:45, 24 January 2016

Attend a Panel to Hear from Workers and Requesters

Deliverable

Observations:

- panel members are actively involving with Turk

- Most of them are run a blog, forum or a different kind of helping channels to help out the Turk society.

- They maintain a communication channel with requesters and recommend them how to choose workers for their tasks.

- Most of them don't have fixed working hours they work whenever they get a good task.

- Chance of getting a task is unpredictable.

- They take extra precautions when they work with a new requester. Because they don't like to get a negative feedback from them.

- it will take a couple of years to have a good worker profile.

- Even a physically disabled person can get a very good income using crowdsource platforms.

- Researchers find it's very easy to create surveys.

- Requesters use their own templates

- Earnings are various day by day. Its cant be predicted.


Interpretations:

- They all agree that there should be a better system.

- Some workers get the requesters contact details when they going to start their work. If they got a rejection they could contact the requester and can get an explanation.

- Estimations will help them to understand about the time frames.


Needs:

- Workers should complete their task with the highest quality.

- Workers should maintain a good profile, in order to get high pay tasks.

- Requesters Should craft their question carefully, so they could get a good answer.

- There should be a workflow system.

Reading Others' Insights

Worker perspective: Being a Turker

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

- The majority of Turkers (56%) are U.S. based.

- The primary reason for Turking is to earn money.

- Unfair rejection of work, slow payment, low wage and lack of communication with requesters.

- Requesters put the cost to workers of poorly designed tasks.

- There are no equivalent means for rating Requesters

- AMT deliberately hides the relationship between Turkers and Requesters

- Some Turkers had made decisions to accept somewhat (but not much) lower pay if a task was more enjoyable

- One of the turker has mentioned that “I would much rather have my salary back and drop turking” because of the low pay.


2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

- Actions can be taken against bad Requesters but the two sides are not treated in an evenhanded manner

- Requesters should understand that nurturing and supporting relationships should be highly beneficial.

- Anonymity does not mean a desire for no communication and cooperation

Worker perspective: Turkopticon

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

- Workers have no legal recourse against employers who reject work and then go on to use it.

- Amazon has little reason to prioritize worker needs in a market with a labor surplus.

- Workers are taking a risk when they accept a task because payments are not guaranteed.

- the workers as contractors subject to laws designed for freelancers and consultants; this framing attempts to strip workers of minimum wage requirements in their countries.

- By hiding workers behind web forms and APIs, AMT helps employers see themselves as builders of innovative technologies, rather than employers unconcerned with working conditions


2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

- Once a worker submits work, the requesters can choose whether to pay for it.

Requester perspective: Crowdsourcing User Studies with Mechanical Turk

1) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

- In certain tasks which involves subjective reasoning, requesters have to go through a large number of answers just to find out that some of them are from malicious users.

- There involves a lot of struggle in structuring and re-structuring of tasks for which there are no predefined templates.

- Running a batch of tasks(using API or command-in-line) is a cumbersome process for many requesters.

- Pro requesters are more successful in achieving better results as compared to their novice peers.

- There is always some confusion regarding the hourly wages assigned to tasks in the minds of requesters.

Requester perspective: The Need for Standardization in Crowdsourcing

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

Both perspectives: A Plea to Amazon: Fix Mechanical Turk

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

Soylent: A Word Processor with a Crowd Inside

What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

- Workers having a basic knowledge of written English who can can support both novice and expert writers.

- Workers are asked to shorten, proofread, and otherwise edit parts of their requesters’ documents on demand.

- Masses of volunteer editors flag spam edits on Wikipedia


What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

- Writers (i.e., the requesters) often turning to other people for help with pragmatics, complex cognition and manipulation tasks, rather than relying on authoring tools

- Expert writers being bound to routinely make style, grammar and spelling mistakes

- Writers executing daunting numbers of non trivial tasks, like making high level decisions while changing the tense of the passage or fleshing out citation sketches into a true references section, across the entire document

Do Needfinding by Browsing MTurk-related forums, blogs, Reddit, etc

Deliverables

'Observations

Reddit, Mturk grind

- Mturk Relies too much on automation

- There should be care for workers's concerns, morality needs to be there, workers are imperfect and not mind readers

- Workers end up working without compensation

- There is no standard quality in the way approvals and rejections work. Requesters reject on whim

- All Hits are mixed and finding a hit with appropriate interest levels are too high especially when user wanted to work on academic hits

- There is no ethics board for workers to reach out to

- Unreliable pay

UnreliablePay

- The site is not mobile responsive

- Rating, block or flag requesters

- Sort by rating

- Better power distribution, requesters have too much power

- Reduce repeated tasks

- Separate hits to work in the area of interest

- Hit with a defined pay range

- Instructional videos, support, tutorials and interface

Reddit discussion

- Require requesters to respond via email

- Be more transparent with worker standards... for example, I recently got Masters. Why? What aspect of my work prompted Amazon to do that? And what is Amazon specifically looking for when they decide to either allow me to keep it or take it away?


Mturkwishlist


- Display more hits per page

-The search engine to be upgraded to allow for searching multiple terms.

-A star or checkbox could be added to HITs on the search page allowing people to mark and save their preferred work.

-Functionality could be added to allow workers to subscribe to preferred requesters and receive notifications upon posting.

-The option could be added for workers to be able to block HITs or requesters from showing up in their search results.


Filtering



- workers should pay attention

- workers should understand the work properly

- Being able to change amount paid to workers on the same hit ( preferably make it higher only)

- Want to view stats properly

- High fees charged and money not reaching the workers, amazon has raised their fees



Approvals



Worker's needs

- Workers have a need to be recognised and have their concerns addressed

- Workers have a need to be compensated fairly when they put in decent amount of time and energy

- Workers have a need for a standard set of regulations for quality and avoid being rejected on whim

- Workers have a need for better ways to communicate with requesters

- Workers have a need to be able to reach out to someone for their grievances

- Workers have a need to be able to separate hits based on their nature ( academic hits vs batch hits )

- Workers have a need to be able to consistently find work and support their families

- Workers have a need to be able to work on hits on the go

- Workers have a need to be able to give feedback for a requester by ratings

- Workers have a need for better power distribution

- Workers have a need to reduce redundancy

- Workers have a need to be supported and given basic training


Requester's needs

- Requester needs to be able to hire serious and committed workers

- Requester needs to be able to rely on workers to pay attention to the given task

- Requester needs workers to understand the work properly before accepting

- Requester need to be able to change the pay per hit to a higher rate without having to create a new hit

- Requesters need to get a proper stat about workers in order to be able to hire qualified workers

- Requesters need the mturk fees to be nominal

- Requesters need genuine workers who do not lie about themselves

Synthesize the Needs You Found

List out your most salient and interesting needs for workers, and for requesters. Please back up each one with evidence: at least one observation, and ideally an interpretation as well.

Worker Needs

A set of bullet points summarizing the needs of workers.

  • Example: Workers need to be respected by their employers. Evidence: Sanjay said in the worker panel that he wrote an angry email to a requester who mass-rejected his work. Interpretation: this wasn't actually about the money; it was about the disregard for Sanjay's work ethic.

Requester Needs

A set of bullet points summarizing the needs of requesters.

  • Example: requesters need to trust the results they get from workers. Evidence: In this thread on Reddit (linked), a requester is struggling to know which results to use and which ones to reject or re-post for more data. Interpretation: it's actually quite difficult for requesters to know whether 1) a worker tried hard but the question was unclear or very difficult or an edge case, or 2) a worker wasn't really putting in a best effort.


Need finding and cultural considerations

Needfinding is the initiation of a Design Process, all the products & are resultant of certain needs, in planning a UX design, we must identify and define the target audience very carefully so that we cover caputre scenarios & context realting to multiple cultures & enviorments.

In the United States, the number 13 is unlucky—even hotels do not have a 13th floor, in Italy, the number 13 is considered as lucky

One classic example is Mac Donald's websites & customization of products based on demographics. Even different websites were carefully designed with different cultural values in mind.

In reference to Hofstede’s Cultural Dimension of Collectivism vs Individualism, Mexico falls under the category of collectivisim. In mexico Family is considered the most important thing in their culture – even more important than money. For example, it is a cultural norm for people to live with their families until they get married. From a design perspective, it’s important to understand these cultural nuances. In this culture, people make decisions based on the opinions of family members and friends.

The most successful design strategy should begin & end the need-finding with proper cultural considerations

[1]

Milestone Contributors

S.S.Niranga @niranga,

Alka Mishra @alkamishra,

Deepika Nayak @dnayak,

Kanwar Arora @kanwar_arora