Difference between revisions of "WinterMilestone 2 fourtypercent"

From crowdresearch
Jump to: navigation, search
(Deliverable)
 
(46 intermediate revisions by 2 users not shown)
Line 1: Line 1:
  
  
== Attend a Panel to Hear from Workers and Requesters ==
+
====Main Observations: Being a Requester====
  
=== Deliverable ===
+
=====Chris=====
  
For the note taking of the panel we used troll to work visually.
 
  
/Users/ferdinandlinke/Library/Group Containers/Q79WDW8YH9.com.evernote.Evernote/Evernote/quick-note/ferlin___Evernote/quick-note-wByEH1/attachment--Wbakro/screenshot.png
+
[[File:Chris_1-2.png]]
  
Report on some of the observations you gathered during the panel.
 
  
Main Observations: Being A Turker
+
[[File:Chris_2-2.png]]
[[File:http://crowdresearch.stanford.edu/w/index.php?title=File:1_Requester_Chris.png]]
+
  
Main Observations: Being a Requester
 
  
Chris (In General):
+
=====Shawn=====
  
What works well:
+
[[File:2 Requester Shawn.png]]
  
  
"one of the best things...as a requester... you can test and make hypothesis very fast"
+
=====Peter=====
  
Often we post a small badge to see if the Instructions make things
 
often they are emailing...
 
it would be better maybe to do it on turkeys nation... "Still a work around here"
 
  
Clearly convey to work what should be done is important
+
[[File:3 Requester Peter.png]]
  
QUALITY AND REPUTATION
 
  
he had someone to do the commounication with the turkers for him
+
====Main Observations: Being A Turker====
now not anymore but needs someone again
+
  
Whis for automatic adjudications system (work could be done by people clearly well on this tasks)
 
  
Workers are represented as Serial numbers! Inhuman! He says this is why they are treated bad
+
=====Chrissie=====
  
REJECTION
+
[[File:4 Requester-Worker Chrissie.png]]
  
being aware of people that work conchíously
 
(has threshold system)
 
  
 +
=====Rachel=====
  
Confident Guy... Knows what he does
+
[[File:Rachel 1-2.png]]
 +
 
 +
[[File:Rachel 2-2.png]]
 +
 
 +
 
 +
=====Laura=====
 +
 
 +
[[File:Laura New 1-2.png]]
 +
 
 +
[[File:Laura New 2-2.png]]
  
 
== Reading Others' Insights ==
 
== Reading Others' Insights ==
  
=== Worker perspective: Being a Turker  ===
 
  
1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.
 
 
2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.
 
  
 
=== Worker perspective: Turkopticon ===
 
=== Worker perspective: Turkopticon ===
  
 +
Worker perspective: Turkopticon
 
1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.
 
1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.
  
Line 76: Line 71:
  
 
Requesters have to carefully design a study/task on AMT. Task which are fast and easy will likely lead to responses with reduced quality. Creating a request is not simple and needs attention to detail.
 
Requesters have to carefully design a study/task on AMT. Task which are fast and easy will likely lead to responses with reduced quality. Creating a request is not simple and needs attention to detail.
 
=== Requester perspective: The Need for Standardization in Crowdsourcing ===
 
 
1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.
 
 
2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.
 
 
=== Both perspectives: A Plea to Amazon: Fix Mechanical Turk ===
 
 
1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.
 
 
2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.
 
  
 
== Do Needfinding by Browsing MTurk-related forums, blogs, Reddit, etc ==
 
== Do Needfinding by Browsing MTurk-related forums, blogs, Reddit, etc ==
  
List out the observations you made while doing your fieldwork. Links to examples (posts / threads) would be extremely helpful.
 
  
 +
Klaus (requester, german entrepreneur who works with crowd working and has many workers from the Philippines; our own interview)
  
 +
:*need for more binding contacts to ensure high quality and reliability (“otherwie they screw you over”)
 +
:*cultural differences (“If you tell a philipino that he did something wrong, he will be ashamed and won’t contact you for two days (because of shame)”)
 +
:*cross-nation (“financial win-win situation, I get it done cheaper and he gets richer. 10 bucks is a lot of money there”)
 +
:*in our talk it seemed like he was defending himself to request turks ethically  -Interpretation down on the paper.
  
 +
== Synthesize the Needs You Found ==
  
 +
=== Worker Needs ===
  
  
 +
'''Chrissie'''
 +
::*more appreciation and respect
 +
::*faster search results / customized
 +
::*dashboard
  
=== Soylent: A Word Processor with a Crowd Inside ===
+
'''Rochelle'''
 +
::*more structured workflow (no planned breaks) / more secure job structure
 +
::*more transparency with employers / need for more security when experimenting with new jobs
 +
::*more influence on ratings
 +
::*better information on the time of tasks (workaround is online forums)
 +
::*more secure job flow (“you don’t wanna use your time for penny hits”, “should I make lunch or work?”)
 +
::*need for more predictiable cash flow (“from pennys to $100”)
  
1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.
+
'''Laura'''
 +
::*need for community and personal/emotional support/ appreciation/ respect  (Turker Nation... the only one where people care for each other / and don't just say you have to work hard (bitter))
 +
::*need for better search function (“searches work via going up and down in the thread”)
 +
::*need for pauses and breaks (“sometimes I can’t take care of my kids because I am in a survey”)
 +
::*more security for new employes (“would never risk to go below 99”)
 +
::*need for better follow up with contacting the requester (“every badge I make sure I have a contact form ... phone number etc. do screenshot of that”
 +
::*better negotation in case of a rejection (“if there is a rejection I go immediately there... try to be polite what can I do there”)
 +
::*need to teach new requesters about the importance of bread works TURKS, to have ratings, maybe they (the requesters) don't understand how that works... etc. there are reasons... but you need that backup ::*OVERALLNEED BEHIND THIS: More intelligent and less crucial Rating System
 +
::*about mechanical turk “Im good there" > need for apreciation and respect
  
There are two types (lazy and eager) of workers, which can be both equally negative for a project.
+
=== Requester Needs ===
  
2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.
 
  
Requesters have to put in a lot of thought into designing tasks. Fast and easy designed tasks will very likely attract lazy turkers who will not complete a task with a high quality. The key is to find a middle path between both. The paper shows that one successful possiblity is to break down a big task (such as proofreading/shortening/optimizing a paper) into many small tasks and have the results verify by multiple turkers (by accepting similar findings).
+
'''Chris'''
 +
::*fast testing of hypothesis
 +
::*quick and dirty feedback for instructions
 +
::*admin for communication with the turkers
 +
::*internal reviewing program / profile
 +
::*more personal contact
  
== Synthesize the Needs You Found ==
+
'''Shawn'''
 +
::*need for more trust (reviewing program/profile)
  
List out your most salient and interesting needs for workers, and for requesters. Please back up each one with evidence: at least one observation, and ideally an interpretation as well.
+
=== Interpretation ===
  
=== Worker Needs ===
+
::*smaller circles build more trust
 +
::*high potential on both sides
 +
::*Enabling work for discriminated people: disabled, etc.
 +
::*more respect for the work itself (there’s a lot of shame on both sites)
  
A set of bullet points summarizing the needs of workers.
 
 
* Example: Workers need to be respected by their employers. Evidence: Sanjay said in the worker panel that he wrote an angry email to a requester who mass-rejected his work. Interpretation: this wasn't actually about the money; it was about the disregard for Sanjay's work ethic.
 
 
=== Requester Needs ===
 
  
A set of bullet points summarizing the needs of requesters.
+
== Milestone Contributors ==
  
* Example: requesters need to trust the results they get from workers. Evidence: In this thread on Reddit (linked), a requester is struggling to know which results to use and which ones to reject or re-post for more data. Interpretation: it's actually quite difficult for requesters to know whether 1) a worker tried hard but the question was unclear or very difficult or an edge case, or 2) a worker wasn't really putting in a best effort.
+
fourtypercent = @ferlin87, @baxterstockman

Latest revision as of 11:00, 24 January 2016


Main Observations: Being a Requester

Chris

Chris 1-2.png


Chris 2-2.png


Shawn

2 Requester Shawn.png


Peter

3 Requester Peter.png


Main Observations: Being A Turker

Chrissie

4 Requester-Worker Chrissie.png


Rachel

Rachel 1-2.png

Rachel 2-2.png


Laura

Laura New 1-2.png

Laura New 2-2.png

Reading Others' Insights

Worker perspective: Turkopticon

Worker perspective: Turkopticon 1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

Workers are being exploit by the requesters. Requesters can reject tasks at any time. If the requesters decides to reject a completed task, they have little possiblity to get paid for a provided service. The requester on the other hand can keep the results. Fast payment is very important (30 days of waiting time is too much). The main issue is fairness. Either, there is a lack of transparency or worker do get exploit. Not many workers are actually complaining to Amazon. In general it seems like they value the possiblity of AMT. Workers do like the possiblity of a platform to exchange experiences (and reviews). A rating/ranking platform can be seen as therapeutic option.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

Requesters want high quality work done cheap and fast. "Just in quality" (instead of just in time) can be seen as a theme.

Requester perspective: Crowdsourcing User Studies with Mechanical Turk

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

In this study, the turkers rated the quality of Wikipedia articles in a simple 7 point Likert scale. The results were then compared with ratings from professionals (wikipedia admins). The results were rather disappointing since the ratings from the admin differt a lot from the turkers. In a second experiment, they used the same setup, however with additional questions about the structure of the article itself (number of images, references, paragraphs) and a summary before they could submit their rating. The rating was closer to the admins' ratings. Also it deceased the number of invalid responses and increased the duration time.

From that we conclude that most of the turkers are looking for simple tasks that can be done fast and easy. Quality here, does not matter much.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

Requesters have to carefully design a study/task on AMT. Task which are fast and easy will likely lead to responses with reduced quality. Creating a request is not simple and needs attention to detail.

Do Needfinding by Browsing MTurk-related forums, blogs, Reddit, etc

Klaus (requester, german entrepreneur who works with crowd working and has many workers from the Philippines; our own interview)

  • need for more binding contacts to ensure high quality and reliability (“otherwie they screw you over”)
  • cultural differences (“If you tell a philipino that he did something wrong, he will be ashamed and won’t contact you for two days (because of shame)”)
  • cross-nation (“financial win-win situation, I get it done cheaper and he gets richer. 10 bucks is a lot of money there”)
  • in our talk it seemed like he was defending himself to request turks ethically -Interpretation down on the paper.

Synthesize the Needs You Found

Worker Needs

Chrissie

  • more appreciation and respect
  • faster search results / customized
  • dashboard

Rochelle

  • more structured workflow (no planned breaks) / more secure job structure
  • more transparency with employers / need for more security when experimenting with new jobs
  • more influence on ratings
  • better information on the time of tasks (workaround is online forums)
  • more secure job flow (“you don’t wanna use your time for penny hits”, “should I make lunch or work?”)
  • need for more predictiable cash flow (“from pennys to $100”)

Laura

  • need for community and personal/emotional support/ appreciation/ respect (Turker Nation... the only one where people care for each other / and don't just say you have to work hard (bitter))
  • need for better search function (“searches work via going up and down in the thread”)
  • need for pauses and breaks (“sometimes I can’t take care of my kids because I am in a survey”)
  • more security for new employes (“would never risk to go below 99”)
  • need for better follow up with contacting the requester (“every badge I make sure I have a contact form ... phone number etc. do screenshot of that”
  • better negotation in case of a rejection (“if there is a rejection I go immediately there... try to be polite what can I do there”)
  • need to teach new requesters about the importance of bread works TURKS, to have ratings, maybe they (the requesters) don't understand how that works... etc. there are reasons... but you need that backup ::*OVERALLNEED BEHIND THIS: More intelligent and less crucial Rating System
  • about mechanical turk “Im good there" > need for apreciation and respect

Requester Needs

Chris

  • fast testing of hypothesis
  • quick and dirty feedback for instructions
  • admin for communication with the turkers
  • internal reviewing program / profile
  • more personal contact

Shawn

  • need for more trust (reviewing program/profile)

Interpretation

  • smaller circles build more trust
  • high potential on both sides
  • Enabling work for discriminated people: disabled, etc.
  • more respect for the work itself (there’s a lot of shame on both sites)


Milestone Contributors

fourtypercent = @ferlin87, @baxterstockman