Difference between revisions of "WinterMilestone 2 Jinesh"

From crowdresearch
Jump to: navigation, search
(Created page with "Template for your submission for Winter Milestone 2. '''DO NOT EDIT THIS DIRECTLY''' - instead, make a new page at WinterMilestone 2 YourTeamName or whatever your team...")
 
(Worker Needs)
 
(6 intermediate revisions by the same user not shown)
Line 1: Line 1:
Template for your submission for [[Winter Milestone 2]]. '''DO NOT EDIT THIS DIRECTLY''' - instead, make a new page at [[WinterMilestone 2 YourTeamName]] or whatever your team name is, and copy this template over. You can view the source of this page by clicking the Edit button at the top-right of this page, or by clicking [http://crowdresearch.stanford.edu/w/index.php?title=WinterMilestone_2_Template&action=edit here].
+
== Learn about Need Finding ==
 +
I discovered this field of Human Computer Interaction with the help of Scott Klemmer's Coursera HCI lectures. Below is a brief summary of these lectures and other lessons in Needfinding.
 +
 
 +
*Solving and Identify existing problems.
 +
 
 +
*Finding the environment in which the device or product is going to be used.
 +
 
 +
*Exploring or Finding a need is much more easier task if one can participate in that task(Field Work)
 +
 
 +
*Getting an idea or Getting Conclusions just by hearing out what people talk often about their work is a one piece of puzzle but to get the complete solution, we have to go beyond the surface in order to find out things they do.
 +
 
 +
*Start to find the needs of people in any direction can broaden the path on a later stage by revealing the hidden needs.
 +
 
 +
*Need Finders have a different way of ranking than the normal viz. when a problem occurs, the optimal solution provided is always by the one who has practical experience for that thing and next best solution is provided by the person who is somewhat less experienced but considering this rare case, one who has observed person having best skillsets are at higher rank than those with more experience than this one.
 +
 
 +
*Understand from where you can get your questions properly answered for your needs. Higher Rankings may not always provide you the best answer.
 +
 
 +
*Problems are source of supply for need finders.
 +
 
 +
*Finding loopholes in nearby workspace can really reduce the climbing on the hill.
 +
 
 +
*The lesser you know your tech, higher are the chances for finding exclusive needs.
 +
 
 +
*Conclusions noted on the base of old survey or just what people are saying may lead your need finding task to disasters.
  
 
== Attend a Panel to Hear from Workers and Requesters ==
 
== Attend a Panel to Hear from Workers and Requesters ==
 +
'''Deductions'''
 +
 +
*Requesters find that there's a lot of pressure not to reject bad work, but also can't accept everything
 +
 +
*Some requesters circumvent this by temporarily removing qualifications from bad workers so they can prevent them from sending bad hits without lowering their rating
 +
*Occasionally, language can present a barrier to proper communication (this doesn't seem too common)
 +
 +
*Requesters feel that they don't have enough time to respond to worker messages
 +
 +
*There seems to be no proper process to protest an unfair hit rejection
 +
 +
*Hard for requesters to be sure workers maintain attention span and give good hits
 +
 +
*Willing to pay more for good workers who they trust (often found on forums)
 +
 +
*Sometimes workers provide feedback to help requester improving the HITs
  
=== Deliverable ===
+
*Most of workers has no specific time for work it changes from day to another
  
Report on some of the observations you gathered during the panel.
+
Here are some excerpts from the discussion
  
 
== Reading Others' Insights ==
 
== Reading Others' Insights ==
Line 11: Line 50:
 
=== Worker perspective: Being a Turker  ===
 
=== Worker perspective: Being a Turker  ===
  
1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.
+
* Make requesters give the inner motivations why a work should be done and to show what the job has led the requesters to accomplish, in a way to make workers part of a common adventure.
  
2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.
+
* Increasing the vibes in the turker-requester relationship may lead to better designed HIT’s and support cooperation providing more control with respect to market functionality
 +
 
 +
*Workers should be open minded.
 +
 
 +
*Accepting a mistake may stronger the bond between a worker and requestor.
 +
 
 +
*they need to achieve good results, they need to trust their workers, to feel secure and to collaborate with experienced people.
 +
 
 +
*Turkers believe they can create a balance work community by deciding their requesters, so they don't want interference from government or academy which might cause changes or even closing up the whole crowdsource market.
  
 
=== Worker perspective: Turkopticon ===
 
=== Worker perspective: Turkopticon ===
 +
 +
*Quick Payment is highest priority for them.
 +
 +
*If a worker's rating drastically falls, it is hard for him/her to pick it back up.
 +
 +
*Workers also take a risk when they accept a job because their work is compared through an algorithm that may or not be functioning correctly. And if the algorithm is incorrect, they won't get paid.
  
1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.
+
*Valid argument must be present for rejecting a quality work.
  
2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.
+
*Majority of the work done by worker is treated unfairly.
  
 
=== Requester perspective: Crowdsourcing User Studies with Mechanical Turk ===
 
=== Requester perspective: Crowdsourcing User Studies with Mechanical Turk ===
  
1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.
+
*Skill set or the geographical location of the worker is not known to the requestors.  
  
2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.
+
*Faster response to the work done leads to high skill growth for worker.
 +
 +
*Standardization would mean less guess work on the side of both requesters and workers
  
=== Requester perspective: The Need for Standardization in Crowdsourcing ===
+
*Worker can choose to perform tasks of varying difficulty.
  
1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.
+
*Workers must learn the intricacies of each requester, including the different interfaces for their tasks and the various quality requirements for each requester.
  
2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.
+
*Workers are forced to constantly adapt due to the lack of standardization of tasks.
  
 
=== Both perspectives: A Plea to Amazon: Fix Mechanical Turk ===
 
=== Both perspectives: A Plea to Amazon: Fix Mechanical Turk ===
  
1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.
+
* People are demanding for evolution
 +
*Core Structure is weak.
 +
*People’s Problems have never been addressed properly.
 +
*Hands-off approach not properly issued.
 +
*Poor Interface and ill repute system.
 +
*Difficulty with respect to finding full time worker.
 +
*Proof of Authenticity to every new requestor.
 +
*Lack of Interface Standardization.
 +
*Problem regarding things that have to be built from scratch.
 +
*Easy manipulation to worker’s profile.
 +
*No one is trusted due to bad review of some people.
 +
*Redundancy is more.
 +
*As long as the work is done properly, the reputation of the worker simply does not matter.
 +
*Requestor can do whatever they want to do!
 +
*Trial and Error method is followed by the workers for a new requestor.
 +
*New Requestor tries to give a big task but can’t get good results as workers doing that task are spammers and inexperienced which results in less number of new requestors
 +
*Progress of payment not seen by worker.
 +
*Rejections are not verified.
  
2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.
 
 
== Do Needfinding by Browsing MTurk-related forums, blogs, Reddit, etc ==
 
 
List out the observations you made while doing your fieldwork. Links to examples (posts / threads) would be extremely helpful.
 
  
 
== Synthesize the Needs You Found ==
 
== Synthesize the Needs You Found ==
Line 51: Line 118:
 
A set of bullet points summarizing the needs of workers.
 
A set of bullet points summarizing the needs of workers.
  
* Example: Workers need to be respected by their employers. Evidence: Sanjay said in the worker panel that he wrote an angry email to a requester who mass-rejected his work. Interpretation: this wasn't actually about the money; it was about the disregard for Sanjay's work ethic.
+
* No less nor more money should be paid to the worker.Decision should be taken on basis of amount of work they did.
 +
 
 +
*Need the ability to search HITs according to the requesters who posted them.
 +
 
 +
*Complete schedule of payload should be provided to worker prior to the task.
 +
 
 +
*Base rating should be established.
 +
 
 +
* Time Requirement factor should be focused more.
 +
 
 +
* Workers need some leverage over requesters. In the current set up requesters hold all the power in the relationship, and many of the papers noted the lack of some kind of way to rate requesters meaningfully.
 +
 
 +
*Need a reliable mechanism to trust the requesters they work with. Often this leads them use some external scripts/tools or plugins to ensure requesters ratings.
 +
 
 +
*Workers also try to work with those requesters who they have experience working with in past.
  
 
=== Requester Needs ===
 
=== Requester Needs ===
Line 57: Line 138:
 
A set of bullet points summarizing the needs of requesters.
 
A set of bullet points summarizing the needs of requesters.
  
* Example: requesters need to trust the results they get from workers. Evidence: In this thread on Reddit (linked), a requester is struggling to know which results to use and which ones to reject or re-post for more data. Interpretation: it's actually quite difficult for requesters to know whether 1) a worker tried hard but the question was unclear or very difficult or an edge case, or 2) a worker wasn't really putting in a best effort.
+
*Requesters need to access to workers who consistently produce good results. One of the requesters mentioned in the panel that he would actively reach out to communities like TurkerNation to discuss work. Requesters seem to be willing to pay for good results but feel distrustful due to the relative anonymity of the current set up.
 +
 
 +
* Authenticity of the work should be verified properly in order to gain the attention of good workers.
 +
 
 +
* Better System to tackle untrustful answers.
 +
 
 +
* Requesters need to reduce the amount of work they do in order to get good results. Evidence: In the paper, "A Plea to Amazon: Fix Mechanical Turk", the author reported requesters building their own quality assurance system, ensuring qualifications from workers, ranking workers according to quality, etc. Interpretation: Requesters have to invest a lot of time into maintaining the quality of results that they get back from workers.
 +
 
 +
=== Contributors ===
 +
This milestone was completed by @jinesh

Latest revision as of 11:13, 25 January 2016

Learn about Need Finding

I discovered this field of Human Computer Interaction with the help of Scott Klemmer's Coursera HCI lectures. Below is a brief summary of these lectures and other lessons in Needfinding.

  • Solving and Identify existing problems.
  • Finding the environment in which the device or product is going to be used.
  • Exploring or Finding a need is much more easier task if one can participate in that task(Field Work)
  • Getting an idea or Getting Conclusions just by hearing out what people talk often about their work is a one piece of puzzle but to get the complete solution, we have to go beyond the surface in order to find out things they do.
  • Start to find the needs of people in any direction can broaden the path on a later stage by revealing the hidden needs.
  • Need Finders have a different way of ranking than the normal viz. when a problem occurs, the optimal solution provided is always by the one who has practical experience for that thing and next best solution is provided by the person who is somewhat less experienced but considering this rare case, one who has observed person having best skillsets are at higher rank than those with more experience than this one.
  • Understand from where you can get your questions properly answered for your needs. Higher Rankings may not always provide you the best answer.
  • Problems are source of supply for need finders.
  • Finding loopholes in nearby workspace can really reduce the climbing on the hill.
  • The lesser you know your tech, higher are the chances for finding exclusive needs.
  • Conclusions noted on the base of old survey or just what people are saying may lead your need finding task to disasters.

Attend a Panel to Hear from Workers and Requesters

Deductions

  • Requesters find that there's a lot of pressure not to reject bad work, but also can't accept everything
  • Some requesters circumvent this by temporarily removing qualifications from bad workers so they can prevent them from sending bad hits without lowering their rating
  • Occasionally, language can present a barrier to proper communication (this doesn't seem too common)
  • Requesters feel that they don't have enough time to respond to worker messages
  • There seems to be no proper process to protest an unfair hit rejection
  • Hard for requesters to be sure workers maintain attention span and give good hits
  • Willing to pay more for good workers who they trust (often found on forums)
  • Sometimes workers provide feedback to help requester improving the HITs
  • Most of workers has no specific time for work it changes from day to another

Here are some excerpts from the discussion

Reading Others' Insights

Worker perspective: Being a Turker

  • Make requesters give the inner motivations why a work should be done and to show what the job has led the requesters to accomplish, in a way to make workers part of a common adventure.
  • Increasing the vibes in the turker-requester relationship may lead to better designed HIT’s and support cooperation providing more control with respect to market functionality
  • Workers should be open minded.
  • Accepting a mistake may stronger the bond between a worker and requestor.
  • they need to achieve good results, they need to trust their workers, to feel secure and to collaborate with experienced people.
  • Turkers believe they can create a balance work community by deciding their requesters, so they don't want interference from government or academy which might cause changes or even closing up the whole crowdsource market.

Worker perspective: Turkopticon

  • Quick Payment is highest priority for them.
  • If a worker's rating drastically falls, it is hard for him/her to pick it back up.
  • Workers also take a risk when they accept a job because their work is compared through an algorithm that may or not be functioning correctly. And if the algorithm is incorrect, they won't get paid.
  • Valid argument must be present for rejecting a quality work.
  • Majority of the work done by worker is treated unfairly.

Requester perspective: Crowdsourcing User Studies with Mechanical Turk

  • Skill set or the geographical location of the worker is not known to the requestors.
  • Faster response to the work done leads to high skill growth for worker.
  • Standardization would mean less guess work on the side of both requesters and workers
  • Worker can choose to perform tasks of varying difficulty.
  • Workers must learn the intricacies of each requester, including the different interfaces for their tasks and the various quality requirements for each requester.
  • Workers are forced to constantly adapt due to the lack of standardization of tasks.

Both perspectives: A Plea to Amazon: Fix Mechanical Turk

  • People are demanding for evolution
  • Core Structure is weak.
  • People’s Problems have never been addressed properly.
  • Hands-off approach not properly issued.
  • Poor Interface and ill repute system.
  • Difficulty with respect to finding full time worker.
  • Proof of Authenticity to every new requestor.
  • Lack of Interface Standardization.
  • Problem regarding things that have to be built from scratch.
  • Easy manipulation to worker’s profile.
  • No one is trusted due to bad review of some people.
  • Redundancy is more.
  • As long as the work is done properly, the reputation of the worker simply does not matter.
  • Requestor can do whatever they want to do!
  • Trial and Error method is followed by the workers for a new requestor.
  • New Requestor tries to give a big task but can’t get good results as workers doing that task are spammers and inexperienced which results in less number of new requestors
  • Progress of payment not seen by worker.
  • Rejections are not verified.


Synthesize the Needs You Found

List out your most salient and interesting needs for workers, and for requesters. Please back up each one with evidence: at least one observation, and ideally an interpretation as well.

Worker Needs

A set of bullet points summarizing the needs of workers.

  • No less nor more money should be paid to the worker.Decision should be taken on basis of amount of work they did.
  • Need the ability to search HITs according to the requesters who posted them.
  • Complete schedule of payload should be provided to worker prior to the task.
  • Base rating should be established.
  • Time Requirement factor should be focused more.
  • Workers need some leverage over requesters. In the current set up requesters hold all the power in the relationship, and many of the papers noted the lack of some kind of way to rate requesters meaningfully.
  • Need a reliable mechanism to trust the requesters they work with. Often this leads them use some external scripts/tools or plugins to ensure requesters ratings.
  • Workers also try to work with those requesters who they have experience working with in past.

Requester Needs

A set of bullet points summarizing the needs of requesters.

  • Requesters need to access to workers who consistently produce good results. One of the requesters mentioned in the panel that he would actively reach out to communities like TurkerNation to discuss work. Requesters seem to be willing to pay for good results but feel distrustful due to the relative anonymity of the current set up.
  • Authenticity of the work should be verified properly in order to gain the attention of good workers.
  • Better System to tackle untrustful answers.
  • Requesters need to reduce the amount of work they do in order to get good results. Evidence: In the paper, "A Plea to Amazon: Fix Mechanical Turk", the author reported requesters building their own quality assurance system, ensuring qualifications from workers, ranking workers according to quality, etc. Interpretation: Requesters have to invest a lot of time into maintaining the quality of results that they get back from workers.

Contributors

This milestone was completed by @jinesh