Difference between revisions of "WinterMilestone 2 Jinesh"

From crowdresearch
Jump to: navigation, search
Line 1: Line 1:
Needfinding was a fun exercise. It was much more challenging than it looks on the surface, but fun. Here is my experience and learning from the Winter Milestone 2
 
 
== Learn about Need Finding ==
 
== Learn about Need Finding ==
 
I discovered this field of Human Computer Interaction with the help of Scott Klemmer's Coursera HCI lectures. Below is a brief summary of these lectures and other lessons in Needfinding.
 
I discovered this field of Human Computer Interaction with the help of Scott Klemmer's Coursera HCI lectures. Below is a brief summary of these lectures and other lessons in Needfinding.
Line 173: Line 172:
  
 
=== Worker perspective: Turkopticon ===
 
=== Worker perspective: Turkopticon ===
 +
 +
*Quick Payment is highest priority for them.
 +
 +
*If a worker's rating drastically falls, it is hard for him/her to pick it back up.
  
*Majority of the workers believe their work is regularly rejected unfairly
+
*Workers also take a risk when they accept a job because their work is compared through an algorithm that may or not be functioning correctly. And if the algorithm is incorrect, they won't get paid.
*faster payments is important to them
+
*Reason and justification of rejection from requesters
+
*want redressal system for ‘unfair’ rejections
+
*Frustration against Amazon for not doing enough to improve the conditions for the workers
+
*Want a greater say in the system
+
*They believe Amazon is partial towards requesters
+
*No unity on critical reform ideas, among the workers
+
  
=== Requester perspective: Crowdsourcing User Studies with Mechanical Turk ===
+
*Valid argument must be present for rejecting a quality work.
  
*Requesters primary motivation is the low cost which they can avail the services of the market place.
+
*Majority of the work done by worker is treated unfairly.
  
*Since there is no single correct answer to most problems it is difficult to identify malicious users while    grading
+
=== Requester perspective: Crowdsourcing User Studies with Mechanical Turk ===
 
+
*Requesters do not know the skill set or the geographical location of the workers.
+
  
*Requesters benefit from the rapid speed of response on the platform
+
*Skill set or the geographical location of the worker is not known to the requestors.
  
*Requesters are frustrated with workers who try to ‘game’ the task
+
*Faster response to the work done leads to high skill growth for worker.
 
   
 
   
*A small minority of workers are responsible for the majority of the invalid ‘gamed’ responses.
+
*Standardization would mean less guess work on the side of both requesters and workers
  
*They have to give a lot of thought in designing the tasks such that in the evaluation phase it is easy to weed out the ‘gamed’ invalid responses.  
+
*Worker can choose to perform tasks of varying difficulty.
  
*Requesters have to unnecessarily add trap sections to the questions to verify the input provided by the turkers.
+
*Workers must learn the intricacies of each requester, including the different interfaces for their tasks and the various quality requirements for each requester.
  
*Detailed and well designed tasks ( with the above said mechanisms built-in ) invite fewer invalid response
+
*Workers are forced to constantly adapt due to the lack of standardization of tasks.
  
*A well designed task takes more time to be completed
 
 
*In-order to generate a high percentage of verifiable results the tasks should have a component of verifiable questions and the description should indicate that the worker’s tasks will be scrutinized.
 
 
*The tasks should be designed in such a way that the time required to complete the task honestly is less than or equal to the time required to complete the task maliciously.
 
  
 
=== Both perspectives: A Plea to Amazon: Fix Mechanical Turk ===
 
=== Both perspectives: A Plea to Amazon: Fix Mechanical Turk ===
'''
 
Requesters:'''
 
 
* Scaling up
 
* Managing complex API
 
* Managing Execution Time
 
* Easier UI: The interface is so complex that the requester needs to recruit a person to manage the the AMT platform
 
* The requesters do not have the tools to differentiate workers (on merit) reliably
 
* They have the same task executed multiple times to weed out spam
 
* Nearly impossible for new requesters to get good results if they post work in big tranches.
 
  
'''Workers :'''
+
* People are demanding for evolution
 +
*Core Structure is weak.
 +
*People’s Problems have never been addressed properly.
 +
*Hands-off approach not properly issued.
 +
*Poor Interface and ill repute system.
 +
*Difficulty with respect to finding full time worker.
 +
*Proof of Authenticity to every new requestor.
 +
*Lack of Interface Standardization.
 +
*Problem regarding things that have to be built from scratch.
 +
*Easy manipulation to worker’s profile.
 +
*No one is trusted due to bad review of some people.
 +
*Redundancy is more.
 +
*As long as the work is done properly, the reputation of the worker simply does not matter.
 +
*Requestor can do whatever they want to do!
 +
*Trial and Error method is followed by the workers for a new requestor.
 +
*New Requestor tries to give a big task but can’t get good results as workers doing that task are spammers and inexperienced which results in less number of new requestors
 +
*Progress of payment not seen by worker.
 +
*Rejections are not verified.
  
* Do not trust new requesters
 
* Experienced workers only do a small amount of work for new requesters, while spammers do not exhibit such behaviour.
 
* Workers cannot search for a particular requester.
 
* Workers use priority queues to pick the tasks to work on.
 
  
 
== Synthesize the Needs You Found ==
 
== Synthesize the Needs You Found ==
Line 236: Line 228:
 
A set of bullet points summarizing the needs of workers.
 
A set of bullet points summarizing the needs of workers.
  
*The workers need a predictable workflow schedule so that they can plan their day and finances in advance. Uncertainty leads to unnecessary mental trauma. Evidence: explicitly mentioned in the panel
+
* No less nor more money should be paid to the worker.Decision should be taken on basis of amount of work they did.
  
* They need a mechanism to ensure their ratings remain above a certain level to be able to avail meaningful employment.  
+
*Need the ability to search HITs according to the requesters who posted them.
  
* Need a trustworthy mechanism to predict the amount of time it will take them to complete a given task.
+
*Complete schedule of payload should be provided to worker prior to the task.
  
* Need to know what kind of tasks are safer for them. Tasks which as individuals they are certain they can complete accurately. This was mentioned in the panel when a worker said that she preferred typing tasks than bubbling tasks to ensure ratings, also in the readings people prefer tasks which are not open to interpretation and have a black and white answer.
+
*Base rating should be established.
  
* They need to have a data dashboard-->tools at their disposal which can give the different kind of information that is not readily available (eg how many rejections can they sustain to maintain their rating, Alert them when certain kind of HITs become available etc). Basically data for which veteran turkers write scripts should be easily achievable, even to the newbies.
+
* Time Requirement factor should be focused more.
  
* Need a mechanism for shielding them from rejections, a redressal mechanism where they can understand why their work was rejected and contest wrongful rejections. A mechanism where the whole process is not so opaque and their dignity and self-worthiness is upheld.
+
* Workers need some leverage over requesters. In the current set up requesters hold all the power in the relationship, and many of the papers noted the lack of some kind of way to rate requesters meaningfully.
  
* Need fair pay for the amount of work they put in.
+
* They need to have a data dashboard tools at their disposal which can give the different kind of information that is not readily available (eg how many rejections can they sustain to maintain their rating, Alert them when certain kind of HITs become available etc). Basically data for which veteran turkers write scripts should be easily achievable, even to the newbies.
  
* Need to know if they are being treated fairly wrt other workers doing similar tasks as evident from Worker perspective: Being a Turker.
+
* Need a mechanism for shielding them from rejections, a redressal mechanism where they can understand why their work was rejected and contest wrongful rejections. A mechanism where the whole process is not so opaque and their dignity and self-worthiness is upheld.
 
+
* Need a good channel of communication with the requesters which ensure that the requesters can feel empathy towards the workers and value their hard work.
+
 
+
* Need a mechanism to trust new requesters, so that they can complete their HITs without running the risk of mass rejection which leads to a drop in their ratings.
+
 
+
*Need the ability to search HITs according to the requesters who posted them.
+
  
 
=== Requester Needs ===
 
=== Requester Needs ===
Line 262: Line 248:
 
A set of bullet points summarizing the needs of requesters.
 
A set of bullet points summarizing the needs of requesters.
  
*The low cost of the platform is very important to the requesters and they need it to remain low. The primary motivation of using the crowd sourcing platform is to be able to get work done at a cheaper rate compared to the real world
+
*Requesters need to access to workers who consistently produce good results. One of the requesters mentioned in the panel that he would actively reach out to communities like TurkerNation to discuss work. Requesters seem to be willing to pay for good results but feel distrustful due to the relative anonymity of the current set up.
 
+
* They need to feel more empathy for the workers,in the present set-up it is very difficult to do so even if they want to because at the end of the day they are interacting with serial numbers. At the same time do not want to be overwhelmed by emails from the workers.
+
 
+
* Need a mechanism to ensure that the description of their HITs is clear to the workers (Panel) to ensure a good percentage of valid results.
+
  
* Need to ensure their ratings are sustained so that they attract good workers.
+
* Authenticity of the work should be verified properly in order to gain the attention of good workers.
  
* Need a better mechanism to rapidly identify malicious answers.
+
* Better System to tackle untrustful answers.
  
* Need to get accurate results on their HITs in the shortest time possible all while interacting with a neat UI which allows them some independence without inducing complexity.
+
* Requesters need to reduce the amount of work they do in order to get good results. Evidence: In the paper, "A Plea to Amazon: Fix Mechanical Turk", the author reported requesters building their own quality assurance system, ensuring qualifications from workers, ranking workers according to quality, etc. Interpretation: Requesters have to invest a lot of time into maintaining the quality of results that they get back from workers.
  
 
=== Contributors ===
 
=== Contributors ===
This milestone was completed by @vinayak
+
This milestone was completed by @jinesh

Revision as of 09:13, 25 January 2016

Learn about Need Finding

I discovered this field of Human Computer Interaction with the help of Scott Klemmer's Coursera HCI lectures. Below is a brief summary of these lectures and other lessons in Needfinding.

observing people to find out their needs, goals and values need: existing problem finding need yields untapped opportunity for design observing people helps build empathy, which is something that I have found to be really important as a data science enthusiast, unless you put yourself in your users shoes and understand their problems you cannot build effective solutions for them.

Deep Hanging out : By spending time with people while they do their work and live their lives you can get beyond the surface of what they say and learn about what they actually do.

What do people do now what values and goals do people have How are these particular activities embedded in a larger ecology similarity and differences across people other type of contexts

Process vs Practice Apprentice :

  • Set up partnership with the people to be observed
  • Be taught the steps in the process
  • observe all of the practices
  • validate what you are observing with those observed as you go along
  • Look for workarounds and hacks
  • Errors are ‘Goldmines’

Walmart lessons: pay attention to what people do rather than what they say don’t ask leading questions


Verbs not nouns Do not mix observation with interpretation


Interviews:

  • chose the participants wisely
  • Approximate if necessary
  • importance of being curious


Attend a Panel to Hear from Workers and Requesters

Deliverable

Attending the panel discussion was very insightful. It allowed me to understand the human perceptive which no amount of reading could have done. After the meeting was over I interacted with some of the workers on slack. The observations from both the meeting and the conversations that followed are listed below

Worker Perspective

cooperation of fellow workers

unpredictable workflow : When do high paying, interesting HITs arrive → Need some timetable : People forced to leave everything and switch to turking, good in the sense that it gives a certain level of freedom, the downside is that you are not guaranteed to find work when you have the time window

Worker rating a big concern : script to predict worst case scenario . Hinders the workers willingness to try out tasks from new requesters email and check with requesters to check out their nature and helpfulness of response to determine if to work on their HIT

  • Need accurate estimation of time needed, given estimation are totally off (deliberate + unconscious inaccurate time reported by Requesters) → solution used: experience + relying on time reported by other workers

Workers reported time also varies person to person ( typing speed etc)

prefer to perform certain kind of tasks based on their abilities (good typing → does not prefer bubble survey)

variable earning → based on how you budget your time ( spending time on small tasks vs wait for big tasks) financial uncertainty a big problem

employ scripts and hacks

Rejection:

  • death for new workers
  • access to decently paying work depends on working
  • Need for feedback, Requesters reject based on intuition → time taken etc
  • share screenshot with Requesters and ask what went wrong and try to rectify that (desperate to avoid rejection)

Describe your experience in 5 words

  • anonymous frustrating complicated tricky
  • absolutely unpredictable great and terrible
  • power frustrating empowering lonely tiering


Some sense of competition between workers to get to the good tasks before others


Requesters Perspective

  • allows rapid deployment
  • important - understand the task to ensure a correct and clear description of the task

they use a kind of prototyping method by releasing a small batch and checking results → depends on workers initiative and participation in alerting and emailing

  • irritated by spam mails from the workers, don’t have time to interact with workers on a large scale

outsource crowdsource

  • reputation of requesters also important to ensure good workers engage with your tasks
  • fairness to both Workers & Requesters needed
  • no personal connect with Workers (interacting with serial numbers)


  • Need ability to build your own template

Rejecting:

  • gold tasks to weed out spammers
  • tend to reject workers in between to show them they are doing it wrong


5 Words to Describe your experience

  • enabling scientific research
  • dynamic futuristic enabling humans
  • power frustrating empowering lonely tiring

Here are some excerpts from the discussion

Reading Others' Insights

Worker perspective: Being a Turker

  • Care a lot about the pay scale
  • angry at someone that suggests that turking can be for fun
  • sceptical of the requesters
  • clear that they are doing it only for the cash
  • In it for the money , don’t really like turking, would be willing to leave for a better paying opportunity
  • flexi timing and anytime access important
  • Have a feeling of being left out, want to know how is everyone else doing
  • feel they do not work hard enough, but are generally positive about their future
  • believe they can improve their earnings by working harder
  • More experienced turker make more
  • Not earning enough
  • desperate for money
  • going through difficult times → Should be a cheerful platform
  • need to know the type of requester they are interacting with
  • eager to avoid bad/low paying requesters
  • acceptance , fair pay important → Money
  • Strong sense of community, want to help other turkers out by telling them which requesters are good
  • In a sense also want to levrage the bond of the community to fight back/ issue a blow to the requesters who have wronged them (in their mind ) and against whom they are generally powerless (No other means of retaliation)
  • Have a sense of pride and want to be respected
  • want to be taken seriously in the forums
  • want other turkers to believe them and follow their advice ( whom to avoid/ not avoid)
  • Requesters feel that workers are lazy and not worthy of the pay.
  • The primary reason of their using AMT is their laziness (Their fault) and could be in a better shape if they took control of their lives
  • frustrated that they cannot get back at the requester
  • want a way to appeal blocking
  • Want to feel empathy from the requesters and want their work to be valued by them
  • Communication with requesters important, want to feel needed
  • Requesters availability and responsiveness important as it not only improves the work speed (questions etc ) but gives a psychological feeling of being needed/valued
  • Want to trust (new) requesters, but are vary as they have a lot to lose (time, potential money , ratings etc )
  • The amount of time it takes to receive approval is important, do not like the uncertainty.

lack of information on the part of the requester is viewed as a big negative

  • Do not care attitude towards requesters well being, but willingness to help if they reach out.

Believe if requesters are cheated it is their own fault

  • Open to changing their minds about requesters
  • Open to admitting fault and not holding a grudge against the requester, if their fault is explained to them and they deem it to be fair
  • workers tend to look at the compensation as a package, The pay combined with the ‘benefits’ of being able to work from home , whenever they want etc.
  • Searching for a good HIT is critical to them; time means money
  • Are sceptical of change in the platform (bought by academics and journalists interfering)
  • Have self esteem and do not want charity/ do not want to be ‘rescued’ by people whom they see to be better off.

Worker perspective: Turkopticon

  • Quick Payment is highest priority for them.
  • If a worker's rating drastically falls, it is hard for him/her to pick it back up.
  • Workers also take a risk when they accept a job because their work is compared through an algorithm that may or not be functioning correctly. And if the algorithm is incorrect, they won't get paid.
  • Valid argument must be present for rejecting a quality work.
  • Majority of the work done by worker is treated unfairly.

Requester perspective: Crowdsourcing User Studies with Mechanical Turk

  • Skill set or the geographical location of the worker is not known to the requestors.
  • Faster response to the work done leads to high skill growth for worker.
  • Standardization would mean less guess work on the side of both requesters and workers
  • Worker can choose to perform tasks of varying difficulty.
  • Workers must learn the intricacies of each requester, including the different interfaces for their tasks and the various quality requirements for each requester.
  • Workers are forced to constantly adapt due to the lack of standardization of tasks.


Both perspectives: A Plea to Amazon: Fix Mechanical Turk

  • People are demanding for evolution
  • Core Structure is weak.
  • People’s Problems have never been addressed properly.
  • Hands-off approach not properly issued.
  • Poor Interface and ill repute system.
  • Difficulty with respect to finding full time worker.
  • Proof of Authenticity to every new requestor.
  • Lack of Interface Standardization.
  • Problem regarding things that have to be built from scratch.
  • Easy manipulation to worker’s profile.
  • No one is trusted due to bad review of some people.
  • Redundancy is more.
  • As long as the work is done properly, the reputation of the worker simply does not matter.
  • Requestor can do whatever they want to do!
  • Trial and Error method is followed by the workers for a new requestor.
  • New Requestor tries to give a big task but can’t get good results as workers doing that task are spammers and inexperienced which results in less number of new requestors
  • Progress of payment not seen by worker.
  • Rejections are not verified.


Synthesize the Needs You Found

List out your most salient and interesting needs for workers, and for requesters. Please back up each one with evidence: at least one observation, and ideally an interpretation as well.

Worker Needs

A set of bullet points summarizing the needs of workers.

  • No less nor more money should be paid to the worker.Decision should be taken on basis of amount of work they did.
  • Need the ability to search HITs according to the requesters who posted them.
  • Complete schedule of payload should be provided to worker prior to the task.
  • Base rating should be established.
  • Time Requirement factor should be focused more.
  • Workers need some leverage over requesters. In the current set up requesters hold all the power in the relationship, and many of the papers noted the lack of some kind of way to rate requesters meaningfully.
  • They need to have a data dashboard tools at their disposal which can give the different kind of information that is not readily available (eg how many rejections can they sustain to maintain their rating, Alert them when certain kind of HITs become available etc). Basically data for which veteran turkers write scripts should be easily achievable, even to the newbies.
  • Need a mechanism for shielding them from rejections, a redressal mechanism where they can understand why their work was rejected and contest wrongful rejections. A mechanism where the whole process is not so opaque and their dignity and self-worthiness is upheld.

Requester Needs

A set of bullet points summarizing the needs of requesters.

  • Requesters need to access to workers who consistently produce good results. One of the requesters mentioned in the panel that he would actively reach out to communities like TurkerNation to discuss work. Requesters seem to be willing to pay for good results but feel distrustful due to the relative anonymity of the current set up.
  • Authenticity of the work should be verified properly in order to gain the attention of good workers.
  • Better System to tackle untrustful answers.
  • Requesters need to reduce the amount of work they do in order to get good results. Evidence: In the paper, "A Plea to Amazon: Fix Mechanical Turk", the author reported requesters building their own quality assurance system, ensuring qualifications from workers, ranking workers according to quality, etc. Interpretation: Requesters have to invest a lot of time into maintaining the quality of results that they get back from workers.

Contributors

This milestone was completed by @jinesh