Milestone 2 sundevils

From crowdresearch
Jump to: navigation, search

Attend a Panel to Hear from Workers and Requesters

Deliverable Report on some of the observations you gathered during the panel.

 1.The main motivation for workers is money.
 2. Initial stages of turking is stressful because of difficulty in finding jobs, no proper HITs to start with.
 3. Workers choose jobs not only based on the money but also whether the HIT is ethical or not.
 4. MTurk is pretty difficult to work with due to steep learning curve, no support and bad interface design.
 5. Requesters give HITs to reduce the amount of money spent and to give equal opportunity.
 6. The pay for a HIT is set based on how much time and skill it takes.
 7. Assigning a high pay for a HIT is not advisable since highly paid jobs are interpreted as difficult and requires skill.
 8. The best way to contact a requester is to email them. They do not have time to check the comments in forums.
 
 

Reading Others' Insights

Worker perspective: Being a Turker

1.What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

 1. Turkers work for fun but mostly they do it for the monetary compensation.
 2. Many a turkers do crowdsourcing for hand to mouth monetary support purpose.
 3. Workers tend to laud requester when they approve HIT and criticize when HIT gets rejected.rejected. Turkers have an emotional involvement.
 4. A good engaging design, fast response time are a couple of features turkers expect from requester.
 5. Experienced turkers are concerned about the acceptance rate whereas the novice workers are more concerned about the HIT count.
 6. Turkers do not want media interfering and ruining the freedom of crowdsourcing.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

 1. Not always all workers provide quality work. They should accept it and take responsiility for it.
 2. Requesters expect turkers not to quickly judge and abuse the requester in public forums. 
  

Worker perspective: Turkopticon

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

 1. Unfair relationship exists between worker and requester where the requester is given more power and can reject to pay for a workers effort.
 2. There is a huge gap when it comes to communication between a worker and requester.
 3. Turkopticon is a system that was created to allow workers to publicize and evaluate their relationships with employers. 
 4. Workers want the requester to be based on communicability, generosity fairness and promptness.
 5. Turkopticon brings together diverse range of workers and promotes social interchange and inter dependency.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

 1. Requesters do fail to prioritize workers needs at times due to surplus labor.
 2. The cost and effort of justifying every reject is more than the cost invested for the HIT's.
 3. Tactical quantification must be followed as it is more rational and accurate.

Requester perspective: Crowdsourcing User Studies with Mechanical Turk

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

  1. Workers generally prefer tasks with higher compensation.
  2. Workers tend to game the system by providing random answers just for the sake of it.
  3. Workers needs to fill quantitative answers than just random questions with binary answers.
  4. Since the workers are not experts in the field, it is unfair to expect perfect answers from them.
  5. Workers can be coaxed into doing great by providing more obvious questions initially and then gradually increase the difficulty.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

  1. Requesters can opt for more compensation less workers than less compensation more workers.
  2. It would be benificial to frame the questions in such a way that invalid answers need more effort than the valid ones.
  3. It would be wise to provide questions to the workers such that there is incremental knowledge development with the area they are working on.
  4. Requesters must understand workers are not expert in the domain and they have to scale the questions accordingly.

Requester perspective: The Need for Standardization in Crowdsourcing

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

  1. There is very little innovation on how ratings are given to the workers, how tasks are given.
  2. It is still inadequate for a worker to find a job that suits his talent.
  3. Workers benefit from standardization in a way that, they get uniformly paid, they can get more compensation by market makers if no one is taking up the task. 

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

  1. Requesters pay different wages for different activities of the same kind. eg: Text editing 10 lines costs different from text editing 100 lines. 
  2. Once the task is assigned to a worker, it is very difficult to find when the task would be done or whether the worker can be relied upon. 
  3. Requester has the freedom to create his own set of tasks, pay according to his scale. 
  4. Requesters benefit more from standardizing the tasks by building reliability.

Both perspectives: A Plea to Amazon: Fix Mechanical Turk

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

  1. Need a mechanism to find out the trustworthiness of a requester.
  2. Workers must be able to see certain objective characteristics of a requester like speed of payment, rejection rate etc.
  3. Ability to search for HITs based on requester, most accepted etc.
  4. The HITs should be categorized and indexed so that workers can access the HITs relevant to them.
  

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

  1. Need problems that have been over years to be addressed(Quality, estimated time, API management etc.)
  2. The interface for requester is too complicated. 
  3. If a requester needs quality work done, he needs to take care of lot of things (Quality assurance, qualification of workers, break tasks to workflow etc.)
  4. With reduction in quality, requesters tend to treat good and bad workers alike. 


Do Needfinding by Browsing MTurk-related forums, blogs, Reddit, etc



List out the observations you made while doing your fieldwork. Links to examples (posts / threads) would be extremely helpful.

1. Needfinding is the art of talking to people and discovering their needs.

One scenario was at The Dark Knight Rises: Premier at the Metreon, where the general crowd trend was Large groups of people in lines, though not really milling about. People were either waiting in line for their tickets or in the theater. The observation made was, there was a lot of excitement in the air.This was generally thought of as an event “you will remember.” There was a large “I was there” feeling to being at the premier of TDKR. Even though there were many people there for the event, the lobby area of the theater wasn’t a pleasant or special place any more than usual. It was still just about getting into the movie theater. [1]


2. Needfinding is watching and asking people to learn about their goals and values to be able to uncover user needs and opportunities for improvement [2]

Needs are so important for a variety of reasons. Needs last longer than any specific solution so efficient innovative brands can be made, it rules out creating entirely new offerings that satisfy the need in different ways. Needs provide a clear road-map for development, suggesting the way a process can proceed.Needs are opportunities waiting to be explored and not just guesses about the future trend.Needs spur action, the decision or clear vision is provided so that it is easier now to start catering the user.Once a need has been identified, designers can visualize a problem to be solved. Needfinding is an opportunity to get to know the users way of thinking. It helps to capture the empathy of users and also look for extreme users who define the limits of the needs.


3.Needs of workers and requesters based on needfinding [3]

The workers forums are less professional and lot of stuff not related to the turking concept are posted. But as far as reading through many discussions on the forum, workers are looking for a place where Successful HITs, easy HITs and well paying requester information are posted. Workers are also looking for a place to resolve or get feedback for the technical issues or glitches they face during taking up a HIT.

The requesters forum are simple and very professional.Only relevant posts are being made. The requesters are looking for a place to communicate with the workers and especially co-requesters. Mostly the requesters are looking at discussing about rejecting a worker or how to handle poor quality work and what can be done to increase the speed of results obtained. Discussions about a group of workers and what pay to fix for a task is also discussed widley.


Synthesize the Needs You Found

List out your most salient and interesting needs for workers, and for requesters. Please back up each one with evidence: at least one observation, and ideally an interpretation as well.

Worker Needs:

  1.Workers need a system to check genuinity, fairness and trustworthiness of requester. Evidence: Many workers post the rating for a particular
    requester and share their thought about their experience. Interpretation: Workers need help and feedback from other workers who have already
    taken the HIT. They are looking for information about requesters before working for one.
  2. Better support for beginners. Evidence:Tom posted that initially he was very impressed by the idea of turking and started on it, only to 
     find no HITs for his qualification as a beginner and also very low wage.He eventually quit turking. Interpretation: MTurk and many other           
     Crowdsourcing platforms are not beginner friendly.The Criteria of completing so and so number of HITs before taking up good paying jobs 
     demotivates a beginner and he may quit.
    
  3. A good user interface. Evidence: Manasa has posted in a forum that the requester had rejected her HIT because she failed to include 
    information from a website while completing the HIT. Manasa mentions that the requester did not properly communicate that in his 
    requirements section.Interpretation: Workers expect a good and clear interface where the instructions and keywords for HIT are laid out properly.

Requester Needs:

  1. Reliability that a HIT taken will be completed responsibily. Evidence: Requesters keep insisting that when a worker takes up a HIT, should 
     be able to complete it and within a set time frame. Interpretation: Requesters need results. They do not want a turker to take up a HIT they 
     cannot finish and also they expect a good quality work to be done.
  2. Rejection and make up HIT. Evidence:Bob mentions that a worker did not complete the HIT as per his instructions and is in a dilemma to 
     reject block that worker or to ask for a make up HIT(not paid). Interpretation: Requesters are concerned about giving rejects and blocking.
     Because they dont want to reject a worker unnecessarily and also not get bad ratings.