Difference between revisions of "WinterMilestone 2 codexxx"

From crowdresearch
Jump to: navigation, search
Line 156: Line 156:
 
== Do Needfinding by Browsing MTurk-related forums, blogs, Reddit ==
 
== Do Needfinding by Browsing MTurk-related forums, blogs, Reddit ==
  
 +
Afer browsing through all related forums, we concluded that:
 
*Exploring other options
 
*Exploring other options
 
*Follow others
 
*Follow others

Revision as of 09:27, 24 January 2016

In wintermilestone 2, the topic we researched on is needfinding.Thanks to Prof Micheal Bernstein for a great meeting on needfinding and helping us understand the topic.Doing a bit research on needfinding we came to know "needs are verbs not noun",also the converstion between the workers and requesters helped us to know merits and demerits about mturk.So,our this article includes our various studies on different crowdsourcing platfoms and what all observations we were able to make about workers and requesters.

Attend a Panel to Hear from Workers and Requesters

Report on observations

Rochelle:Work- *Functions as both worker and requester.

              *Active on reddit.
              *Enjoys doing work as both worker and requester.
        Difficulties- *Don't get schedule breaks.
                      *Requests are posted anytime.
                      *Edge of the seat waiting for HITS.
                      *Alerts can come anytime about the HITS.
        *select HITS that have good ratings.
        *Hesitates working with new requesters.
       Needs- *Tools similar to (http://crowd-worker.com).

Christie(spam-girl):Work-*Worked as requester for 10 years.

                        *Community manager of turkernation.
                        *Likes to work in mturk.
                  
                   Needs-*need support for workflows.

Xiao:Work-*Role is requester

         *Uses survey datas and behaviour datas to study the workers.
         *Top 10 batches of survey.

Laura:Work-*Joined mturk in 2009.

     Difficulties-*HITS posted in turkernation are not posted in threads.
                  *Very difficult to do other job  if in the middle of 
                   survey.
      *Don't want to dip below (99%) approval.

Chris:Work-*Works as requester.

     Difficulties-*gets request from non-native english speaker.
     Suggestion-*Aspect of design is most important.
                *Workers should understand the task.
                *Work done should be clearly conveyed.
                *The system should have automatical judicial system.

Peter:Work-*Works as requester.

          *HITS designed for productivity.
     Difficulties-*Sometimes workers give crap results.
    *If somebody falls below a particular percentage qualification then the worker qualification is revoked.
    Suggestions-*mturk.

Reading Others' Insights

Worker perspective: Being a Turker

1) Observations about workers

  • Workers does turking to have fun.
  • Workers feel frustrated when they are not able to make enough money.
  • Workers living in different circumstances depend on AMT to make ends meet.
  • Majority of workers(56%) are US based but there is a growing number of Indian Turkers(36%) and other nationalites.
  • Workers consider AMT as a labour market.
  • Money is the best reason why turkers like to do turking on AMT.
  • Turkers on Turkernation are not getting rich but are working in rather low pay work.
  • Workers set themselves targets eg:-to make $10 per day.

2) Observations about requesters

  • Requesters get more benefits from the design of AMT than turkers and there is information asymmetry and imbalance of power between them.
  • Requesters have better information on the turkers than vice-versa as well as greater power of redress.
  • When requesters fail to acknowledge the relationship and the emotional and moral involvement turkers have,there are consequences for both parties.
  • Turkers are understandably offended when requesters reject submissions and there are hasty judgements by requesters result in unfair treatment of turkers and can rob them of pay for completed work and access to future work though they can also block a turker.
  • Threads on requesters often contain assessments of their communication practices and their willingness to work with turkers.While some requester threads are brief and clear.
  • Requester who probably does not understand the requirements and challenges of Turker work and reads the post as cheating.


Worker perspective: Turkopticon

1) Observations about workers

  • Turkopticon allows workers to create and use reviews of employers when choosing employers on AMT.
  • Some workers needed money to pay for rent on groceries and there was another set of workers who did it for fun or to “kill me”.
  • US workers are a significant minority,numbering at 46.8% in recent surveys.
  • AMT brings together crowds of workers as a form of infrastructure,rendering employees into reliable sources of computation.
  • Workers responses to the question of a “Bill of Rights” revealed a range of concerns.Of 67 responses,42 workers raised a number of issues.
 -35 workers felt that their work was regularly rejected unfairly.
 -26 workers demanded faster payment.
 -7 explicitly mentioned a “minimum wage” or  “minimum payment” per HIT.
 -14 mentioned “fair” compensation.
 -8 expressed dissatisfaction with employer’s and amazon’s lack of response to their concerns.
  • Workers enter reviews of employers that they have worked with,entering ratings of four qualities of employers.


2) Observations about requesters

  • AMT helps employers see themselves as builders of innovative technologies rather than employers unconcerned with working conditions.
  • AMT employers define HITS on AMT by creating web-based forms that specify an information task and allow workers to input a response.
  • Employers defines criteria that candidate workers must meet to work on the task.This criteria include the worker’s “approved rating”. Employers can reject work that does not meet their needs,but also enables wage theft


Requester perspective: Crowdsourcing User Studies with Mechanical Turk

1)Observations about workers

  • Users are paid a very small amount upon completion and consisted of over 1 lakh users from over 100 countries.
  • Paradigms for user evaluation tradionally employ far fewer users with more complex tasks which incur higher participation costs.
  • In experiment 1,users had to rate articles on a 7-point likert-scale according to a set of factors including how well written,factually accurate,neutral,well-structured and overall high quality the article was.Users were required to fill out a free-form text box describing what improvements they thought the article needed.
  • In experiment 2,users were required to complete four questions that had verifable quantitative answers before rating the quality of the article.The questions required users to input how many references,images and sections the article had.In addition,users were required to provide a 4-6 keywords that would give someone a good summary of the contents of the article.
  • There is support for qualifying users by using automated pre-tests,or for including or excluding users from future tasks based on their responses to past tasks.


2)Observations about requesters

  • Requesters collect input from users on the web,including tools for user surveys(eg:surveymonkey.com,vividence.com),online experiments and remote usability testing.
  • Micro-task markets offer the requester a way to quickly access a large user pool,collect data and compensate users with micro-payments.

Requester perspective: The Need for Standardization in Crowdsourcing

1)Observations about workers.

  • Workers within firms are employees who have been screened, trained for their jobs and are have incentives for good performance—at a minimum, poor performance can cause them to lose their jobs.
  • For many jobs good performance is very well defined, in that workers must adhere to a standard set of instructions.
  • Workers need to learn the intricacies of the interface for each separate employer.
  • Workers need to adapt to the different quality requirements of each employer.

2)Observations about requesters

  • Every employer has to implement from scratch the “best practices” for each type of work. For example, there are multiple UI’s for labeling images, or for transcribing audio. The long term employers learn from their mistakes and fix the design problems, while newcomers have to learn the lessons of bad design the hard way.
  • Every employer needs to price its work unit without knowing the conditions of the market and this price cannot fluctuate without removing and reposting the tasks.
  • If requesters want to prioritize their own tasks, they can simply price them higher than the current market price.
  • Among both buyers and sellers, one can find scammers; some buyers are simply recruiting accomplices for nefarious activities.

Both perspectives: A Plea to Amazon: Fix Mechanical Turk

1)Observations about workers

  • Workers use priority queues to pick the tasks to work on.
  • Legitimate workers will do a little bit of work and then wait and see. Nobody wants to risk a mass rejection, which can be lethal for the reputation of the worker.
  • Requesters should trust the workers to complete the work, and workers should not have to worry about unreasonable behavior of the workers. This minimizes the search costs associated with finding a trustworthy partner in the market.
  • When requesters cannot differentiate easily good from bad workers, they tend to assume that every worker is bad. This results in good workers getting paid the same amount as the bad ones. With so low wages, good workers leave the market. At the end, the only Turkers that remain in the market are the bad ones

2)Observations about requesters

  • The requester is the buyer in the MTurk market. So, indeed, there should not be a need for a reputation system for requesters, but the workers should be rated.
  • For trustworthiness guarantee, we need to show the rejection rate of the requestor.
  • When new requesters come to the market, they are treated with caution by the experienced, good workers.
  • MTurk is shooting themselves in the foot by encouraging requesters to build their own interfaces and own workflow systems from scratch!


Do Needfinding by Browsing MTurk-related forums, blogs, Reddit

Afer browsing through all related forums, we concluded that:

  • Exploring other options
  • Follow others
  • Acquire multiple job related skills
  • Build trust in an organization
  • Get regular income
  • Get positive and fair rating
  • Earn respect
  • Collaboration with other workers and requestors
  • Receive early feedback on deliverable
  • Motivating workers
  • Get best out of worker
  • Conversation with workers about their lead in the work
  • Understanding mentality of workers and dealing accordingly
  • Posting clear requirements for best quality work
  • Get the best quality work at less cost
  • An easy-to-use interface to maintain work


Synthesize the needs you found

Worker needs:

  • Exploring other options
  • Follow others
  • Acquire multiple job related skills
  • Build trust in an organization
  • Get regular income
  • Get positive and fair rating
  • Earn respect
  • Collaboration with other workers and requestors
  • Receive early feedback on deliverable

Requestor Needs:

  • Motivating workers
  • Get best out of worker
  • Conversation with workers about their lead in the work
  • Understanding mentality of workers and dealing accordingly
  • Posting clear requirements for best quality work
  • Get the best quality work at less cost
  • An easy-to-use interface to maintain work


Contributors

@anmol_agarwal,@paul_abhratanu