Milestone 2 pnp

From crowdresearch
Jump to: navigation, search

Reading Others' Insights

Worker perspective: Being a Turker

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

Turkers have different levels of aims/objectives as is evident from the paper. On the whole most of them desperately turk for cash (though it is a low pay job) while some of them claimed that they do it for fun i.e they found some HIT’s to be fun,interesting and educational. We cannot make conclusions on average amount of cash they earned as we don’t know how much time they spend on AMT turking. Some facts of interest are:

  1. Turkers are interested in comparison to gain information and knowledge: they start threads like this so they can understand where they are in relation to others and what earning potential they might reach and how quickly if they spend more time or progress to greater proficiency.
  2. They set themselves targets: e.g. to make $10 per day
  3. The importance of their AMT income varies depending on earning ability and other life circumstances: as has been noted elsewhere, for some, AMT is their primary source of income, for others it is supplementary - sometimes to buy something special, sometimes to buy vital everyday things. We can also see that some turkers live in different circumstances where AMT work is vital for generating income to meet their basic needs. AMT will form a safety undernet for such digitally underclass people.
  4. Interestingly “the current federal minimum wage is $7.25 per hour. If a minimum wage worker is employed full-time (forty hours per week for 52 weeks), that worker would earn $15,080 annually7”.
  5. They expect the requestors to respond to them quickly and also they don’t expect themselves to get exposed to media.

An important thing to realise is that broadly speaking the Turkers on Turker Nation fall into two categories, novice and experienced. Novices have to concentrate on getting their numbers up for both their approval rating (>90%) and their HIT count (1000 or even 5000). It is not true that crowdworkers are “perfectly happy” to work for extremely low pay. Equating a wage that people will accept with a wage that they are perfectly happy with is a fallacy unless they have sufficient information, opportunity and choice. Some key findings are that they treat their activities as work where pay is the most important factor and that they understand and orient to AMT as a labour marketplace.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

In detail there can be two types of requestors. ‘A good requestor’ and a ‘A bad requestor’ . A good requestor is the one who approves all HITS and gives a fair pay for a loyal turker for his/her tasks.

  • When Requesters fail to acknowledge the relationship and the emotional and moral involvement Turkers have, there are consequences for both parties .
  • This will lower the rating of turkers ( Indirectly this is going to affect their future opportunities ). Turkers are understandably offended when Requesters reject HIT submissions for reasons they do not understand. This not only deprives them of money they believe they have rightly earned, but it has a damaging effect on their approval rating. Such requestors can be called bad requestors.
  • “A high approval rating (90% or above) is a key metric for getting access to better paid and more professional, interesting work. Faulty judgments by Requesters result in unfair treatment of Turkers, and can rob them of pay for completed work and access to future work through potentially no fault of their own.”
  • Turkers may be more likely to come to negative conclusions when Requesters do not play by the ‘rules’ of good practice that Turkers orient to such as: designing tasks well; pricing them fairly; providing rapid approval and payment; responding to communication; and being polite. Lack of information seems to increase the adversarial tension between Turkers and their Requesters.
  • There should be few qualities of requestors which will beneficial to both parties as turkers are dependent on requestors
  1. Direct, open, polite, and respectful communication is highly valued.
  2. A good requestor is a honest and good communicator.

Worker perspective: Turkopticon

Turkopticon, an activist system that allows workers to publicize and evaluate their relationships with employers.

Turkopticon allows workers to create and use reviews of employers when choosing employers on AMT. It supports a thriving collective of workers engaged in mutual aid, brought together by our simple browser extension and web- based technology.

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

Turkopticon started Workers’ “Bill of Rights” to provoke workers imaginations about the infrastructural possibilities.

From this they inferred,

  • Few workers felt that their work was regularly rejected unfairly or arbitrarily.
  • Few demanded a faster payment.
  • Some explicitly mentioned a ‘minimum payment’ per HIT.
  • Some mentioned fair compensation.
  • some expressed dissatisfaction with employers and AMT’s lack of response to their concerns.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

Turkopticon collects quantitative ratings from reviewers on four qualities based on the worker’s bill as follows:

How responsive has this requester been to communications or concerns you have raised?
How well has this requester paid for the amount of time their HITs take?
How fair has this requester been in approving or rejecting your work?
How promptly has this requester approved your work and paid?

Requester perspective: Crowd-sourcing User Studies with Mechanical Turk

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • Generally workers will prefer the tasks from requesters who are good and expect to complete tasks which provide higher wage.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • Requestors should establish a proper infrastructure for their project allowing them to easily utilize a crowd and access participants.
  • Requestors should keep in mind about all kinds workers and prepare tasks so as to reach many.

Requester perspective: The Need for Standardization in Crowdsourcing

Standardization would make it easier to set prices, spread best practices, build meaningful reputation systems and track quality.

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • Paid crowd-sourcing gives people in poor countries access to buyers in rich countries, enabling a kind of virtual migration.
  • There can be a lot of mismatch in regard to bad workers to good requestors and vice versa. Standardization addresses such problem.
  • Workers need to learn the intricacies of the interface for each separate employer.
  • Workers need to adapt to the different quality requirements of each employer.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • Requestors can benefit more from standardization of tasks as it can provide, re-usability at true market pricing without all the mess.
  • No employer can publish his own evaluation of the workers that work for his, as this is a signal earned after a significant cost for the employer.

Both perspectives: A Plea to Amazon: Fix Mechanical Turk

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

Workers need:

  • A trustworthiness guaranteed requestors. eg. - Fast response and payments from requestors
  • A better user interface for browsing & searching etc.
  • Good and Bad workers should be segregated and payment should be done accordingly.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

Problems faced by Requester and are never addressed :

  • Scaling up
  • Managing the complex API
  • Managing execution time
  • Ensuring quality of tasks

It is not good for the requestors as well as workers in either ways as both are dependent.

Requesters' Needs :

  • A better interface to post tasks.
  • A true reputation system for workers.

Needs of requesters

  • Efficient ways of ensuring quality of work by workers.
  • Lower commission rates on jobs.
  • Get some kind of guarantee of completion of task successfully within time, penalty(monetary or in terms of rating) on failing to do so.
  • Authenticate workers, who might be dealing with your valuable/sensitive information.

Needs of workers

  • Transparent reviewing process. The employee should be told what the review is about, including weaknesses and challenges to work on and advantages and strengths.
  • Block abusive employers who post job vacancies with rates offered way below decency.
  • Provide a user-friendly framework where workers can quickly find relevant jobs rather than applying for various job profiles and wasting a lot of time.
  • Block scammers, based on reports by other users.
  • Remove restriction on workers that they can only start either with non-relevant jobs or have to pay a deposit amount.
  • In case requester didn't accept the work done by a worker, the requester should get back to the worker on why his/her work was rejected.
  • Cannot find relevant tasks may be due to lack of sufficient ones or bad organization of the platform.
  • Mentorship

Combined needs

  • Monitor fraud either way, to clients as well as to workers.
  • Proper division of power(who will close the task and who will accept/reject the work of a worker?).
  • Transparent, responsive, irrational, accessible, empowered and trustworthy customer care dealing with all sorts of problems, among clients, workers and the framework/platform.
  • Make it easier to create account, avoid asking personal/valuable information.
  • Add ability to rate workers, clients, jobs to filter reliable ones, if multiple workers are required for a job.
  • Make it easier to set up yourself as a new worker/employer as the account review process might extend upto weeks.
  • Standardized payment gateway.
  • Restrict multiple accounts so that everyone is uniquely identified, should be able to see his achievements/failures.
  • Internationalization in every domain, no restriction at all based on location, or other societal discriminations.
  • Restrict extensive bidding as it reduces the cost to below its actual value, thereby mostly resulting in low quality output.
  • Establishment of trust between customers and crowd-source platforms lacks.
  • Use newer technology.

Suggested solutions

  • Monetary worth of jobs posted by any requester must increase progressively, in order to avoid scam. This ensures that highly paid jobs are put-up only by requestors who are trustworthy, i.e., they have shown their commitment to some workers prior to this deal.
  • Tasks can be divided into multiple categories : hourly-basis, milestone-basis or complete. Different payment strategies can be decided by the requester depending on the type of work desired.
  • Conflict issues between requestors and workers should be managed by a powerful middleman, who is impartial to favour either.
  • Prompt customer service helps prevent loss of users. The chances of a user abandoning the platform depends directly on the time it takes to resolve a query.
  • Skill level of any worker should be identified by a rating process, where both requesters and workers rate each other. This helps users to make future decisions better.
  • There should be “Trusted Requestors” and “Trusted Workers” which have earned this “trusted” tag by their history. While giving a task to a trusted worker, you can be sure that he will deliver. Similarly, while taking upon a task from a trusted requestor, you can be sure that he won’t scam you so you can work w/o any risk. This is particularly helpful in cases where the requestor has a strict deadline for his task and can’t afford non-delivery by the chosen worker.
  • There should be enough random behaviour in the platform so as to minimize bot usage. Many people write scripts (e.g. check for new tasks and apply to all) which negatively impact the platform as a whole.
  • Escrow system is a must!
  • Workers tend to prefer a platform which is most favorable to them in terms of monetary value, i.e. which gives them the most earnings for same work. The amount charged by our platform should be kept as low as possible to attract more workers. Having more workers will ultimately result in having more requestors on the platform. (The platform’s success will be dependent on the user base.)
  • to be successful where competition is high, workers tend to lower their hire price which in turn compromises quality (bad for both)
  • it’s the primary source of income for many, so any scam would affect them greatly

One Observation: workers tend to build a network via these platforms from the first few tasks and then contact the clients directly for more tasks outside the marketplace (bad for us). any way to keep people in our system?

We came across a surprising comment from a user:

“Hi Dainis, Nice to meet you! Well, it started when this guy gave me two projects at the same time in oDesk, 1 project went well and paid me $500 but on his second project which is far more complicated than the other one that cost more than $2000. He didn’t pay me! He always tells that his friend, the owner of the second project, told him that he is not interested anymore and wants to cancel it. Even though the project is already at 90% done. All instructions have been completely followed accurately basing from this guy’s instructions. Oh man, I was really frustrated after 2 months, I saw his friend’s domain, its in slightly new web design but the contents and other complicated modifications that my team made is there. It looks like he lied to me after all. So lesson learned guys, do not trust this kind of clients even if they paid you in the first project. Don’t you ever upload your project to a client’s domain. Just upload your project to their domain if the payment is already paid 100%. I think this is the best way for now. Just use your own personal domain so that your client sees the status of your project.”

We need an escrow for code too so that worker doesn't hand over the full code w/o getting paid. so worker hands full code to us, client hands money to us, we’ll give the client an option to check the quality of work w/o giving him the full code, say by remote login (ssh). once satisfied, release both code and funds. This may be the “only” possible way to prevent scam..!

Team Members:

  • Kunal Khandelwal
  • Vinay Chandragiri
  • Swapnil Agarwal