Summer Milestone 9 Evaluations of reputation systems on other crowdsourcing platforms

From crowdresearch
Jump to: navigation, search

Evaluate Platforms

  1. Summary of the reputation system
  2. Identify what is working and what is failing on the crowdsourcing platform.
  3. Find inequality between how requesters and workers are treated by the system.


Ideally, each person will evaluate at least two crowdsourcing platforms. Besides adding your Slack username, all your contributions will be 100% accounted for. Be sure to read and avoid making duplicate content.


Amazon Mechanical Turk

rahulsheth1016 WORK IN PROGRESS

  1. Summary of the reputation system
  • There is a third-party software employed by a lot of the workers called Turkopticon. While you are browsing hits, it allows you to click a button and and look at how other workers have reviewed that requester.
  • It specifically goes over five attributes: "Communicativity", Generosity, Fairness, and promptness. These are all rated out of five points with the ability for someone to have a decimal rating. EG (2.25).
  • In addition to this, it gives you an explanation of the four attributes, how many reviews this is based on, and the ability for you to give the requester a review if they have been your employer before.
  • Turkopticon.png
  • On the other end, requesters can make sure that their workers are qualified by subjecting potential workers to qualification tests. They are devised by the requesters to gauge how well they think a worker will do.
  • In addition, requesters review the work that is submitted by the workers before they pay them to make sure that it's up to par.

Identify what is working and what is failing on the crowdsourcing platform.

Working

  • Turkopticon is decently efficient at giving the whole story about a requester.
  • The usage of a qualification test provides for an individualized assessment of all the workers, making sure that every person is ready for this specialized task.

Not Working

  • Turkopticon is not built into Amazon MTurk so there are workers who have no way of checking up on their requesters.
  • There is no way for requesters to give reviews for workers, which could add for another level of assessment for future requesters.

Find inequality between how requesters and workers are treated by the system.

  • Workers have no way of knowing whether they will get paid.
  • Some tasks don't have qualification tasks so there are certain occasions when a requester cannot ensure worker quality.

Upwork/oDesk

Jsilver (work in progress)

Summary of the reputation system

  • Upwork's 5-star rating system (5RS)is based on a calculated average of six criteria: Skills, Quality, Communication, Availability, Deadlines, and Cooperation.

Identify what is working and what is failing on the crowdsourcing platform

  • The 5RS is subject to rating inflation.
  • Some workers plead for a perfect 5-star rating, sometimes in exchange for (more) unpaid work, often regardless of worker's skills or output and client's satisfaction rate.
  • Client and/or worker can edit his/her feedback/rating for the other party. Feedback change mechanism is triggered by the other party, while the client/worker may or may not proceed with the feedback change.
  • The 5RS criteria is not explained to workers nor requesters.
  • Scammers (those who post fake jobs and those who don't pay for submitted work) are mixed with real clients. Scam workers and those who send spam applications are also present.
  • Needs to toughen stance against scammers. Payment is not 100% guaranteed with hourly projects (despite oDesk claim), if you encounter a scam client who files and wins a dispute. Payment is much less guaranteed when a workers takes on a fixed-price project.

Find inequality between how requesters and workers are treated by the system

  • Seem to favor clients more than workers, especially when it comes to disputes.

Freelancer

Reviewed by @arichmondfuller (WORK in PROGRESS)

Summary of the reputation system

  • On Freelancer both requesters and workers rate each other and leave comments regarding performance by giving 1 to 5 star ratings and written comments as soon as a project or contest is completed and payment has been turned over.
  • Each project page will show a list of bidders that requesters can choose from, with freelancer ratings and bid amounts available at a glance.
  • They give each other 1-5 star ratings related to quality and timeliness.
  • View profile page containing worker reputation elements at https://www.freelancer.com.au/u/BRD11.html
  • Workers profile pages show a 1 to 5 star average rating for all completed projects along with the average price they charge per hour.
  • Workers can outline their education, certificates and qualifications on their profile page.
  • Workers can collect badges to highlight their commitment and skills such as completed 100 projects, made over $1000, provided 500 bits of feedback in the forum, passed English Exam Level 3, etc.
  • Workers can also take exams provided by Freelancer to demonstrate various skills. These exams such as Freelancer Orientation, English Level 3, Numeracy Level 1, Employer Orientation, Worker Orientation, etc. are viewable on the profile page.
  • View this posted task to see elements of requester reputation at https://www.freelancer.com.au/projects/threed-animation/Whiteboard-sketch-simple-character/ The posted tasks show the requesters' 1 to 5 star average ratings along with how many tasks they have posted. You can click through for further details. It also shows how many projects were completed and how many reviews given.
  • Requesters are rated on: Clarity in Specification; Communication; Payment Promptness; Professionalism; Would Work for Again
  • There is also an indication of whether or not the requester's payment method is verified.

Identify what is working and what is failing on the crowdsourcing platform

  • Star ratings are inflated
  • Workers often resort to begging requesters for 5 star ratings at the end of projects.
  • Requesters often withhold payment for tasks.
  • Several scammers operate on this site if we believe the reviews identified on a quick google search.
  • More work needs to be done to verify requester accounts
  • Often worker accounts are a company that has a group of workers who do the work. The star rating may not tell the full story if it's actually several different people doing the work.
  • Because copies of the chat aren't saved by the system, workers often say things that bully the requester and then are immediately erased so there is no evidence of the demands. I imagine this happens the other way as well.

Find inequality between how requesters and workers are treated by the system

  • Because payment at milestones is optional, workers are often uncertain as to whether or not they will be paid in full at the end of a task.
  • Requesters are subject to begging for 5 star ratings upon completion of a task.
  • Workers bully requesters at the beginning of tasks asking for more than 50% paid up front before any work is done.
  • Because the milestone payment process is not formalised on the platform novice requesters at times feel that workers bully them into more payment than deserved being paid up front.

PeoplePerHour

Jsilver +

  1. Summary of the reputation system
  2. Identify what is working and what is failing on the crowdsourcing platform.
  3. Find inequality between how requesters and workers are treated by the system.

Fiverr.com

Reviewed by Claudia Flores Saviaga @claudiasaviaga and other Pumas (WORK in PROGRESS)

  1. Summary of the reputation system.
Review.gif

Feedback ratings are made up a five start rating system and made available to both buyers and sellers after an order is delivered. Once an order is marked as Delivered(Complete) the users have up to 30 days to provide feedback.

The seller. The seller is presented with the following question to rate the buyer: How would you rate your overall experience with this buyer?

The buyer. The buyer is presented with the following questions to rate the seller:

  • Communication With Seller: How responsive was the seller during the process?
  • Service as Described: Did the result match the Gig’s description?
  • Buy Again Or Recommend: Would you recommend buying this Gig?

Comments made while rating will appear on the seller's Gig page with an accompanying work sample when eligible (as part of the Live Portfolio feature). Buyers cannot complete an order without leaving feedback. If nothing is done, the order will autocomplete three days after the delivery date and marked as completed. Buyers can add or remove a work sample to their review.

  • Both buyers and sellers can remove or edit their feedback three days after its placed. After three days, feedback cannot be edited.
  1. Identify what is working and what is failing on the crowdsourcing platform.
    1. Stuff that Work
      1. Showing how much a worker takes to execute a particular task. Let's recall that for requestors one of the main concerns when finding workers is how long they take to execute the task (see e.g., Why is Amazon Mechanical Turk so Popular? )
        See example Fivver profile highlighting a worker's execution time:

Yane.jpg

  1. Stuff that Does not Work
    1. The rating system is based on a 1 to 5 stars, and is public. Rating is completely optional and not necessary to complete an order.
  1. Find inequality between how requesters and workers are treated by the system.
    1. Requestors don't have to show average time it takes them to pay workers.
    2. Requestors don't have space where workers can leave comments on how they treated them.
    3. Requestors don't have to show when was the last time they paid a worker.

Detailed info see: [1]

DigiServed

Jsilver +

  1. Summary of the reputation system
  2. Identify what is working and what is failing on the crowdsourcing platform.
  3. Find inequality between how requesters and workers are treated by the system.

Prolific Academic at https://prolificacademic.co.uk/

Reviewed by William Dai @williamdai; work in progress

Prolific is relatively new. It was started to provide a platform for researchers specifically. Looking at the available jobs, most only had around 25 workers, while the tasks require several hundred.

  1. Summary of the reputation system

Workers are recruited by Prolific and prescreened. Prolific screens potential workers and connects them to requestors based on criteria provided for each task. Requestors are encouraged to reward "diligent participants" to "motivate good performance" and to reject unsatisfactory work when necessary. The requestor pays the company, which then pays the workers. This might alleviate problems stemming from requestors not paying workers. The problem of "fair wages" is also solved by Prolific's stipulation that requestors pay at least $7.25 for every hour's worth of work. The workers must sign up using their Facebook account, or by providing a university email. This would restrict the quality of the workers to those with education, and also restrict bots/spamming (the social media option).

  1. Identify what is working and what is failing on the crowdsourcing platform.

There doesn't seem to be a system in place for rating workers/requestors. It is clear that arguments over payment would not occur, since the second step to setting up a HIT? is to fill the requestor account with money (paying Prolific), and Prolific ultimately distributes the payments to the workers.

  1. Find inequality between how requesters and workers are treated by the system.

Airtasker

  1. Summary of the reputation system
  2. Identify what is working and what is failing on the crowdsourcing platform.
  3. Find inequality between how requesters and workers are treated by the system.

Vayable at https://www.vayable.com

Reviewed by Juechi Zhou @juechi and Xi Wang@alfsonxw

Summary of the reputation system

Vayable is a pioneer in online travel. This website can help you arrange a wonderful trip with a local tour guide. There’s an expert to arrange your accommodation, experience, airport transfers and custom itinerary. If you pay more, you can get better service.

Workers: Vayable Insiders are independent people who create unique experiences to share with others. They are the workers at this platform.

Requesters: Visitors who want to have a nice and unique trip are the requesters here.

In general, there is no strict rating standards at this platform. Everyone at this platform can get a five-star rating.(@alfsonxw and @Juechi both signed up a new account at this platform and both of us have a five-star rating from the very beginning). People at this platform is free to review an Insider. But we cannot give an Insider 1 to 5 star ratings.

In particular, there’s an “Ambassador Program” at this platform to filter qualified Insiders. The requirements for this program are as follows: 

1.At least 5 glowing reviews. 

2.You respond quickly to message an reservation request.

3.A well-curated profile and experience page that includes: 

  • At least 5 high resolution photos
  • A completed profile page (including a photo of yourself)
  • A completed experience page (title, description, itinerary, availability)


Identify what is working and what is failing on the crowdsourcing platform.

working:

a) They provide an “Ambassador Program” to verify Insiders with high quality.

“The Vayable Ambassador Program is a special place in the community to nurture, celebrate and reward our most dedicated and outstanding guides.”

Those who are verified by the Ambassador Program have the following benefits:

  • Improved visibility in search rankings
  • free experience passes (some of which are transferrable!)
  • Exclusive get togethers and dinners
  • Priority for press opportunities as well as mentions in official Vayable communications.

These benefits can incentivize Insiders to work hard and provide more qualified services. And in my opinion, improved visibility in search ranking is the most incentive.

b) Anyone who wants to post an experience (namely, your service for visitors) is asked for a profile video to ensure trust, safety and a high level of quality.

c) Insiders can attach an introduction video to your service and then will get a “video verified” badge to interest more visitors.

Not working:

a) Anyone at this platform is free to review any insiders no matter what the review content is and whether you have transaction with him/her.

b) Anyone can get and keep a 5-star rating with no requirement.

Find inequality between how requesters and workers are treated by the system.

Requesters and workers are treated so unfair at this platform. There is no reputation system for the requesters. Although people have access to see all the reviews (both the review from worker and requester), there is no official rating standard for requesters. We cannot verify a requester, for example, whether his/her review for the workers is useful.

Taskrabbit

  1. Summary of the reputation system
  2. Identify what is working and what is failing on the crowdsourcing platform.
  3. Find inequality between how requesters and workers are treated by the system.

Zooniverse

  1. Summary of the reputation system
  2. Identify what is working and what is failing on the crowdsourcing platform.
  3. Find inequality between how requesters and workers are treated by the system.

Gigwalk

  1. Summary of the reputation system
  2. Identify what is working and what is failing on the crowdsourcing platform.
  3. Find inequality between how requesters and workers are treated by the system.

Thumbtack www.thumbtack.com

  1. Summary of the reputation system
  2. Identify what is working and what is failing on the crowdsourcing platform.
  3. Find inequality between how requesters and workers are treated by the system.