WinterMilestone 3 Reputation@ftw

From crowdresearch
Jump to: navigation, search

A Ranking System

A ranking system that would apply to both micro and macro workers. In this ranking system based on the type of task, the worker would earn points. The number of the points varies according to the complexity of the task, once the points accumulate, there would be three ranks of workers: Novice: 0-300, intermediate: 300-600 and pro workers: 1000+. A requester can choose the class/level of workers (the higher the rank, the more experienced they will be), given the certainty they will have to pay them more. If the time limit is tight but the budget is not, rather than going through all of the tasks one by one to check the quality of the work, the requester could pay extra to get the “platform-guaranteed” best workers on those types of tasks and not spend more time reviewing, while maintaining peace of mind.

Pricing in a ranking system | Introducing a Moderator

We propose a leveling system that is correlated to the worker's wage. In Daemo you can opt to be a “type” of either requester or worker, but you can also act as a moderator. A Moderator is a pro worker (1000+ points), who assesses others’ work, or helps the requesters to make their task better, or helps the newcomers to navigate and improve while doing tasks. A moderator acts as the link between the two, who gets a slice of the overall profit to be made. Assuming Daemo is going to be an international platform, the prices can vary according to the minimum wage of the country of the worker.

Increasing status along with rank

As user gains experience, he or she will gain new features to customize their profile. These features might be the increased ability to introduce custom backgrounds, image icons, external certifications, and so on. Only those achieving a high enough rank will achieve this feature. Initially, it is suggested to have a leader who works with Daemo to have such a profile and embed into the community.

Building social identity: Rip the Dressing

When expertise is recognized by a more senior person, a less experienced user, that is less experienced based upon the proposed ranking system, will be able to claim that user's badge, background, image, etc. to merge as a part of their own profile. They may or may not have the customization access of the other.


Brainstorm for Reputation

How to improve reputation system beyond Boomerang? (read this paper to learn more about Boomerang).

  • Add a small feedback button at the end of a prototyping task for both the worker and the requester.
  • at scale, does it matter? the person will get new opportunities without a rating
  • use knowledge from boomerang to corral users into skill set groups
  • specialize: use knowledge from boomerang to give people shorter tasks rather than delays. it results in the same thing but maybe it will lessen the cognitive load.
  • use boomerang to establish foundational relationships between worker-requestor that progress to bigger things
  • show losers the door to another site(MTurk)
  • open opportunities for people to collaborate or lose it
  • allow people to see successes beyond cash flow
  • allow newcomers to become reviewers
  • enable proportionality --- people should not do exceptionally more than they have done in the past. baby steps.
  • if a person doesn't complete a task, find task properties that led to failure (language, questions, quantity,...)
  • match task type set by requestor with the current intent of the user.
  • the design should get people to work together more incrementally and focus on implementation goals as an heuristic
  • User profiles that updating a profile picture, bio and contact information is mandatory.
  • Professional social media accounts such as Linked in and Twitter can be linked to the user profile

How would reputation work for new comers - workers or requesters?

  • newcomers just enter, let them explore
  • have the system choose tasks that they cannot fail at and incrementally become more difficult
  • offer professional paths based upon skill that match with different tasks
  • let them self-rate and go through the system
  • requestors would need to be informed with case studies about what doesn't help workers
  • requestors should be exposed to the turkers good work practices
  • workers have coaches tied to skill, system knowledge
  • both should be introduced to the system
  • both should be introduced to a short video of the typical other
  • interactions should be left to questions only with experts to begin
  • exclusive newbie zones

How do workers find relevant work, and requesters find ideal worker?

  • workers are matched with jobs based on their extensive worker profile

- desired wage

- time availability

- perceived quality of work

  • requesters based on the profile matches of workers but also based on some other aspects that can be fine-tuned by requester

- time limit for task to be completed

- expected level of quality

- number of tasks/batches that can be completed by the same worker

Contributers to this page

@anotherhuman @vlado @mahsa