Milestone 3 pixelpals

From crowdresearch
Jump to: navigation, search


Initial Brainstorm

The ideas you brainstormed, (at least 10 ideas for trust, and at least 10 ideas for power). Provide them in whatever format you want - diagrams, sketches, descriptions, or a combination (the wiki supports images, see here for instructions on uploading them).

Trust-related Ideas

1. Increasing trust by establishing a peer - rating system (5/7 Likert scale ) (by fellow workers ). Increasing trust by establishing a requester rating system ( 7 Likert scale ). Requester rate workers on a variety of criteria’s like Speed, Accuracy, Commitment, etc.

2.Background: Workers need to be confident that they understand the goal of the task. Many times workers avoid tasks that have ambiguous goals or might result in errors that get them rejected. They become risk-averse when a task might be confusing and affect their reputation or payment.

Solution: How might we create a platform where workers can be confident about the task? There should forum associated with the task where workers can ask questions and requesters can clarify them. There are certain common queries associated with a task which may be missed out and needs to be clarified. In this way, people can post queries in the public forum and everyone can benefit from it, reducing the ambiguity associated the task. In fact fellow workers can also clarify them. Solutions or clarifications can be upvoted or downvoted by the workers. Although there exists a mailing system, but generally workers mail the requesters for payment and other issues. Moreover requesters generally don’t respond so quickly and in many cases they don’t reply because there is no compulsion. In the embedded Question-Answer forum, if requesters don’t clarify the doubts it highlights that the requesters are not interested and are not helpful. On the basis of this workers can trust or distrust requesters. Alternatively we can also create a chat room where workers can clarify their doubts.

3. Background: Requesters need workers who have appropriate demographics to do their tasks. Requesters have a concern regarding the demographics of the workers for surveys. The demographic information is self-reported by the workers. The information is not viewed as trustworthy by the requests. This is a problem for surveys and correct demographic information is required.

Solution: IP based demography verification can easily be fooled by the use of proxies. A person sitting in India can easily claim to be from other part of the world say, USA through IP spoofing. We propose a verification system based on unique security code. For such HITS we intend to send OTP to the mobile device of the worker. Majority of the people in this world use mobile phones [1]. In US, connection per number of citizens is ~ 103%. Mobile phones are subscribed to Telephone service provider which identifies the network of the device. The worker will enter OTP before starting the task. Using this in addition to the IP address, requester can ascertain the demography of the worker. Additionally, there are API’s in Android and Iphone OS which can tell the network subscriber. Workers with Smartphones can choose this option and can download an App which will receive the OTP and share the required network (area) information.


4 Background: Requesters need workers to have appropriate skills which they require for certain tasks. Workers self-report their skills which are not viewed as trustworthy.

Solution: To overcome this, the platform or the requester can have a short pre-requisite quiz. In the quiz specific questions could be asked to ensure the worker possess specific skills. Many workers might not like taking quiz every time and it could be difficult for requester to create such questions. Therefore, we propose the platform to take care of this. The platform could offer short-term courses for certain skills and at the end it could take exams and quizzes to certify the worker. Since, this certification would be provided by the platform the requester could trust it. For instance, one specific skill could be editing or reviewing a blog post fir which proficiency in English is a pre-requisite. In this way, workers won’t exaggerate their skill sets since the skill-set is verified by the platform. This, in addition to the short quiz before the exam could be more effective.

5 Background: Requesters need workers to trust them. Workers review requester on forums like Turk nation and they might also give bad reviews. Requesters are reluctant to reject work because they fear they might get bad reviews. And since workers work for trustworthy requesters, they might ignore requesters with bad reviews.

Solution: So, the question I discussion is ‘How might we make workers trust he requesters?’ The existing system is point based where there is rat-run for approval points on sites like Turk Nation. Workers talk about requesters and rate them. We need to eliminate this point system but at the same time ensure workers are able to trust the requesters. We propose a certification system. Initially when a requester joins he/she is uncertified. According to certain amount of HITs completed and on the basis of other factors like payment given on time, responsiveness to the worker etc requester can get certification. Having a Certification means the Requester could be trusted. Whenever a worker completes a HIT, he will be asked to review the requester. A simple yes/no question could be asked like ‘Do you think the requester should attain the certificate?’ If for a HIT more than 60% or 70% workers voted against the requester then the certificate would be taken from the requester. Requesters whose certificate is taken could know that they need to improve. They could get the certificate again by responding fairly to the workers. They might need to increase the wages for the HITS for this. People tend to have more faith on the official system than going out and searching on forums. In the certification system, there is no race for points. The requester don’t have to worry that hurting few workers would bring bad name to him (as what used to happen on forums like Turk Nation). But if majority of workers voted against him (i.e. they feel requester is not trustworthy) then it is a clear indication that didn’t treated majority of the workers.

6. Automated pricing range establishment based on skills and average time to complete work. Determine the "pricing range" for a work by calculating the avg completion time and existing pricing/wage standards. Ask certain skill criteria (questions) from each requester related to his task. Put these into pre defined categories associated the pre defined price ranges. Add the total price as value added bonus from such skill values to the price based on avg completion time. The requesters must price his works within this range determined for each of their tasks . Since the pricing will be determined and enforced by the crowd sourcing system, both workers and requesters will feel a sense of trust in the pricing system since (/if) they feel a sense of trust for the crowd sourcing system.

7. Increasing trust by ensuring mandatory feedback from requesters upon each HIT rejection (maybe acception too). Increasing transparency increases trust b/w workers and requesters.

8.[Maybe] A forum/page where requesters can put up the HIT received as a response to their work to show an exceptionally poor or exceptionally great work. [Possible privacy concern.] The intended purpose is to rewards deserving works which will lead to possible rating increase. While at the same time, shame spammers/poor quality work so that these workers either improve or face ratings drop and possibly removal from the system.

9. Requester Sample Solution System. Requester is enforced by the system to himself solve/complete his own work to clarify further what type of answers/solution he is expecting. By pre - completing few of his HITs himself, the requester can ensure he gets better quality results for his test. Also possible is ensuring a system enforced check for validity of the answers according to type of answer supplied by requester himself – this way the requester also has a sense of trust on the works that the system has approved.

10. Re-evaluation of a Rejected HIT by a centralized committee. Workers can request for re-evaluation of their rejected task by a pre-determined committee. This will help build trust amongst workers that the platform is fair and not biased towards requesters. In case of a re-evaluation request, the committee (possible made of some pre-determined requesters and workers with high rationgs and not connected to the task in question in any way) will study the request. They can even ask the requester to give justification of rejection. The decision of the committee will be the final. If the decision is in favour of the worker then the requester is made to pay the complete payment to the worker plus a 10% penalty which can go to the committee. In case of a requester favouring result the worker will be made to pay a small penalty fee. This will not only ensure a possible payment/funding source for the committee and improve its efficiency but also ensure both workers and requesters take this task(Rejection re-evaluation) seriously.

Power-related Ideas

1.Problem: Workers output rejected on loose grounds. Results in lowering of worker reputation. requester not affected by gray reviews. Solution: Double feedback mechanism and Conflict Resolution. Apart from requester review of worker deliverables, the worker should be able to give feedback on the review. If there is a conflict, the result and expected results from from worker and requester can be resolved by a third party provided by the crowdsourcing platform. A small penalty can be laid down for conflict resolution (to be paid by the losing party). Each conflict resolution that goes against the user, the user (worker/requester) reputation will be decremented considerably. This feature should deter flimsy reviews or incorrect worker feedback. Further implementation: To avoid an extra burden of this system on the platform, conflict resolution of jobs can be posted as jobs by the crowdsourcing platform itself. These would be easy, low payment jobs for workers who will be paid through the penalty laid down for conflict resolution on requester and worker. This entire system would lead to more jobs and greater user confidence on the job evaluation schema.

2. Problem: Workers output rejected because of minor errors in output. Workers looses out on money even though he/she put in time and effort. Solution: To avoid conflicts in larger jobs that require time and effort on part of a skilled worker, the requester can have the option of partial payment. The conditions for partial payment can also be mentioned by the requester in the job description. If worker output does not meet the criteria, no payment would be justified.

3. Problem: Every time a worker joins a new platform, he/she must build his reputation from scratch, even though he/she may be an experienced crowd market worker. Solution: Portability of worker credentials from one platform to another. This would require standardization of worker credentials across platforms and coordination/agreement among platforms to honor the credentials provided.

4. Workers should access to give valuable feedback to requesters regarding the tasks. Backgroud: Workers feel unstatisfied with the amount they are paid after completion of the task. So they wish to tell their experience to requesters regarding the tasks.

5. Workers can challenge requesters of being rejected. Current skills mentioned should not always be the criteria to judge whether the tasks could be done by worker or not. Solution: A series sample tasks should be presented before worker , then worker should choose the task according his convenience and then workers should be do the tasks and then rating should take place. If workers passes certain threshold rating then worker is eligible to do those kind of tasks. Here workers preferences is kept into consideration as well as his eligiblity to do the task (from requester point of view).

6.If a worker a techolonogiss then obviusly tasks according to his stadard should be presented to him. "How might we allocate tasks according workers designation"

7.Requesters should be free to set their preferences like number of workers needed to do the task,minimum wage to be given to workers. Solution: No hard thresold should be set by mturks.

Dive Deeper into Specific Ideas

For each of the 4 ideas (2 for trust, 2 for power), describe (using diagrams, sketches, storyboards, text, or some combination) the ideas in further detail.

Please create a separate wiki page for each of your ideas, so we can link to them individually. The title of the wiki page should be Milestone 3 followed by your team name and a description of the idea itself (ex: Milestone 3 YourTeamName TrustIdea 1: Automatic Pricing for Tasks based on Average Completion Time). Once done, post a link to each of your trust-related ideas to and a link to each of your power-related ideas to when done, following the instructions at

Trust-related Ideas

Power-related Ideas

Dark Horse idea

Describe your dark horse idea (using diagrams, sketches, storyboards, text, or some combination).