Milestone 4 padawans

From crowdresearch
Jump to: navigation, search

Inspiration taken from Milestone 3

1. Task clarity: How might Workers + Requesters work together help produce higher-quality task descriptions?


Similarities and Differences in the ideas:


The ideas are targeted to improve the platform for the workers. Majority of them tend to involve the workers too to improve the task clarity and task description through polling, feedback, reviewing etc. All of them want the HITs to undergo multiple reviews that is, they are reviewed by many workers at many levels and if there are many unsatisfied workers, then it is sent back to the requester for improvement.


PumasPower: They are involving workers to perform a quality check of both the task and the work submitted. Triple Clicks: They want a special voting mechanism on the crowdsourcing platforms to grade the requesters based on various aspects like quality of task description, HIT design, payment, etc. They also proposed that requesters should review the task description if there are too many downvotes Innovation: Even they have proposed feedback mechanism to assess the requesters. However, they have proposed a mechanism to provide a detailed feedback where workers can comment on the task description, payment, time taken etc. This mechanism should provide star rating too

PixelPerfect: We found their idea to be very different and practical. They have proposed a solution that shall enforce task clarity while designing the task. Through proper, refined templates, they shall compel the requester to provide proper task description

Btezy: I found their idea interesting and unique. They have proposed that a mechanical turker(imitating worker) shall be used for testing the HIT, its design and description etc. Once the artificial turker approves of it, then only the HIT shall be posted on the turker platform

2. Empathy: How might we build empathy between workers and requesters?

Similarities and Differences in the ideas:

Similarities: All the ideas aim at creating a bond between the workers and the requesters. The bond of trust helps the workers and the requesters to better understand and respect each other. For example, one of the ideas suggests using a platform where workers and requesters can switch roles. This helps them visualize the role of the other. This way they can see the other person's perspective and better understand them. Another idea suggests that workers and requesters interact more with each other. This way they build a relationship and get to know each other's needs better.

Differences: The ideas make use of different approaches to build trust and understanding between the workers and the requesters. The ideas are different as:

1) Puma's TrustIdea : The idea makes use of a platform wherein workers and requesters exchange roles.

2) CrowdBear's Idea : The idea actually suggests that the workers and requesters meet and interact with each other. This idea suggests that they go one step forward. Also there is no anonymity and this removes many doubts and suspicions. Now the workers and the requesters can better trust each other.

3)Opera DarkHorse's Idea : They have added human touch to the platform and a reward system. The requesters have an option of sending surprise gift to the workers. This ensures a worker's happiness and a more satisfied worker.

3. Transparency:How might we make payment clear and transparent?


Note: The implementation not only comprises of the changes that we need to make in Technology but getting the parties agree on it too. For example, we need to make sure that the compensation decided by the authorities annoys neither the workers nor the requesters. Hence, implementing it might take longer than a year.

Similarities and Differences in the ideas:


All the ideas require us to have a separate “managing body” that gets to make several important decisions. Having a look at all the ideas: • Standardizing task pricing: The managing body decides which task belongs to which category and what is the correct amount that should be paid upon successful completion of the task.

• Required minimum wage? if you stay above the 15th percentile of “good work throughput”: This good work throughput needs to be decided by an unbiased body. Plus, in order to implement the idea of deciding “worthwhileness”, we need a group of people who understand the perspective of the workers.

• Checkpoints where you get reviewed and paid after every N tasks: Though this idea can work without a managing body, but in order to ensure that the requester is being fair, we need it.

• Offer increased compensation to the first few people who take the task: We need someone to make sure that instead of giving more compensation to the top workers, requester does not underpay the rest of the workers.

Consider the following:

Scenario1: requester agrees to pay $10.

Scenario2: Requester pays $10 to top workers but an amount lower than that, say $5 to the rest.


Some ideas like offering increased compensation and having checkpoints require development of a whole new software/algorithm that can be run on the current crowdsoucing platforms. Others, apart from making minor changes in the current software, need mutual trust between workers and requesters. We need the workers and requesters to play fair.

Flare and Focus : Generating Ideas for the Themes

1. Task clarity: How might Workers + Requesters work together help produce higher-quality task descriptions?

Problem addressed:

The theme addresses the issues faced by workers about clarity of task description. Often, requesters provide a vague or incomplete description of a task. This misguides the workers who work on the HIT and they end up providing a response which is wrong or not upto the expectation of the requesters. This leads to rejections and many unsatisfied workers and requesters. This theme involves many new ideas which can be implemented to improve task description.

Idea credit:

Idea that inspired was the one presented by team Innovation2. As per their idea, they want a proper feedback system wherein workers can provide a detailed review of the task description and rate the requester. This feedback would be visible to any worker


We found that almost all feasible ideas have already been mentioned under the theme. Yet, we tried to come up with a DARK HORSE idea to improve task clarity. Our idea is to compel the requesters to carefully design the task and give a detailed description of the job along with his expectations.

If a worker who has worked on the HIT, finds that the task description is not as per expectation and finds a lot of flaws, then he can report it on an official forum. If the flaw found by him is true and his justification is valid, the requester needs to pay an extra amount as a penalty to the worker.

The official forum we are talking about should be standard one, like an official complaint forum for the platform. Here, other workers and requesters can vote on an issue. And if an issue has votes greater than a specific number, the requester shall have to pay the penalty.

This idea would not only compel the requesters to provide correct task description, but would give power to workers too. Moreover, this idea would be of great help to such workers who spend their time doing a HIT, but just because of wrong task description, they end up doing the wrong task and their work is rejected(which is a great loss for a worker on a crowdsourcing platform). Such workers not only lose their credibility but also end up losing both time and money.

Limitations: The above idea might lead to unnecessary delay for a HIT to be rectified. With this we mean that the issue raised by a worker needs to gain enough votes which might be time consuming. During this period, other workers might face the same issues and report the same on the forum leading to extra penalties paid by the requester. So what can be done is, as soon an issue is found in the task description and is reported on the complaint forum, the requester shall be intimated immediately about the same so that he can improve the task description. This would prevent other workers from facing the same problems.

2. Empathy: How might we build empathy between workers and requesters?

Problem: Some of the problems being faced by workers include no guarantee of a reasonable pay rate for everyone who works on a task, no encouragement for doing very good work, no trust that they will be paid what they are worth. Problem faced by requesters is that they don't get good quality work done quickly. Also they can't be sure whether the worker is good enough for the job. We need a way by which we can make workers and requesters can trust each other and be assured of good work and pay.


Influencing Ideas: Use of a platform where workers and requesters switch roles to visualise each other's needs. Meet ups and interaction between workers and requesters to build stable relationships and trust.

Building Trust In order to build trust, the requesters must provide an apt definition of the job. Sometimes the requester cannot make out whether the job definition is properly defined. To make sure that it is, the requester should take feedback from a set of workers initially. The feedback helps make the job definition better. This way the requester can be sure that the worker has understood what is expected from them for the job properly. Some information about the worker should be made available to the requester so that they can have some assurance that the worker is capable for delivering good work. To achieve this, there should be a platform where workers and requesters can interact (chats/mails/voice calls etc). This platform will not share personal information. Workers on the other hand need the assurance that they will be paid for what they are worth. To achieve this, there should be rating system where each worker is rated on past submissions. By means of votes, the rating of a worker should be decided. The workers should then be paid according to their ratings. Now because of this rating system, the workers will be encouraged to do very good work. For motivating the workers towards good quality work, there should be some incentives for them. These incentives could include awards, points and rankings.

Limitations: On one hand there will be a lack of trust due to anonymity, on the other hand sharing too much personal information can lead to misuse of the information. There can be a lot of debate on how much information should be shared so that it ensures a person's personal information safety and still shares sufficient information so that workers and requesters can build trust and confidence in each other.

3. Transparency:How might we make payment clear and transparent?

Problem: Some of the issues currently being faced by workers include low wages and unfair refusal of completed task. In both the cases, we require a system that ensures that the workers are not underpaid. Plus, they should be able to know the reason for rejection of their work. So, we need a system that is transparent and lets the workers have an answer to why a requester didn't accept his/her work. If the system is rendered fair, many of these problems will be solved to a great extent.


Influencing Ideas: The ideas for increased compensation and minimum wage cannot be successful as long as there is no kind of “discrimination” on grounds of reputation of the worker.

For workers

1)To ensure good wages: Even workers new to the industry can be highly skilled. Or there may be workers who have a humble reputation but possess the capability to perform a particular task really well. So, ALL workers should take up a task and submit it anonymously. This will ensure fairness and the requester will not be able to pay less just because the worker has a comparitively poor reputation or the worker is new. This way the new workers will be able to establish themselves a little easily.


Limitation: The relationship between workers and requesters cannot be maintained due to anonymity. In other words, the social relationship between the two parties shall be greatly affected as there is no way for the requester to know which worker he is rewarding.

2) Since hundereds of workers are present and the works of only a few gets accepted, it is not possible for the requester to send the reason for rejection to every worker. The requester should instead be made to give out the reason to the workers with reputation greater than a threshold or may top 10% of the workers should be notified of the reason for refusal.

Limitation: Suppose 500 workers submit the task but only one worker's work gets accepted. This means that 50 people will have to be notified and provided with reason. The requester may not have that kind of time.