Difference between revisions of "Milestone 3 Sky"

From crowdresearch
Jump to: navigation, search
(Dark Horse idea)
 
(28 intermediate revisions by 2 users not shown)
Line 1: Line 1:
'''Due date (PST): 11:59 pm 18th March 2015 for submission, 9 am 20th March 2015 for voting and commenting on others' ideas.'''
+
== Initial Brainstorm ==
  
This week, we will take the set of needs that we collectively identified in the previous milestone and use those insights to generate design ideas.
+
Now it's time for your team to brainstorm some ideas based on these needs.
  
* Youtube link of the meeting today: [https://www.youtube.com/watch?v=mn_hTHjbFDM watch]
+
Work with your team to brainstorm as many ideas as you can under two headings: '''trust''' and '''power'''. Here are some examples of "[[:Media:How_might_we.pdf | How might we]]" questions (a technique which can inspire specific brainstorms) which can drive the generation of your ideas:
* Meeting 3 slideshow: [[:Media:Week3 presentation.pdf | pdf]]
+
  
== Needs from Milestone 2 ==
+
* “[[:Media:How_might_we.pdf | How might we]]” enable workers to trust the requester’s intention to pay?”
 +
* “How might we” enable requesters to trust the results they get back?
  
We synthesized the main needs groups identified in Milestone 2 in the following table:
+
Use whatever tools you need - if you can get together in-person, whiteboards and sketchbooks are great tools, while if you're a remote team, services like [http://docs.google.com Google Docs] and [http://sketchboard.io sketchboard.io] should help you with the brainstorming process.
  
=== Worker Needs ===
+
Sketch out enough ideas until you find a set that you’re inspired to explore further. It should be at least 20 ideas total - 10 ideas for trust, and 10 ideas for power. This brainstorm should be wild and broad. Focus not on usability patches, but deeper design innovations.
  
{| class="wikitable"
 
|-
 
! rowspan = "2" | Needs
 
! colspan = "2" | Evidence
 
|-
 
! Observations
 
! Interpretations
 
|-
 
|Workers need to be able to quickly find tasks they'd want to work on
 
|Monday evening panel workers cite the challenge in identifying or distinguishing tasks because of poor tagging, Reddit discussion also cited the exorbitant amount of time that they spend trying to find tasks and do the mental calculations to find the opportunities that match them best (e.g. time to complete the task on average, average $ per minute on task, requirements to complete the task).
 
|Finding a task that matches well with the worker's skillset and pays well takes a significant amount of (unpaid) time.
 
|-
 
|Workers need to feel they are being fairly compensated for their work.
 
|Reddit discussion cites that the payment system for HITs is not adaptive and does not take into account changing marketplace conditions (supply/demand) and pricing of tasks based on those changes.
 
|Monetary compensation is the primary motivator for many crowd workers.
 
|-
 
|Workers need to feel like they are treated fairly and respectfully, and have a voice in the platform
 
|Comment on Turkopticon: "Got a mass rejection from some hits I did for them! Talked to other turkers that I know in real life and the same thing happened to them. There rejection comments are also really demeaning. Definitely avoid!"
 
|Unreasonable rejections and low payments lead workers to feel disrespected. The implicit assumption on MTurk is that workers are unskilled and replaceable. They can do little if their work is rejected.
 
|-
 
|Workers need to be able to expose their skills so they can get work they are qualified for and advance their skills
 
|Monday evening panel workers from oDesk cite that most employers will not work with them until they have enough feedback or past work on the platform
 
|If users cannot get new work without feedback, this makes it difficult for new users to establish their reputation and get jobs that will help develop their skillsets.
 
|-
 
|-
 
|Workers need to be confident that they understand the goal of the task, and quickly
 
|Workers avoid tasks that have ambiguous goals or might result in errors that get them rejected
 
|Workers become risk-averse when a task might be confusing enough to threaten their reputation or payment
 
|-
 
|}
 
  
=== Requester Needs ===
+
=== Needs of Workers ===
  
{| class="wikitable"
 
|-
 
! rowspan = "2" | Needs
 
! colspan = "2" | Evidence
 
|-
 
! Observations
 
! Interpretations
 
|-
 
|Requesters need to get their HITs completed (quickly / correctly)
 
|Requester asking on forum why nobody is doing his HITs (7-minute, 25-cent surveys - a very low wage)
 
|Requesters want their HITs done, and when nobody's doing them, they do not know the reason why (e.g. it is because he is underpaying workers)
 
|-
 
|Requesters need to be able to trust the results they get
 
|Requesters will often rely on previous workers whose results they can trust, and add mechanisms to detect spammers, or manually verify some results.
 
|If spammers are not caught, this brings the correctness of results into question. If requesters are not sure the results are correct, they may need to discard the data.
 
|-
 
|Requesters need to have workers who have the appropriate skills and demographics do their tasks
 
|Requesters worry that they are not able to verify self-reported demographics for surveys.
 
|Workers' self-reported skills and demographics are often not viewed as trustworthy. This is a problem for surveys, which need to have correct demographic data to be useful.
 
|-
 
|Requesters need to be able to easily generate good tasks
 
|Companies hire full-time developers to deal with the complexities of posting microtasks on MTurk. Requesters often develop their own tools and workflow systems on top of Amazon's.
 
|The process of authoring HITs is currently difficult and makes crowd-work inaccessible to potential requesters
 
|-
 
|Requesters need to price their tasks appropriately
 
|Requesters asking on forums about the appropriate amount they should pay for their HITs
 
|Requesters often don't have a good intuition of what the appropriate wage for their task would be in terms of price per HIT.
 
|-
 
|Requesters need workers to trust them
 
|Requesters say they are reluctant to reject work, because they fear they might get bad reviews.
 
|Workers are more likely to do HITs if the requester seems trustworthy. Requesters do not want bad reviews, because they may result in workers ignoring the requester's HITs
 
|-
 
|}
 
  
=== Michael Bernstein's synthesis ===
 
These needs boil down to two main issues: 1) trust, and 2) power.
 
  
Trust:
+
===== Trust Based Search =====
* How do I trust who you say you are?
+
* How do I trust that the results I get are results that will be good?
+
* How do I trust that you’ll respect me as a worker, and pay me accordingly?
+
  
Power:
 
* Who has the power to post work?
 
* To edit other peoples’ posted work?
 
* To return results to the requester? Can I, as a worker, send it back myself, or does someone else need to vet it?
 
  
As we brainstorm, we should be thinking about solutions that holistically address these issues of power and trust, not just surface fixes that get at micro-elements of the system.
 
  
== Recommended Readings ==
+
Having search based on requesters reputation would increase the trust of worker ---> requester. So we need to give workers this opportunity to search HIT according to their trust level. According to our need finding in milestone 2 workers wants to trust two things. First Requester's intention and second HIT description.
  
Coming up with good, novel visions and ideas is a crucial part of doing successful research, and reading other researchers' visions and ideas can help you come up with better ideas yourself. These readings discuss visions that crowdsourcing researchers have thought of related to future crowd marketplaces. This week's readings are optional and don't have a deliverable, but are highly recommended.
+
====== Realtime HIT-Description statistic panel ======
  
=== Design notes for a future crowd work market ===
+
As we mentioned we need something to fill the gap of trust in workers to HIT description.  We can have a panel that have many statistics. This panel must be realtime based on previous HIT attempts. Average completion time, Average Handy score of HIT, Safety score of HIT, Clarity score,  Fun score and etc.
 +
These panel's element is statistic based on previous HIT so they are trustable. we will explain it in details in nest step.
  
[https://medium.com/@silberman/design-notes-for-a-future-crowd-work-market-2d7557105805 Design notes for a future crowd work market] - This is a Medium post written by researchers involved with Turkopticon in response to hearing about this research project. It discusses their vision for a future crowd marketplace, where workers are more involved in the management of the marketplace.
+
Having this panel enable workers to search based on trust-based statistics gathered from HITs and other workers that create more trust in workers to believe HIT descriptions.
  
=== The future of crowd work ===
+
   
 +
====== Realtime Requester-intention statistic panel ======
  
[[:Media:The_future_of_crowd_work_(private).pdf | Kittur A, Nickerson J V, Bernstein M, et al. The future of crowd work. Proceedings of the 2013 conference on Computer supported cooperative work. ACM, 2013: 1301-1318.]] - This paper envisions a future crowd marketplace that emphasizes on workers' long-term development, and where people can be proud to be workers. It is a long paper. Feel free to focus on just the parts that particularly interest you.
+
As we discussed we need also something to fill the trust gap between workers and requesters. One mechanism is having realtime statistic for requesters like approval rate ( in all of his request ), number of approve/rejection for each HIT (average is not good here), Response rate( In all of his request ), worker's rating average, Number of total request, Total payment , Average hourly payment (Payment /average completion time ).  
 +
 +
Having this panel enable workers to search based on trust-based statistics gathered from HITs and other workers that create more trust in workers to believe requester's intention .
  
== Initial Brainstorm ==
 
  
Now it's time for your team to brainstorm some ideas based on these needs.
+
===== Power of Selection before Starting HIT =====
  
Work with your team to brainstorm as many ideas as you can under two headings: '''trust''' and '''power'''. Here are some examples of "[[:Media:How_might_we.pdf | How might we]]" questions (a technique which can inspire specific brainstorms) which can drive the generation of your ideas:
+
Items that are listed below are going to balance distribution of power in system among requesters and workers by giving workers to find the best matched HIT to them. Having match maker will also reduce the rejection rate of HIT.
  
* “[[:Media:How_might_we.pdf | How might we]]” enable workers to trust the requester’s intention to pay?”
+
       
* “How might we” enable requesters to trust the results they get back?
+
====== Pining/subscribing requesters, HITs and keywords ======
  
Use whatever tools you need - if you can get together in-person, whiteboards and sketchbooks are great tools, while if you're a remote team, services like [http://docs.google.com Google Docs] and [http://sketchboard.io sketchboard.io] should help you with the brainstorming process.
+
n this mechanism workers can pin their favorites keyword , requesters and HIT's for future recommendations. I Its like subscription on specific requester or category of HITs.
  
Sketch out enough ideas until you find a set that you’re inspired to explore further. It should be at least 20 ideas total - 10 ideas for trust, and 10 ideas for power. This brainstorm should be wild and broad. Focus not on usability patches, but deeper design innovations.
+
====== Real-Time KeyWord suggestion based on active keywords ======
  
=== Deliverable ===
+
Having an efficient tagging system and also tag search system. The key important point in this idea is 'Active Keywords'.  Active keywords are existing keywords in current available HIT's.  We need to index most valuable keyword and suggest them to workers as they search keyword in the search box.
  
The ideas you brainstormed, (at least 10 ideas for trust, and at least 10 ideas for power). Provide them in whatever format you want - diagrams, sketches, descriptions, or a combination (the wiki supports images, [http://www.mediawiki.org/wiki/Help:Images see here] for instructions on uploading them).
 
  
== Dive Deeper into Specific Ideas ==
+
====== News Feed based on Workers favorites and History ======
  
Select 2 ideas per heading (trust, power) that you would like to pursue, and expand on them a bit further. In addition to describing the idea itself, make sure you also tell us:
+
We have a panel that is always list just created  HIT in the time which is related to the user in some way.
  
* What are the goals of the design? For example, Google's Android design goals are: delight me in surprising ways, simplify my life, and make me amazing (e.g., grant me special powers).
+
===== Trust =====
* Which aspects of your design reflect each goal? How does your design solution addresses the users' needs?
+
  
=== Deliverable ===
+
====== Dynamic overall and specific payment statistics  panel ======
  
For each of the 4 ideas (2 for trust, 2 for power), describe (using diagrams, sketches, storyboards, text, or some combination) the ideas in further detail.
+
Having statistics like Total payment, Average hourly payment , Average time completion can fill this trust gap in future payment by requesters.
  
Please <strong>create a separate wiki page for each of your ideas</strong>, so we can link to them individually. The title of the wiki page should be Milestone 3 followed by your team name and a description of the idea itself (ex: [[Milestone 3 YourTeamName TrustIdea 1: Automatic Pricing for Tasks based on Average Completion Time]]). Post a link to each of your trust-related ideas to http://crowdresearch.meteor.com/category/milestone-3-trust-ideas and a link to each of your power-related ideas to http://crowdresearch.meteor.com/category/milestone-3-power-ideas when done.
+
====== Task Pricing Standard======
 +
By standardizing the task payment and pricing, based on a comprehensive, quality based per hour price list of the HITs, we ensure the workers of the fair payment.
  
== Dark Horse idea ==
+
====== Average payment ======
 +
Mturk should have an ability to show real-time average payment for workers so as long as they are close or above average they feel they are paid in fairness.
  
Now that you've identified some design directions you like, it's time to change tack and toss in a dark horse idea. A dark horse, in horse racing, is a contender who most people don't think will win, but may turn in an unexpectedly strong performance and produce a huge payoff. Dark horse ideas are intended to be something far out there or nearly impossible. In the best case, your dark horse ideas might end up winning the race. However, even in the worst case, they can give us tremendous design insight and prevent design fixation, where the design space shrinks too rapidly.
+
====== A minimum wage schema ======
 +
There should be a minimal payment for turkers so they can trust they get at least an amount of money out of their work. It might actually affect some turkers in a negative way but I assume it will benefit the majority. This could be implemented globally or a requester should specifically define a minimum amount of payment for each HIT
  
There are three requirements for dark horse ideas. First, they must be "dark": they must explore a space that is risky, radical, infeasible, and/or in a direction orthogonal to previously explored solutions. They should feel slightly uncomfortable. Second, they must be brainstormed after the more traditional ideas — you can't have a dark path without a traditional "light" path to contrast it against. Third, they must be refined enough that they could be prototyped and objectively tested. That is, it cannot be infeasible: it needs to be something that we could put in front of real people to see whether it would work.
 
  
If you have dark horse ideas that came up in your initial brainstorm, you can use that. If you're not satisfied, brainstorm some! Try using [https://dschool.stanford.edu/groups/k12/wiki/faf1d/Powers_of_Ten.html Powers of Ten] and other techniques to push further and generate even more. After you brainstorm and sketch out dark horse ideas, choose one that you'd like to include among your set of two top candidates from before. Expand on your dark horse idea like you did in the previous section.
+
===== Power =====
  
=== Deliverable ===
+
In Mturk after completing the task, requester was the person who has the power to approve or reject HITs with any reason. By these Ideas below we are going to give some power to worker or restrict requester's  power and benefit. This would decrease the rejection rate and consequently workers power.
 +
  
Describe your dark horse idea (using diagrams, sketches, storyboards, text, or some combination).
+
====== Power to define rates ======
 +
Currently workers don't have any means to negotiate payments.
 +
We can add a bidding mechanism so workers could come and enter their minimum hourly payment and requester can choose the minimum or they can balance worker quality and payment.
  
Please create a separate wiki page for your dark horse idea so we can link to it individually. Post the link on http://crowdresearch.meteor.com/category/milestone-3-dark-horse-ideas when done
+
====== Power to report abuse ======
 +
Workers should have the ability to report a requester that doesn't pay enough and there should be an ability to check on that. So there should be a god view somewhere so that an authority can check the fairness based on statistics. There could be an automated process that if enough workers report a requester, the requesters account is suspended automatically and be reported for an investigation.
  
== Submitting ==
 
  
=== Create a Wiki Page for your Team's Submission ===
+
======  STOPING requesters to use rejected Worker's HIT output efficiently ======
  
Please create a page for your team's submission at http://crowdresearch.stanford.edu/w/index.php?title=Milestone_3_YourTeamName&action=edit (substituting in YourTeamName with the team name), copy over the template at <strong>[[Milestone 3 Template]] </strong>. If you have never created a wiki page before, please see [http://www.mediawiki.org/wiki/Help:Starting_a_new_page this] or watch [https://www.youtube.com/watch?v=83-lCpAnaFw this].
+
We need to see workers job as service or product that can be return. Then we need eBay Like payment system, Keep product until both satisfied by the service. We can have two step for delivering the result to he requesters. First step is Filtering stage which they can filter their result based on they criteria and second is exporting the result. In exporting sage we just allow the requester to export output of approved workers. Requester might export all data manually in Filtering stage but its not as easy as before. So requesters for reducing overhead tend to just reject those that are not really useful.
 +
====== Micro-Contracts======
  
=== Post the links to your ideas, upvote ones you like, and comment on them ===
+
By having a contract-based system, penalizing the requester "Reputation" for unjustified rejection and enabling micro-contracts between worker and requester, we ensure worker is being paid and mediate the power of payment between worker and the requester.
  
<strong>Meteor Sign-up Instruction: </strong> Please use the ''same username as your Slack'', this will help us identify and track your contributions better.
+
====== Standardized Rejection/Acceptance======
  
-<strong>Submission by 11:59 pm 18th March 2015</strong>
+
By having a standardized platform for HIT acceptance/rejection, like quality, accuracy, completeness we can resign the power for rejection from the requester and enforce a system-based, standardized platform to accept or reject.
 +
Using the method we address the unjustified rejection and emotional involvements as well as unfair payment or payment rejection.
  
We have a service on which you can post the links to the wiki-pages for the individual ideas you generated, explore them, and upvote them. There are 3 submission categories:
 
  
1- http://crowdresearch.meteor.com/category/milestone-3-trust-ideas where you can post links to the wiki pages for each of the 2 trust-related ideas you generated in the "Dive Deeper into Specific Ideas" stage
+
===== Trust =====
  
2- http://crowdresearch.meteor.com/category/milestone-3-power-ideas where you can post links to the wiki pages for each of the 2 power-related ideas you generated in the "Dive Deeper into Specific Ideas" stage
+
====== Worker-Requester relationship guideline ======
 +
Just like any other social situation, there should be some rules that everyone must follow while interacting with each other. This could be implemented as a bot that detects bad language or as a community supervised authority or the owner of the website must impose these guidelines. These rules should be updated based any complaint on bad language and worker abuse.
  
3- http://crowdresearch.meteor.com/category/milestone-3-dark-horse-ideas where you can post a link to the wiki page for your dark horse idea
+
====== A rating and review scheme ======
 +
In order to build trust for workers, there should be a reviewing scheme so that workers could trust requesters for treating them respectfully. There should also be an imposing rule on requesters so that they can't just change account and continue their bad behaviour.
  
-<strong>Peer-evaluation from 12:05 am 19th March until 9 am 20th March 2015</strong>
+
===== Power =====
  
Post submission phase, you are welcome to browse through, upvote, and comment on others' ideas. We encourage you especially to look at and comment on ideas that haven't yet gotten feedback, to make sure everybody's ideas gets feedback. You can use http://crowdresearch.meteor.com/needcomments to find ideas that haven't yet gotten feedback, and http://crowdresearch.meteor.com/needclicks to find ideas that haven't been yet been viewed many times.
+
====== Forums ======
 +
Workers need to have a forum so they can discuss their experiences with requesters and talk with moderates and the owner of the website.
  
COMMENT BEST-PRACTICES: As on Crowdgrader, <strong>everybody</strong> reviews at least 3 ideas, supported by a comment. The comment has to justify your reason for upvote. The comment should be constructive, and should mention positive aspect of the idea worth sharing. Negative comments are discouraged, rather make your comment in the form of a suggestion - such as, if you disliked an idea, try to suggest improvements (do not criticize an idea, no idea is bad, every idea has a scope of improvement).
+
====== Distinguished Requesters ======
 +
Workers should have the ability to vote for certain requesters as distinguished. Like requesters of the month and requesters of the week, a trusted badge, etc.
  
=== Milestone 3 Submissions ===
+
===== Power to change by their feedback=====
  
To help us track all submissions and browsing through them, once you have finished your Milestone 3, go to the link below and post the link:
+
====== Efficient Up-Vote enabled feedback system to communicate Managers \ requesters  ======
  
[[Milestone 3 Submissions]]
+
We need to have a feedback system for users. Workers can post their own feedbacks and vote others feedbacks in using system.  And make sure that top feedbacks will be considered in next upgrades. These feeds backs are for improving the system and it's functionalities.
 +
 
 +
Also each HIT can have one page like reddit that is for communication between requester and All of workers. This help workers ask any question or clarification from both requesters and other workers that have done that HIT.
 +
 
 +
 
 +
 
 +
In our opinion there are Four scenario in witch workers have difficulties in exposing their skills. First according to milestone 3 template when they new user without any history of previous job. So in systems like mTurk there are less opportunities cause they are not qualified for lots of HITs. And the second one regardless of workers experience they need to show their previous history in more efficient manner. And third group are those who has high rejection rate for any reason like lack of knowledge in using system or scam requesters. But know they are experienced and we need to give them another chance.  And Fourth We have some workers with specific skills that want to participate but with more value.
 +
 
 +
 
 +
 
 +
===== Power =====
 +
 
 +
====== Tender offer for unqualified , base price for qualified , Auction offer high qualified workers ======
 +
 
 +
As you see in Mturk requesters are specifying a qualification level and then filter lots of workers.  Here every HIT should have three way to apply the job through three prices. Also here every HIT should have base value which is same as current HIT value in Mturk and also base qualification level which can be like current qualification requirement in MTurk. But there are two more way to get the HIT. First offering lower price than base price from currently unqualified workers.  Second are those who think they have specific skills that is valuable for  requesters and they are ready to pay more for that.
 +
 
 +
For base price workers just accept and start to doing that, but Unqualified and also high qualified workers should wait until requester accept their application. workers application should be with a comment to motivate requester to accept it.
 +
 
 +
====== ======
 +
 
 +
 
 +
===== Trust =====
 +
 
 +
====== Efficient Up-Vote enabled feedback system per HIT (forum per HIT)  ======
 +
 
 +
We can have forum like page that is for communication between requester and All of workers about issue related to the HIT . This help workers ask any question or clarification from both requesters and other workers that have done that HIT. Its save time for both requester and workers.
 +
 
 +
====== HIT comprehensive description ======
 +
 
 +
By enabling a description about the goal of the HIT, the procedure the worker should follow and giving the clear directives we can ensure that the user understand the tasks.
 +
In one interview one worker said, if we were aware of the goal of the task, we could contribute better and by knowing the objective and the incentive to the HIT the workers feel they understand it better
 +
 
 +
====== Mandatory face-to-face meeting ======
 +
The first milestone in each task should be a mandatory face to face meeting so that workers could resolve any ambiguity in the goals and the description of the task.
 +
 
 +
 
 +
===== Power =====
 +
 
 +
====== HIT Design Standard platform ======
 +
 
 +
The standard platform for HIT design can eliminate many confusions a worker might face while doing the HIT and clarifies the HIT by standards, and enforce the worker to provide useful directives and procedure along with the HIT.
 +
 
 +
=== Needs of Requester  ===
 +
 
 +
 
 +
===== Trust =====
 +
 
 +
====== Online price calculator ======
 +
Requesters should have access to a transparent online price calculator where they input amount of work required and the quality desired and the calculator shows minimum payment to both requester and worker.
 +
 
 +
====== Categorisation ======
 +
There should be several categories that requesters could put their HIT there. Each category could have an estimated minimum price and average price that requesters could plan accordingly.
 +
 
 +
===== Power =====
 +
 
 +
====== Negotiation ======
 +
There should be contract scheme with several pre-contracts that requesters could negotiate the price with workers so that both could agree upon.
 +
 
 +
====== Statistics ======
 +
Requesters must have access to a statistical data gathered from the free market or from the website itself that determines the appropriate price for each type of work. For example the requester can see that if he/she wants his work done under a week and he/she has a particular type of work, then it could be easily determined that statistically a certain minimum price should be set.
 +
 
 +
 
 +
===== Trust =====
 +
 
 +
====== Requester Reputation Builder======
 +
 
 +
By having a requester reputation builder we enable the previous workers dealing with requester to rank them based on their promptness in payment, HIT design, fairness in payment and responsiveness. What about the new requesters? We set a value for default, the first HIT is the critical point for the requester to build their reputation, if they succeed in the first HIT and get satisfactory rates, they can maintain and even go up, but if they are scammers or "unfair" ones, they will rated downwardly.
 +
 
 +
 
 +
 
 +
====== Authenticity ======
 +
An authenticity scheme should be implemented. For example, the owner of the website should provide a mechanism for requesters to provide IDs to the owner and the owner can confirm the authenticity of the requester.
 +
 
 +
====== Contracts ======
 +
There should be a solid contract between requesters and workers where they can negotiate and come to their terms and there should be a mechanism to compensate for any damage when the contract is breached.
 +
 
 +
 
 +
 
 +
===== Power =====
 +
 
 +
====== Worker-System Ranking Mechanism ======
 +
 
 +
Predefining a set of standards, and qualifications, and terms and agreements, and contracting the requester in the first place, we can give a medium score to the worker to enter the market, and by posting a HIT we enable the Worker or the HIT Micro-Syndicate to vote and rank the requester and change them up. The system and the worker has the power to rate the requester and as mentioned earlier, the wight of the rate coming from the Micro-Syndicate has more importance than the individual weights, to avoid emotional involvement. Also we should enforce requester rating by the worker, maybe provide bonus for giving rank, and prompt them to rate the requester after they got the payment and justifications.
 +
 
 +
 
 +
====== Breach of contract in an event of lack of trust ======
 +
This seems out of reach but requesters should have a limited power to request a breach of contract if there is a lack of trust between workers and requesters.
 +
 
 +
====== Ability to advertise ======
 +
Requesters should have the power to advertise on the platform. Advertisement is the key to establish trust between workers and requesters.
 +
 
 +
== Dive Deeper into Specific Ideas ==
 +
 
 +
 
 +
=== Deliverable ===
 +
 
 +
 
 +
==== Top 2 Trust-based Idea ====
 +
[http://crowdresearch.stanford.edu/w/index.php?title=Milestone_3_Sky_TrustIdea_1:_Collective_Statistics_Panel_%28CSP%29 Trust Idea 1: Collective Statistics Panel]
 +
 
 +
[http://crowdresearch.stanford.edu/w/index.php?title=Milestone_3_Sky_TrustIdea_1:_Multilevel_Result_Delivery_system_to_requesters Trust Idea 2: Multilevel Result Delivery system to requesters]
 +
 
 +
==== Top 2 Power-based Idea ====
 +
[http://crowdresearch.stanford.edu/w/index.php?title=Milestone_3_Sky_PowerIdea_1:_A_Three-priced_Offer_System_with_Negotiation_Mechanism_%28TBA%29 Power Idea 1: A Three-priced Offer System with Negotiation Mechanism]
 +
 
 +
[http://crowdresearch.stanford.edu/w/index.php?title=Milestone_3_Sky_PowerIdea_2:_Three_complementary_part_HIT_Tracer_system Power Idea 1:Three complementary part HIT Tracer system]
 +
 
 +
== Dark Horse idea ==
  
== Fill out this week's survey ==
 
  
Please provide your feedback on this week's meeting and milestone so we can improve it, by filling out this [https://docs.google.com/forms/d/1ERZ_fg5N3Pfzg8WC63-jj8GFOej7K-zPTA4a21WtLwo/viewform?usp=send_form survey]
+
=== Top 2 Dark Horse idea ===

Latest revision as of 02:37, 19 March 2015

Contents

Initial Brainstorm

Now it's time for your team to brainstorm some ideas based on these needs.

Work with your team to brainstorm as many ideas as you can under two headings: trust and power. Here are some examples of " How might we" questions (a technique which can inspire specific brainstorms) which can drive the generation of your ideas:

  • How might we” enable workers to trust the requester’s intention to pay?”
  • “How might we” enable requesters to trust the results they get back?

Use whatever tools you need - if you can get together in-person, whiteboards and sketchbooks are great tools, while if you're a remote team, services like Google Docs and sketchboard.io should help you with the brainstorming process.

Sketch out enough ideas until you find a set that you’re inspired to explore further. It should be at least 20 ideas total - 10 ideas for trust, and 10 ideas for power. This brainstorm should be wild and broad. Focus not on usability patches, but deeper design innovations.


Needs of Workers

Trust Based Search

Having search based on requesters reputation would increase the trust of worker ---> requester. So we need to give workers this opportunity to search HIT according to their trust level. According to our need finding in milestone 2 workers wants to trust two things. First Requester's intention and second HIT description.

Realtime HIT-Description statistic panel

As we mentioned we need something to fill the gap of trust in workers to HIT description. We can have a panel that have many statistics. This panel must be realtime based on previous HIT attempts. Average completion time, Average Handy score of HIT, Safety score of HIT, Clarity score, Fun score and etc. These panel's element is statistic based on previous HIT so they are trustable. we will explain it in details in nest step.

Having this panel enable workers to search based on trust-based statistics gathered from HITs and other workers that create more trust in workers to believe HIT descriptions.


Realtime Requester-intention statistic panel

As we discussed we need also something to fill the trust gap between workers and requesters. One mechanism is having realtime statistic for requesters like approval rate ( in all of his request ), number of approve/rejection for each HIT (average is not good here), Response rate( In all of his request ), worker's rating average, Number of total request, Total payment , Average hourly payment (Payment /average completion time ).

Having this panel enable workers to search based on trust-based statistics gathered from HITs and other workers that create more trust in workers to believe requester's intention .


Power of Selection before Starting HIT

Items that are listed below are going to balance distribution of power in system among requesters and workers by giving workers to find the best matched HIT to them. Having match maker will also reduce the rejection rate of HIT.


Pining/subscribing requesters, HITs and keywords

n this mechanism workers can pin their favorites keyword , requesters and HIT's for future recommendations. I Its like subscription on specific requester or category of HITs.

Real-Time KeyWord suggestion based on active keywords

Having an efficient tagging system and also tag search system. The key important point in this idea is 'Active Keywords'. Active keywords are existing keywords in current available HIT's. We need to index most valuable keyword and suggest them to workers as they search keyword in the search box.


News Feed based on Workers favorites and History

We have a panel that is always list just created HIT in the time which is related to the user in some way.

Trust
Dynamic overall and specific payment statistics panel

Having statistics like Total payment, Average hourly payment , Average time completion can fill this trust gap in future payment by requesters.

Task Pricing Standard

By standardizing the task payment and pricing, based on a comprehensive, quality based per hour price list of the HITs, we ensure the workers of the fair payment.

Average payment

Mturk should have an ability to show real-time average payment for workers so as long as they are close or above average they feel they are paid in fairness.

A minimum wage schema

There should be a minimal payment for turkers so they can trust they get at least an amount of money out of their work. It might actually affect some turkers in a negative way but I assume it will benefit the majority. This could be implemented globally or a requester should specifically define a minimum amount of payment for each HIT


Power

In Mturk after completing the task, requester was the person who has the power to approve or reject HITs with any reason. By these Ideas below we are going to give some power to worker or restrict requester's power and benefit. This would decrease the rejection rate and consequently workers power.


Power to define rates

Currently workers don't have any means to negotiate payments. We can add a bidding mechanism so workers could come and enter their minimum hourly payment and requester can choose the minimum or they can balance worker quality and payment.

Power to report abuse

Workers should have the ability to report a requester that doesn't pay enough and there should be an ability to check on that. So there should be a god view somewhere so that an authority can check the fairness based on statistics. There could be an automated process that if enough workers report a requester, the requesters account is suspended automatically and be reported for an investigation.


STOPING requesters to use rejected Worker's HIT output efficiently

We need to see workers job as service or product that can be return. Then we need eBay Like payment system, Keep product until both satisfied by the service. We can have two step for delivering the result to he requesters. First step is Filtering stage which they can filter their result based on they criteria and second is exporting the result. In exporting sage we just allow the requester to export output of approved workers. Requester might export all data manually in Filtering stage but its not as easy as before. So requesters for reducing overhead tend to just reject those that are not really useful.

Micro-Contracts

By having a contract-based system, penalizing the requester "Reputation" for unjustified rejection and enabling micro-contracts between worker and requester, we ensure worker is being paid and mediate the power of payment between worker and the requester.

Standardized Rejection/Acceptance

By having a standardized platform for HIT acceptance/rejection, like quality, accuracy, completeness we can resign the power for rejection from the requester and enforce a system-based, standardized platform to accept or reject. Using the method we address the unjustified rejection and emotional involvements as well as unfair payment or payment rejection.


Trust
Worker-Requester relationship guideline

Just like any other social situation, there should be some rules that everyone must follow while interacting with each other. This could be implemented as a bot that detects bad language or as a community supervised authority or the owner of the website must impose these guidelines. These rules should be updated based any complaint on bad language and worker abuse.

A rating and review scheme

In order to build trust for workers, there should be a reviewing scheme so that workers could trust requesters for treating them respectfully. There should also be an imposing rule on requesters so that they can't just change account and continue their bad behaviour.

Power
Forums

Workers need to have a forum so they can discuss their experiences with requesters and talk with moderates and the owner of the website.

Distinguished Requesters

Workers should have the ability to vote for certain requesters as distinguished. Like requesters of the month and requesters of the week, a trusted badge, etc.

Power to change by their feedback
Efficient Up-Vote enabled feedback system to communicate Managers \ requesters

We need to have a feedback system for users. Workers can post their own feedbacks and vote others feedbacks in using system. And make sure that top feedbacks will be considered in next upgrades. These feeds backs are for improving the system and it's functionalities.

Also each HIT can have one page like reddit that is for communication between requester and All of workers. This help workers ask any question or clarification from both requesters and other workers that have done that HIT.


In our opinion there are Four scenario in witch workers have difficulties in exposing their skills. First according to milestone 3 template when they new user without any history of previous job. So in systems like mTurk there are less opportunities cause they are not qualified for lots of HITs. And the second one regardless of workers experience they need to show their previous history in more efficient manner. And third group are those who has high rejection rate for any reason like lack of knowledge in using system or scam requesters. But know they are experienced and we need to give them another chance. And Fourth We have some workers with specific skills that want to participate but with more value.


Power
Tender offer for unqualified , base price for qualified , Auction offer high qualified workers

As you see in Mturk requesters are specifying a qualification level and then filter lots of workers. Here every HIT should have three way to apply the job through three prices. Also here every HIT should have base value which is same as current HIT value in Mturk and also base qualification level which can be like current qualification requirement in MTurk. But there are two more way to get the HIT. First offering lower price than base price from currently unqualified workers. Second are those who think they have specific skills that is valuable for requesters and they are ready to pay more for that.

For base price workers just accept and start to doing that, but Unqualified and also high qualified workers should wait until requester accept their application. workers application should be with a comment to motivate requester to accept it.

Trust
Efficient Up-Vote enabled feedback system per HIT (forum per HIT)

We can have forum like page that is for communication between requester and All of workers about issue related to the HIT . This help workers ask any question or clarification from both requesters and other workers that have done that HIT. Its save time for both requester and workers.

HIT comprehensive description

By enabling a description about the goal of the HIT, the procedure the worker should follow and giving the clear directives we can ensure that the user understand the tasks. In one interview one worker said, if we were aware of the goal of the task, we could contribute better and by knowing the objective and the incentive to the HIT the workers feel they understand it better

Mandatory face-to-face meeting

The first milestone in each task should be a mandatory face to face meeting so that workers could resolve any ambiguity in the goals and the description of the task.


Power
HIT Design Standard platform

The standard platform for HIT design can eliminate many confusions a worker might face while doing the HIT and clarifies the HIT by standards, and enforce the worker to provide useful directives and procedure along with the HIT.

Needs of Requester

Trust
Online price calculator

Requesters should have access to a transparent online price calculator where they input amount of work required and the quality desired and the calculator shows minimum payment to both requester and worker.

Categorisation

There should be several categories that requesters could put their HIT there. Each category could have an estimated minimum price and average price that requesters could plan accordingly.

Power
Negotiation

There should be contract scheme with several pre-contracts that requesters could negotiate the price with workers so that both could agree upon.

Statistics

Requesters must have access to a statistical data gathered from the free market or from the website itself that determines the appropriate price for each type of work. For example the requester can see that if he/she wants his work done under a week and he/she has a particular type of work, then it could be easily determined that statistically a certain minimum price should be set.


Trust
Requester Reputation Builder

By having a requester reputation builder we enable the previous workers dealing with requester to rank them based on their promptness in payment, HIT design, fairness in payment and responsiveness. What about the new requesters? We set a value for default, the first HIT is the critical point for the requester to build their reputation, if they succeed in the first HIT and get satisfactory rates, they can maintain and even go up, but if they are scammers or "unfair" ones, they will rated downwardly.


Authenticity

An authenticity scheme should be implemented. For example, the owner of the website should provide a mechanism for requesters to provide IDs to the owner and the owner can confirm the authenticity of the requester.

Contracts

There should be a solid contract between requesters and workers where they can negotiate and come to their terms and there should be a mechanism to compensate for any damage when the contract is breached.


Power
Worker-System Ranking Mechanism

Predefining a set of standards, and qualifications, and terms and agreements, and contracting the requester in the first place, we can give a medium score to the worker to enter the market, and by posting a HIT we enable the Worker or the HIT Micro-Syndicate to vote and rank the requester and change them up. The system and the worker has the power to rate the requester and as mentioned earlier, the wight of the rate coming from the Micro-Syndicate has more importance than the individual weights, to avoid emotional involvement. Also we should enforce requester rating by the worker, maybe provide bonus for giving rank, and prompt them to rate the requester after they got the payment and justifications.


Breach of contract in an event of lack of trust

This seems out of reach but requesters should have a limited power to request a breach of contract if there is a lack of trust between workers and requesters.

Ability to advertise

Requesters should have the power to advertise on the platform. Advertisement is the key to establish trust between workers and requesters.

Dive Deeper into Specific Ideas

Deliverable

Top 2 Trust-based Idea

Trust Idea 1: Collective Statistics Panel

Trust Idea 2: Multilevel Result Delivery system to requesters

Top 2 Power-based Idea

Power Idea 1: A Three-priced Offer System with Negotiation Mechanism

Power Idea 1:Three complementary part HIT Tracer system

Dark Horse idea

Top 2 Dark Horse idea