Difference between revisions of "WinterMilestone 3 @ftw"

From crowdresearch
Jump to: navigation, search
(User Experience Flight)
 
(8 intermediate revisions by the same user not shown)
Line 1: Line 1:
Writeup for [[Winter Milestone 3 by team @FT]].
+
Writeup for [[Winter Milestone 3 by team @FTW]].
 
=Brainstorm for the following=
 
=Brainstorm for the following=
  
Line 42: Line 42:
  
 
===How would reputation work for new comers - workers or requesters?===
 
===How would reputation work for new comers - workers or requesters?===
 +
*newcomers just enter, let them explore
 +
*have the system choose tasks that they cannot fail at and incrementally become more difficult
 +
*offer professional paths based upon skill that match with different tasks
 +
*let them self-rate and go through the system
 +
*requestors would need to be informed with case studies about what doesn't help workers
 +
*requestors should be exposed to the turkers good work practices
 +
*workers have coaches tied to skill, system knowledge
 +
*both should be introduced to the system
 +
*both should be introduced to a short video of the typical other
 +
*interactions should be left to questions only with experts to begin
 +
*exclusive newbie zones
 +
 
===How do workers find relevant work, and requesters find ideal worker?===
 
===How do workers find relevant work, and requesters find ideal worker?===
  
Line 60: Line 72:
 
==Representation==
 
==Representation==
 
===How to make your voice heard? ===
 
===How to make your voice heard? ===
 +
*numbers in mass
 +
*big bold fonts
 +
*funny pictures and short gifs
 +
*sound effects like chimes when the ignorer login
 +
*match the level of the problem with the level of urgency (one person cant speak to 1000 at one)
 +
*novelty of the issue factored with self-determined likelihood of failure and the severity of failure
 +
*put a rating system everywhere, everything, everyone
 +
*game system similar to stackexchange
 +
*provide a mailing list, newsletter, and blog
 +
*use golden eggs through communication channels to incentivize involvement
 +
 
===How to represent concerns/rights as workers and requesters?===
 
===How to represent concerns/rights as workers and requesters?===
 
*automated rejection of work that meets requestor definition of spammers -- represented through the software as a regulatory mechanism
 
*automated rejection of work that meets requestor definition of spammers -- represented through the software as a regulatory mechanism
Line 71: Line 94:
 
*squirrel avators give away rewards
 
*squirrel avators give away rewards
 
*identify those who seem to display appropriate regulatory behaviors through event logs and recruit them as employees.
 
*identify those who seem to display appropriate regulatory behaviors through event logs and recruit them as employees.
 +
*be inspired by stack overflow...
 +
*notifications and updates that give enough time to others so that they can respond
 +
*provide enough time for any and all updates time to be resolved
 +
*consultations regarding development should be as broad as possible to include requestors, regulators, and workers
 +
*documentation and submissions regarding the governance should be publicly available
 +
*engagement with publics should cover multiple channels
 +
*concerns should be vetted through expert committees
 +
*traceable connections between updates in policies and known consequences
 +
*civil society, not ethics, should be monitored. ethics treated on a case-by-case basis
 +
*publish the details of algorithms used. always keep improving and changing to keep abusers behind. planned obsolescence. 
 +
*the timelines must be publicly available and provided in response to any adverse situations
 +
*awareness of development efforts or updates or changes must be scaled to relevant stakeholders
  
 
=== Who has the power to post work? ===
 
=== Who has the power to post work? ===
Line 113: Line 148:
 
*Third, they must be refined enough that they could be prototyped and objectively tested.
 
*Third, they must be refined enough that they could be prototyped and objectively tested.
  
=User Experience Flight=
+
=UX Flight Details Migrated=
Reach out to people within Crowd Research community, who are not participating in this test flight and get them to use a specific feature of Daemo and get their feedback on it. You are free to reach out to people outside Daemo too, but they must have minimal crowd platform experience and don't share it publicly. We're not ready for huge traffic at this point. Report a synthesis of your findings along with your own experience. Your report should include 1) actionable item, one that can be implemented into Daemo quickly; 2) good-to-have long term vision item, one that you'd like to see at some point of time. There is no limit to how many people you work with, it can be 1, it can be 10, it can be none (means you reflect your own findings). Note: be respectful of others time, and don't push people for it - you can always report your own experience only.
+
[[File:CrowdresearchMap-tasks.png|400px]]
+
 
+
'''Timing Results from 6 Daemo Tasks'''
+
{|
+
|''Task Description''
+
|''Simulated Expert Timings (sec)''
+
|''Script''
+
|-
+
|Create a Question: 5 Radio Button Response
+
|45.0
+
|[[File:TaskA1_Script.csv]]
+
|-
+
|Create a Question: create image labeling task - error
+
|23.0
+
|[[File:TaskA2_Script.csv]]
+
|-
+
|Create a Question: create image labeling task
+
|34.9
+
|[[File:TaskA3_Script.csv]]
+
|-
+
|Answer Question: 5 Radio Button Response
+
|6.7
+
|[[File:TaskB1_Script.csv]]
+
|-
+
|Answer Question: label the image task - error
+
|10.8
+
|[[File:TaskB2_Script.csv]]
+
|-
+
|Answer Question: label the image task
+
|14.0
+
|[[File:TaskB3_Script.csv]]
+
|}
+
 
+
[[File:6_Simulated_Daemo_Tasks.csv]]
+
 
+
===Requestor - Create a Question: 5 Radio Button Response===
+
In this task, a simulated human and a real human designed a 5 radio button response question. The requestor asked a question, which was then followed by a set of radio buttons. The simulated human, an artificial expert, was created with a GOMS-KLM model and repeated for this task. For the most part the creation of the response question was smooth. However the requestor demonstrated expert level difficulties with the system. The errors presented by the user, a demonstrated in the youtube video include the following.
+
 
+
{|style="vertical-align:top;"
+
|Time in Video  (m:ss)
+
|Error
+
|Correction
+
|-style="background-color: #CCCCCC; vertical-align:top;"
+
|0:10
+
|User applies CTRL+A shortcut that highlights entire page, rather than the text box content. Relies on mouse here and consecutively to highlight designed text.
+
|Automated clearing out of tips upon mouse click
+
|-style="vertical-align:top;"
+
|0:50, 0:58, 1:06
+
|User chases text cursor with mouse from Option 3 to Option 4, after Option 4 box loads prematurely.
+
|Keep the text cursor in the uppermost and cleared option.
+
|-style="background-color: #CCCCCC; vertical-align:top;"
+
|1:40
+
|User searches for what to do next
+
|Move the publish, save, and delete options to bottom of screen. The user needs to be led downward in this layout.
+
|-style="vertical-align:top;"
+
|2:00 (speed enhancement)
+
|User moves moves across screen to publish task.
+
|Move the publish, save, and delete options to bottom of screen to minimize mouse motion.
+
|}
+
 
+
[https://www.youtube.com/embed/urC1TefPc18 Youtube Video of Human Performing Task]
+
 
+
[[File:TaskA1_Script.csv]]
+
 
+
===Worker - Answer Question: 5 Radio Button Response===
+
Nothing to add.
+
 
+
[https://www.youtube.com/embed/zcu-rtyr-c8 Youtube Video of Human Performing Task]
+
 
+
[[File:TaskB1_Script.csv]]
+
 
+
===Requestor - Create a Question: create image labeling task ===
+
In this task, a simulated human and a real human designed an image labeling task. The requestor posted an image from the web, as per Daemo's current functionalities and asked workers to label the image. The simulated human, an artificial expert, was created with a GOMS-KLM model and repeated for both a task with and without an error. The error being that the requestor forgot to add a text box input for a requestor to insert the answer. In this case, the error can be identified by the time spent, number of motor operations the requestor performed on the task and the number of input options on the "authoring task" page.
+
 
+
[[File:Comparison-of-Requestor-Tasks.jpg|400px|center]]
+
 
+
Possible solutions to this issue include binding any number of Daemo's input options with the insertion of an image container. Further exploration identified that this same problem exists with audio container as well. Potentially the first word of the question typed by the requestor can be used to match the input device with the media container. For example, please "rate" this image would best go to a closed/constrained input such as a slider, drop down menu, or radio button. In contrast, tasks requesting please "describe" this image point towards open input method such as a text box.
+
 
+
[https://www.youtube.com/watch?v=lpfUZ5zU5QY Youtube Video of Human Performing Task Successfully]
+
 
+
[[File:TaskA3_Script.csv]]
+
 
+
[https://www.youtube.com/watch?v=h1hBOZQzvAg Youtube Video of Human Performing Task with Error]
+
 
+
[[File:TaskA2_Script.csv]]
+
 
+
===Worker - Answer Question: label the image task===
+
In this task, a simulated human and a real human designed an image labeling task. The requestor posted an image from the web, as per Daemo's current functionalities and asked workers to label the image. The simulated human, an artificial expert, was created with a GOMS-KLM model and repeated for both a task with and without an error. The error being that the requestor forgot to add a text box input for a requestor to insert the answer. The worker in response cannot submit a successful task for pay due to the error/
+
 
+
From the workers side, the error can be identified by the time spent, number of motor operations the requestor performed on the task, the number of input options on the "authoring task" page, and most strongly by the use of the "skip" button.
+
 
+
[[File:Comparison_of_Worker_Tasks.JPG|400px|center]]
+
 
+
[https://youtu.be/nsZZtWnzO7c Youtube Video of Human Performing Task Successfully]
+
 
+
[[File:TaskB3_Script.csv]]
+
 
+
[https://youtu.be/LOtdSlQVwEQ Youtube Video of Human Performing Task with Error]
+
 
+
[[File:TaskB2_Script.csv]]
+

Latest revision as of 08:54, 31 January 2016

Writeup for Winter Milestone 3 by team @FTW.

Brainstorm for the following

Reputation

How to improve reputation system beyond Boomerang? (read this paper to learn more about Boomerang).

  • at scale, does it matter? the person will get new opportunities without a rating
  • use knowledge from boomerang to corral users into skill set groups
  • specialize: use knowledge from boomerang to give people shorter tasks rather than delays. it results in the same thing but maybe it will lessen the cognitive load.
  • use boomerang to establish foundational relationships between worker-requestor that progress to bigger things
  • show losers the door to another site(MTurk)
  • open opportunities for people to collaborate or lose it
  • allow people to see successes beyond cash flow
  • allow newcomers to become reviewers
  • enable proportionality --- people should not do exceptionally more than they can have done in the past. baby steps.
  • if a person doesn't complete a task, find task properties that led to failure (language, questions, quantity,...)
  • match task type set by requestor with the current intent of the user.
  • the design should get people to work together more incrementally and focus on implementation goals as an heuristic

How would reputation work for new comers - workers or requesters?

  • newcomers just enter, let them explore
  • have the system choose tasks that they cannot fail at and incrementally become more difficult
  • offer professional paths based upon skill that match with different tasks
  • let them self-rate and go through the system
  • requestors would need to be informed with case studies about what doesn't help workers
  • requestors should be exposed to the turkers good work practices
  • workers have coaches tied to skill, system knowledge
  • both should be introduced to the system
  • both should be introduced to a short video of the typical other
  • interactions should be left to questions only with experts to begin
  • exclusive newbie zones

How do workers find relevant work, and requesters find ideal worker?

  • workers are matched with jobs based on their extensive worker profile

- desired wage

- time availability

- perceived quality of work

  • requesters based on the profile matches of workers but also based on some other aspects that can be fine-tuned by requester

- time limit for task to be completed

- expected level of quality

- number of tasks/batches that can be completed by the same worker

Representation

How to make your voice heard?

  • numbers in mass
  • big bold fonts
  • funny pictures and short gifs
  • sound effects like chimes when the ignorer login
  • match the level of the problem with the level of urgency (one person cant speak to 1000 at one)
  • novelty of the issue factored with self-determined likelihood of failure and the severity of failure
  • put a rating system everywhere, everything, everyone
  • game system similar to stackexchange
  • provide a mailing list, newsletter, and blog
  • use golden eggs through communication channels to incentivize involvement

How to represent concerns/rights as workers and requesters?

  • automated rejection of work that meets requestor definition of spammers -- represented through the software as a regulatory mechanism
  • emojis which represent different levels of emotions coupled with hastags and @person to highlight problems
  • a simplified bill of rights for all, read before entering the system. scaled violations enforced.
  • acceptance of public outbursts is encouraged in forums.
  • publicize extreme cases
  • avoid forms of public humiliation, guilt mongering, shaming
  • promote rational micro conversations about isolated incidents with appropriate and agreed to actions
  • tie a rating system to direct social connections and have direct impact on access, responsibility, and privilege levels
  • squirrel avators give away rewards
  • identify those who seem to display appropriate regulatory behaviors through event logs and recruit them as employees.
  • be inspired by stack overflow...
  • notifications and updates that give enough time to others so that they can respond
  • provide enough time for any and all updates time to be resolved
  • consultations regarding development should be as broad as possible to include requestors, regulators, and workers
  • documentation and submissions regarding the governance should be publicly available
  • engagement with publics should cover multiple channels
  • concerns should be vetted through expert committees
  • traceable connections between updates in policies and known consequences
  • civil society, not ethics, should be monitored. ethics treated on a case-by-case basis
  • publish the details of algorithms used. always keep improving and changing to keep abusers behind. planned obsolescence.
  • the timelines must be publicly available and provided in response to any adverse situations
  • awareness of development efforts or updates or changes must be scaled to relevant stakeholders

Who has the power to post work?

  • power to post goes to all, power to raise up some
  • give a random person on a team the ability to post at random, surprise intervals.
  • give a worker who is dedicated community rights to post good ideas from a few.
  • no one has the power to post. it will be AI overlords.
  • sponsor the ability to post work
  • advertise the ability to post work to only those who can afford it
  • only those workers/requestors who talk within a community
  • the person who pays into the system
  • anyone who is credentialed
  • a physical neighborhood in manhattan or ethiopa -- the extreme rich and extreme poor are represented
  • only people who have a turker conversion... worker becomes requestor who knows what it's like to be a regular working person, like joe plumber

What would open governance look like on Daemo?

  • punish privately, award publicly, support directly and behind the scenes
  • personalization of work screens according to level (controlled accessibility and level ups)
  • public cafes in forums
  • physically local cafes
  • private discussion threads, which a publicly known (open secrets)
  • increased pay/discounts for those that get involved tactfully with governance issues for time compensation
  • published academic work on the subject (domain researchers get special access to data beyond the visible and interviewable)
  • holacracy, two people circles get things done
  • a task cannot be completed until two have accepted, even if they don't know each other
  • special invitations to join stanford crew at conferences
  • public yet exclusive meetings with core daemo administration

BIG IDEA for Reputation

What are the goals of the design? For example, Google's Android design goals are: delight me in surprising ways, simplify my life, and make me amazing (e.g., grant me special powers). Which aspects of your design reflect each goal? How does your design solution addresses the users' needs?

BIG IDEA for Representation

What are the goals of the design? For example, Google's Android design goals are: delight me in surprising ways, simplify my life, and make me amazing (e.g., grant me special powers). Which aspects of your design reflect each goal? How does your design solution addresses the users' needs?

DARK HORSE

  • they must be "dark": they must explore a space that is risky, radical, infeasible, and/or in a direction orthogonal to previously explored solutions. *They should feel slightly uncomfortable.
  • Second, they must be brainstormed after the more traditional ideas — you can't have a dark path without a traditional "light" path to contrast it against.
  • Third, they must be refined enough that they could be prototyped and objectively tested.

UX Flight Details Migrated