Difference between revisions of "WinterMilestone 2@anotherhuman"

From crowdresearch
Jump to: navigation, search
Line 6: Line 6:
  
 
Stolee, K. & Elbaum, S. (2010) Written Statements about Requestors
 
Stolee, K. & Elbaum, S. (2010) Written Statements about Requestors
<blockquote>
+
 
Requestor who is a surveyor understand "the importance of having enough subjects (i.e. workers) of the right kind."
+
##Requestor who is a surveyor understand "the importance of having enough subjects (i.e. workers) of the right kind."
</blockquote>
+
##Requestor might "observe students... instead of observing software engineers practicing."
<blockquote>
+
##Requestor might "perform studies without human subjects." [bad practice]
Requestor might "observe students... instead of observing software engineers practicing."
+
##Requestor might "evaluate visualization designs, conduct surveys about information seeking behaviors, and perform NL annotations to train machine learning algorithms."
</blockquote>
+
##Requestor might "leverage a global community... to solve a problem, classify data, refine a product, gather feedback"
<blockquote>
+
##Requestor required "to pass a pretest."
Requestor might "perform studies without human subjects." [bad practice]
+
##Researchers "estimated aptitude by measuring education and qualification score."
</blockquote>
+
##Researchers create qualifications for works by using domain specific knowledge and quality of work history.
<blockquote>
+
##Requestors evaluate work after completion.
Requestor might "evaluate visualization designs, conduct surveys about information seeking behaviors, and perform NL annotations to train machine learning algorithms."
+
##Resquestors made task templates and combined tasks with a shared type ID.
</blockquote>
+
##Resquestors "presented [workers] with treated or untreated pipe for each task."
<blockquote>
+
##Resquestors "could not impose their constraint and control for learning effects."
Requestor might "leverage a global community... to solve a problem, classify data, refine a product, gather feedback"
+
##Turk "caused us to waste some data."
</blockquote>
+
##"An alternate [research] design would be to create..."
<blockquote>
+
##
Requestor required "to pass a pretest."
+
</blockquote>
+
<blockquote>
+
Researchers "estimated aptitude by measuring education and qualification score."
+
</blockquote>
+
<blockquote>
+
Researchers create qualifications for works by using domain specific knowledge and quality of work history.
+
</blockquote>
+
<blockquote>
+
Requestors evaluate work after completion.
+
</blockquote>
+
<blockquote>
+
Resquestors made task templates and combined tasks with a shared type ID.
+
</blockquote>
+
<blockquote>
+
Resquestors "presented [workers] with treated or untreated pipe for each task."
+
</blockquote>
+
<blockquote>
+
Resquestors "could not impose their constraint and control for learning effects."
+
</blockquote>
+
<blockquote>
+
Resquestors "caused up to waste some data."
+
</blockquote>
+
  
 
Stolee, K. & Elbaum, S. (2010) Written Statements about Workers
 
Stolee, K. & Elbaum, S. (2010) Written Statements about Workers
  
<blockquote>
+
##Workers might "select and configure predefined modules and connecting them."
Workers might "select and configure predefined modules and connecting them."
+
##Workers try to avoid the search page and complete tasks.
</blockquote>
+
##Workers try to avoid the search page and complete tasks.
<blockquote>
+
##Workers identify tasks that are of similar types to match their preferences.
Workers try to avoid the search page and complete tasks.
+
##Workers "discover Hits by searching based on some criteria, such as titles, descriptions, keywords, reward or expiration date."
</blockquote>
+
##
<blockquote>
+
Workers try to avoid the search page and complete tasks.
+
</blockquote>
+
<blockquote>
+
Workers identify tasks that are of similar types to match their preferences.
+
</blockquote>
+
<blockquote>
+
Workers "discover Hits by searching based on some criteria, such as titles, descriptions, keywords, reward or expiration date."
+
</blockquote>
+
  
  
Line 68: Line 36:
 
Need-finding
 
Need-finding
  
*Needs identified during the worker-requestor interviews
+
## Needs identified during the worker-requestor interviews
** to eliminate worker variability
+
## to eliminate worker variability
** to control variability
+
## to control variability
** to identify patterns of requestors
+
## to identify patterns of requestors
** to be able to receive work immediately
+
## to be able to receive work immediately
** to be be able to pause work demands
+
## to be be able to pause work demands
** to be able to SLEEP
+
## to be able to SLEEP
** to meet a daily goal (how dos this daily goal shape decision making)
+
## to meet a daily goal (how dos this daily goal shape decision making)
** to quickly test a hypothesis
+
## to quickly test a hypothesis
** to write clear instructions
+
## to write clear instructions
** to post a batch of questions  
+
## to post a batch of questions  
** to receive information on how to improve task design
+
## to receive information on how to improve task design
** to engaging in conversation with workers
+
## to engaging in conversation with workers
** to how to monitor approval ratings
+
## to how to monitor approval ratings
** to gauge the level of threat a requestor is towards approval ratings
+
## to gauge the level of threat a requestor is towards approval ratings
** to scatter work across requestors
+
## to scatter work across requestors
** to select those who provide HITs with good background
+
## to select those who provide HITs with good background
** to have a name and email to contact people before HITs
+
## to have a name and email to contact people before HITs
** to post hits without much interaction
+
## to post hits without much interaction
** to know people are on the other side
+
## to know people are on the other side
** to label X of this item
+
## to label X of this item
** to optimize worker speed in job design an shape UI
+
## to optimize worker speed in job design an shape UI
** to avoid rejecting workers
+
## to avoid rejecting workers
** to maintain quality assurance of workers
+
## to maintain quality assurance of workers
** to send out sample hits to test task
+
## to send out sample hits to test task
** to gauge preferences of requestors
+
## to gauge preferences of requestors
** to be able to manage interactions with workers at scale
+
## to be able to manage interactions with workers at scale
** to manage worker correspondence
+
## to manage worker correspondence
** to rate workers fairly
+
## to rate workers fairly
** to automatically clarify and challenge qualification based rejection
+
## to automatically clarify and challenge qualification based rejection
** to meet face to face with others
+
## to meet face to face with others
** to manage time better as quantity grows (the challenges and methods change)
+
## to manage time better as quantity grows (the challenges and methods change)
** to connect with outside services that expand one's professional capacities
+
## to connect with outside services that expand one's professional capacities
** to have strong best-worker relationships (10 batches)
+
## to have strong best-worker relationships (10 batches)
** to pre-schedule work at intervals  
+
## to pre-schedule work at intervals  
** to assure quality, truthful and honest responses
+
## to assure quality, truthful and honest responses
** to connect with worker communities
+
## to connect with worker communities
** to have a hit within an iFrame
+
## to have a hit within an iFrame
** to be able to build one's own efforts
+
## to be able to build one's own efforts
** to look at the hit and estimate the value of effort... (paper: Estimating Charlie's Run-time Estimator)
+
## to look at the hit and estimate the value of effort... (paper: Estimating Charlie's Run-time Estimator)
** to manage the expectations of work involved and run times tied to HITs
+
## to manage the expectations of work involved and run times tied to HITs
** to know the other worker's personalities and how it affects work
+
## to know the other worker's personalities and how it affects work
** to communicate with others the reality of a job
+
## to communicate with others the reality of a job
** to match job type with work preferences and skill sets
+
## to match job type with work preferences and skill sets
** to earn as much profit (revenue) as  possible
+
## to earn as much profit (revenue) as  possible
** to match typing tasks with those who like typing tasks
+
## to match typing tasks with those who like typing tasks
** to match button clickers with button clicking tasks
+
## to match button clickers with button clicking tasks
** to estimate the value of a day
+
## to estimate the value of a day
** to have a buffer of work demands over time
+
## to have a buffer of work demands over time
** to avoid random responses
+
## to avoid random responses
** to accept as work that is better than chance
+
## to accept as work that is better than chance
** to avoid rejections on the reputation system
+
## to avoid rejections on the reputation system
** to have a constant contact ability with the requestor
+
## to have a constant contact ability with the requestor
** to have a buffer of points for reputation management
+
## to have a buffer of points for reputation management
** to have buffer time that accommodate interruptions and disabilities
+
## to have buffer time that accommodate interruptions and disabilities
  
 
incredible enablers of scientific research
 
incredible enablers of scientific research

Revision as of 04:50, 22 January 2016

Observations

  1. What observations about workers can you draw from the interview? Include any that may be are strongly implied but not explicit.
  2. What observations about requesters can you draw from the interview? Include any that may be are strongly implied but not explicit.
  3. What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.
  4. What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

Stolee, K. & Elbaum, S. (2010) Written Statements about Requestors

    1. Requestor who is a surveyor understand "the importance of having enough subjects (i.e. workers) of the right kind."
    2. Requestor might "observe students... instead of observing software engineers practicing."
    3. Requestor might "perform studies without human subjects." [bad practice]
    4. Requestor might "evaluate visualization designs, conduct surveys about information seeking behaviors, and perform NL annotations to train machine learning algorithms."
    5. Requestor might "leverage a global community... to solve a problem, classify data, refine a product, gather feedback"
    6. Requestor required "to pass a pretest."
    7. Researchers "estimated aptitude by measuring education and qualification score."
    8. Researchers create qualifications for works by using domain specific knowledge and quality of work history.
    9. Requestors evaluate work after completion.
    10. Resquestors made task templates and combined tasks with a shared type ID.
    11. Resquestors "presented [workers] with treated or untreated pipe for each task."
    12. Resquestors "could not impose their constraint and control for learning effects."
    13. Turk "caused us to waste some data."
    14. "An alternate [research] design would be to create..."

Stolee, K. & Elbaum, S. (2010) Written Statements about Workers

    1. Workers might "select and configure predefined modules and connecting them."
    2. Workers try to avoid the search page and complete tasks.
    3. Workers try to avoid the search page and complete tasks.
    4. Workers identify tasks that are of similar types to match their preferences.
    5. Workers "discover Hits by searching based on some criteria, such as titles, descriptions, keywords, reward or expiration date."


Need-finding

    1. Needs identified during the worker-requestor interviews
    2. to eliminate worker variability
    3. to control variability
    4. to identify patterns of requestors
    5. to be able to receive work immediately
    6. to be be able to pause work demands
    7. to be able to SLEEP
    8. to meet a daily goal (how dos this daily goal shape decision making)
    9. to quickly test a hypothesis
    10. to write clear instructions
    11. to post a batch of questions
    12. to receive information on how to improve task design
    13. to engaging in conversation with workers
    14. to how to monitor approval ratings
    15. to gauge the level of threat a requestor is towards approval ratings
    16. to scatter work across requestors
    17. to select those who provide HITs with good background
    18. to have a name and email to contact people before HITs
    19. to post hits without much interaction
    20. to know people are on the other side
    21. to label X of this item
    22. to optimize worker speed in job design an shape UI
    23. to avoid rejecting workers
    24. to maintain quality assurance of workers
    25. to send out sample hits to test task
    26. to gauge preferences of requestors
    27. to be able to manage interactions with workers at scale
    28. to manage worker correspondence
    29. to rate workers fairly
    30. to automatically clarify and challenge qualification based rejection
    31. to meet face to face with others
    32. to manage time better as quantity grows (the challenges and methods change)
    33. to connect with outside services that expand one's professional capacities
    34. to have strong best-worker relationships (10 batches)
    35. to pre-schedule work at intervals
    36. to assure quality, truthful and honest responses
    37. to connect with worker communities
    38. to have a hit within an iFrame
    39. to be able to build one's own efforts
    40. to look at the hit and estimate the value of effort... (paper: Estimating Charlie's Run-time Estimator)
    41. to manage the expectations of work involved and run times tied to HITs
    42. to know the other worker's personalities and how it affects work
    43. to communicate with others the reality of a job
    44. to match job type with work preferences and skill sets
    45. to earn as much profit (revenue) as possible
    46. to match typing tasks with those who like typing tasks
    47. to match button clickers with button clicking tasks
    48. to estimate the value of a day
    49. to have a buffer of work demands over time
    50. to avoid random responses
    51. to accept as work that is better than chance
    52. to avoid rejections on the reputation system
    53. to have a constant contact ability with the requestor
    54. to have a buffer of points for reputation management
    55. to have buffer time that accommodate interruptions and disabilities

incredible enablers of scientific research avoids: anonymous, frustrating, complicated, tricky absolutely unpredictable, great, terrible power, frustrating,empowering, lonely, isolated, tiring diverse and amazing futuristic dynamic humans

what rules of thumb do people use? to try 2 minutes of work and then retry

types or requestors: volume, income