Difference between revisions of "WinterMilestone 2@anotherhuman"

From crowdresearch
Jump to: navigation, search
Line 3: Line 3:
 
# What observations about requesters can you draw from the interview? Include any that may be are strongly implied but not explicit.
 
# What observations about requesters can you draw from the interview? Include any that may be are strongly implied but not explicit.
 
# What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.
 
# What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.
##Stolee, K. & Elbaum, S. (2010) Written Statements
 
 
 
# What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.
 
# What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.
  
Stolee, K. & Elbaum, S. (2010) Written Statements
+
Stolee, K. & Elbaum, S. (2010) Written Statements about Requestors
 
<blockquote>
 
<blockquote>
 
Requestor who is a surveyor understand "the importance of having enough subjects (i.e. workers) of the right kind."
 
Requestor who is a surveyor understand "the importance of having enough subjects (i.e. workers) of the right kind."
 +
</blockquote>
 +
<blockquote>
 +
Requestor might "observe students... instead of observing software engineers practicing."
 +
</blockquote>
 +
<blockquote>
 +
Requestor might "perform studies without human subjects." [bad practice]
 +
</blockquote>
 +
<blockquote>
 +
Requestor might "evaluate visualization designs, conduct surveys about information seeking behaviors, and perform NL annotations to train machine learning algorithms."
 +
</blockquote>
 +
<blockquote>
 +
Requestor might "leverage a global community... to solve a problem, classify data, refine a product, gather feedback"
 +
</blockquote>
 +
<blockquote>
 +
Requestor required "to pass a pretest."
 +
</blockquote>
 +
<blockquote>
 +
Researchers "estimated aptitude by measuring education and qualification score."
 +
</blockquote>
 +
<blockquote>
 +
Researchers create qualifications for works by using domain specific knowledge and quality of work history.
 
</blockquote>
 
</blockquote>
  
 +
Stolee, K. & Elbaum, S. (2010) Written Statements about Workers
  
 +
<blockquote>
 +
Workers might "select and configure predefined modules and connecting them."
 +
</blockquote>
 +
<blockquote>
 +
Workers identify tasks that are of similar types to match their preferences.
 +
</blockquote>
 +
<blockquote>
 +
Workers "discover Hits by searching based on some criteria, such as titles, descriptions, keywords, reward or expiration date."
 +
</blockquote>
  
  
Line 19: Line 48:
  
 
*Needs identified during the worker-requestor interviews
 
*Needs identified during the worker-requestor interviews
 +
** to eliminate worker variability
 +
** to control variability
 
** to identify patterns of requestors
 
** to identify patterns of requestors
 
** to be able to receive work immediately
 
** to be able to receive work immediately

Revision as of 05:35, 22 January 2016

Observations

  1. What observations about workers can you draw from the interview? Include any that may be are strongly implied but not explicit.
  2. What observations about requesters can you draw from the interview? Include any that may be are strongly implied but not explicit.
  3. What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.
  4. What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

Stolee, K. & Elbaum, S. (2010) Written Statements about Requestors

Requestor who is a surveyor understand "the importance of having enough subjects (i.e. workers) of the right kind."

Requestor might "observe students... instead of observing software engineers practicing."

Requestor might "perform studies without human subjects." [bad practice]

Requestor might "evaluate visualization designs, conduct surveys about information seeking behaviors, and perform NL annotations to train machine learning algorithms."

Requestor might "leverage a global community... to solve a problem, classify data, refine a product, gather feedback"

Requestor required "to pass a pretest."

Researchers "estimated aptitude by measuring education and qualification score."

Researchers create qualifications for works by using domain specific knowledge and quality of work history.

Stolee, K. & Elbaum, S. (2010) Written Statements about Workers

Workers might "select and configure predefined modules and connecting them."

Workers identify tasks that are of similar types to match their preferences.

Workers "discover Hits by searching based on some criteria, such as titles, descriptions, keywords, reward or expiration date."


Need-finding

  • Needs identified during the worker-requestor interviews
    • to eliminate worker variability
    • to control variability
    • to identify patterns of requestors
    • to be able to receive work immediately
    • to be be able to pause work demands
    • to be able to SLEEP
    • to meet a daily goal (how dos this daily goal shape decision making)
    • to quickly test a hypothesis
    • to write clear instructions
    • to post a batch of questions
    • to receive information on how to improve task design
    • to engaging in conversation with workers
    • to how to monitor approval ratings
    • to gauge the level of threat a requestor is towards approval ratings
    • to scatter work across requestors
    • to select those who provide HITs with good background
    • to have a name and email to contact people before HITs
    • to post hits without much interaction
    • to know people are on the other side
    • to label X of this item
    • to optimize worker speed in job design an shape UI
    • to avoid rejecting workers
    • to maintain quality assurance of workers
    • to send out sample hits to test task
    • to gauge preferences of requestors
    • to be able to manage interactions with workers at scale
    • to manage worker correspondence
    • to rate workers fairly
    • to automatically clarify and challenge qualification based rejection
    • to meet face to face with others
    • to manage time better as quantity grows (the challenges and methods change)
    • to connect with outside services that expand one's professional capacities
    • to have strong best-worker relationships (10 batches)
    • to pre-schedule work at intervals
    • to assure quality, truthful and honest responses
    • to connect with worker communities
    • to have a hit within an iFrame
    • to be able to build one's own efforts
    • to look at the hit and estimate the value of effort... (paper: Estimating Charlie's Run-time Estimator)
    • to manage the expectations of work involved and run times tied to HITs
    • to know the other worker's personalities and how it affects work
    • to communicate with others the reality of a job
    • to match job type with work preferences and skill sets
    • to earn as much profit (revenue) as possible
    • to match typing tasks with those who like typing tasks
    • to match button clickers with button clicking tasks
    • to estimate the value of a day
    • to have a buffer of work demands over time
    • to avoid random responses
    • to accept as work that is better than chance
    • to avoid rejections on the reputation system
    • to have a constant contact ability with the requestor
    • to have a buffer of points for reputation management
    • to have buffer time that accommodate interruptions and disabilities

incredible enablers of scientific research avoids: anonymous, frustrating, complicated, tricky absolutely unpredictable, great, terrible power, frustrating,empowering, lonely, isolated, tiring diverse and amazing futuristic dynamic humans

what rules of thumb do people use? to try 2 minutes of work and then retry

types or requestors: volume, income