Difference between revisions of "WinterMilestone 2@anotherhuman"
From crowdresearch
Aarongilbee (Talk | contribs) |
Aarongilbee (Talk | contribs) |
||
Line 37: | Line 37: | ||
## requestors "had less control over the [workers] participating... and variations caused by how prominently the study is displayed in the infrastructure search results." | ## requestors "had less control over the [workers] participating... and variations caused by how prominently the study is displayed in the infrastructure search results." | ||
##"even our study uses tasks that are much more complex and time consuming than those recommended by" Turk | ##"even our study uses tasks that are much more complex and time consuming than those recommended by" Turk | ||
+ | ##researchers "must consider if randomized assignment... is appropriate for their study" | ||
Stolee, K. & Elbaum, S. (2010) Written Statements about Workers | Stolee, K. & Elbaum, S. (2010) Written Statements about Workers | ||
Line 56: | Line 57: | ||
## researchers need "some understanding of the system capabilities and constraints" | ## researchers need "some understanding of the system capabilities and constraints" | ||
## the system needs to direct to other services in the business space that are more apt for the task at hand. | ## the system needs to direct to other services in the business space that are more apt for the task at hand. | ||
+ | ## requestors need to have the ability to control the presentation of their work, by priority, sequence, preference, iterative, random, importance... | ||
Needs identified during the worker-requestor interviews | Needs identified during the worker-requestor interviews |
Revision as of 06:43, 22 January 2016
Observations
- What observations about workers can you draw from the interview? Include any that may be are strongly implied but not explicit.
- What observations about requesters can you draw from the interview? Include any that may be are strongly implied but not explicit.
- What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.
- What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.
Stolee, K. & Elbaum, S. (2010) Written Statements about Requestors
- Requestor monitored the updates of tasks completion.
- Requestors devise methods to cull user diversity, user experience, and worker gaming of the system.
- Requestors devised routes to gather information that is normally anonymized by the turk system.
- "open ended answers helped us to understand points of confusion and why participants differed"
- "was the result of a misinterpreted question"
- requestors used the UI to approve work completed and access the results.
- a credit card is required to front load the requestor account.**********
- hit creation can be tested in the developer sandbox
- requestors find that Turk "provides a framework ... for recruiting, ensuring privacy, distributing payment, and collecting results."
- "results can be easily downloaded in a CSV format"
- Requestors "create custom qualification tests .. using the command line tool or API"
- Requestor who is a surveyor understand "the importance of having enough subjects (i.e. workers) of the right kind."
- Requestor "doubled the initial ... reward"
- requestor "sent emails to two internal mailing lists."
- Requestor might "observe students... instead of observing software engineers practicing."
- Requestor might "perform studies without human subjects." [bad practice]
- Requestor might "evaluate visualization designs, conduct surveys about information seeking behaviors, and perform NL annotations to train machine learning algorithms."
- Requestor might "leverage a global community... to solve a problem, classify data, refine a product, gather feedback"
- Requestor required "to pass a pretest."
- Researchers "estimated aptitude by measuring education and qualification score."
- Researchers create qualifications for works by using domain specific knowledge and quality of work history.
- Requestors evaluate work after completion.
- Resquestors made task templates and combined tasks with a shared type ID.
- Resquestors "presented [workers] with treated or untreated pipe for each task."
- Resquestors "could not impose their constraint and control for learning effects."
- Turk "caused us to waste some data."
- "An alternate [research] design would be to create..."
- requestors define the work goals, collect relevant information from the workers
- requestors "had less control over the [workers] participating... and variations caused by how prominently the study is displayed in the infrastructure search results."
- "even our study uses tasks that are much more complex and time consuming than those recommended by" Turk
- researchers "must consider if randomized assignment... is appropriate for their study"
Stolee, K. & Elbaum, S. (2010) Written Statements about Workers
- Workers might "select and configure predefined modules and connecting them."
- Workers try to avoid the search page and complete tasks.
- Workers see the qualifications but might not see the specifications of the requestor.
- Workers identify tasks that are of similar types to match their preferences.
- Workers "discover Hits by searching based on some criteria, such as titles, descriptions, keywords, reward or expiration date."
Need-finding
Stolee, K. & Elbaum, S. (2010)
- to eliminate worker variability
- to control variability
- researchers need "some understanding of the system capabilities and constraints"
- the system needs to direct to other services in the business space that are more apt for the task at hand.
- requestors need to have the ability to control the presentation of their work, by priority, sequence, preference, iterative, random, importance...
Needs identified during the worker-requestor interviews
- to identify patterns of requestors
- to be able to receive work immediately
- to be be able to pause work demands
- to be able to SLEEP
- to meet a daily goal (how dos this daily goal shape decision making)
- to quickly test a hypothesis
- to write clear instructions
- to post a batch of questions
- to receive information on how to improve task design
- to engaging in conversation with workers
- to how to monitor approval ratings
- to gauge the level of threat a requestor is towards approval ratings
- to scatter work across requestors
- to select those who provide HITs with good background
- to have a name and email to contact people before HITs
- to post hits without much interaction
- to know people are on the other side
- to label X of this item
- to optimize worker speed in job design an shape UI
- to avoid rejecting workers
- to maintain quality assurance of workers
- to send out sample hits to test task
- to gauge preferences of requestors
- to be able to manage interactions with workers at scale
- to manage worker correspondence
- to rate workers fairly
- to automatically clarify and challenge qualification based rejection
- to meet face to face with others
- to manage time better as quantity grows (the challenges and methods change)
- to connect with outside services that expand one's professional capacities
- to have strong best-worker relationships (10 batches)
- to pre-schedule work at intervals
- to assure quality, truthful and honest responses
- to connect with worker communities
- to have a hit within an iFrame
- to be able to build one's own efforts
- to look at the hit and estimate the value of effort... (paper: Estimating Charlie's Run-time Estimator)
- to manage the expectations of work involved and run times tied to HITs
- to know the other worker's personalities and how it affects work
- to communicate with others the reality of a job
- to match job type with work preferences and skill sets
- to earn as much profit (revenue) as possible
- to match typing tasks with those who like typing tasks
- to match button clickers with button clicking tasks
- to estimate the value of a day
- to have a buffer of work demands over time
- to avoid random responses
- to accept as work that is better than chance
- to avoid rejections on the reputation system
- to have a constant contact ability with the requestor
- to have a buffer of points for reputation management
- to have buffer time that accommodate interruptions and disabilities
incredible enablers of scientific research avoids: anonymous, frustrating, complicated, tricky absolutely unpredictable, great, terrible power, frustrating,empowering, lonely, isolated, tiring diverse and amazing futuristic dynamic humans
what rules of thumb do people use? to try 2 minutes of work and then retry
types or requestors: volume, income