Difference between revisions of "WinterMilestone 2@anotherhuman"
|Line 107:||Line 107:|
##turkers label individuals as flamers
##turkers label individuals as flamers
##turkers can be overly sensitive to a rejection
##turkers can be overly sensitive to a rejection
Revision as of 07:06, 22 January 2016
- What observations about workers can you draw from the interview? Include any that may be are strongly implied but not explicit.
- What observations about requesters can you draw from the interview? Include any that may be are strongly implied but not explicit.
- What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.
- What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.
Stolee, K. & Elbaum, S. (2010) Observations about Requestors
- Requestor monitored the updates of tasks completion.
- Requestors devise methods to cull user diversity, user experience, and worker gaming of the system.
- Requestors devised routes to gather information that is normally anonymized by the turk system.
- "open ended answers helped us to understand points of confusion and why participants differed"
- "was the result of a misinterpreted question"
- requestors used the UI to approve work completed and access the results.
- a credit card is required to front load the requestor account.**********
- hit creation can be tested in the developer sandbox
- requestors find that Turk "provides a framework ... for recruiting, ensuring privacy, distributing payment, and collecting results."
- "results can be easily downloaded in a CSV format"
- Requestors "create custom qualification tests .. using the command line tool or API"
- Requestor who is a surveyor understand "the importance of having enough subjects (i.e. workers) of the right kind."
- Requestor "doubled the initial ... reward"
- requestor "sent emails to two internal mailing lists."
- Requestor might "observe students... instead of observing software engineers practicing."
- Requestor might "perform studies without human subjects." [bad practice]
- Requestor might "evaluate visualization designs, conduct surveys about information seeking behaviors, and perform NL annotations to train machine learning algorithms."
- Requestor might "leverage a global community... to solve a problem, classify data, refine a product, gather feedback"
- Requestor required "to pass a pretest."
- Researchers "estimated aptitude by measuring education and qualification score."
- Researchers create qualifications for works by using domain specific knowledge and quality of work history.
- Requestors evaluate work after completion.
- Resquestors made task templates and combined tasks with a shared type ID.
- Resquestors "presented [workers] with treated or untreated pipe for each task."
- Resquestors "could not impose their constraint and control for learning effects."
- Turk "caused us to waste some data."
- "An alternate [research] design would be to create..."
- requestors define the work goals, collect relevant information from the workers
- requestors "had less control over the [workers] participating... and variations caused by how prominently the study is displayed in the infrastructure search results."
- "even our study uses tasks that are much more complex and time consuming than those recommended by" Turk
- researchers "must consider if randomized assignment... is appropriate for their study"
Martin, D. & et al. (2014) Observations about Requestors
- to request work such as image tagging, duplicate recognition, transcription, translation, object classification, and generate content
- rely on turk to curate and manage the quality of content for their tasks
- requesters becomes confused about what actions constitute a bad worker [one man's trash is another's treasure]
- requestors block who they consider bad workers
- requestors fundamentally ask for help from the masses and a judgment from the asker is fundamentally a dynamic that is highly disrespectful
Stolee, K. & Elbaum, S. (2010) Observations about Workers
- Workers might "select and configure predefined modules and connecting them."
- Workers try to avoid the search page and complete tasks.
- Workers see the qualifications but might not see the specifications of the requestor.
- Workers identify tasks that are of similar types to match their preferences.
- Workers "discover Hits by searching based on some criteria, such as titles, descriptions, keywords, reward or expiration date."
Martin, D. & et al. (2014) Observations about Workers
- "view AMT as a labor market"
- "unfair rejection of work"
- "to receive pay for work"
- communicate with others through Turk
- workers identify scams
- workers move through poorly designed tasks
- workers develop relationships with requestors
- workers seek some form of relational reciprocity with requestors
- workers gather to collect information about tasks, the platform, and requestors
- workers protect their hall of fame and shame post at http://turkernation.com/forumdisplay.php?13-Requesters-Hall-of-Fame-Shame
- workers find work that they're happy with despite pay rate
- workers discuss money and methods to earn it best
- workers talk about fun, learning and play as a major reason for joining MTurk
- workers earn cash on the Mturk system
- workers contrasts tasks upon a play/pay continuum
- workers criticize the pay attitude in the forums
- workers rely upon MTurk to accentuate cash flow when real world work stops (The purpose for turk changes)
- workers select only the "best" oportunities for pay
- turkers compete with one another and ask questions regard what others earn
- turkers set their own targets
- turkers respond to external events in their lives and adjust how they interact with turk based from those events
- turkers schedule and allot certain times of the day to be on MTurk
- workers rely on it as a source of income -- partially because mturk is available, accessible, and easy to find work due to its requestor diversity
- workers might use turk as a breadline
- workers find mturk ideal because one doesn't have to consider the professional environment and transportation concerns
- workers avoid those requestors who are demeaning and practice mass rejection
- workers compare experiences
- workers seek out requestors based from responsiveness
- workers give positive and negative badges
- workers spend time searching for jobs
- workers need access to decent work
- workers avoid being blocked by requestors
- workers design HITs with requestors
- workers self-monitor communication practices with requestors
- workers expect quick pay
- workers sample tasks to test the requestor
- workers bag several hits from one requestor
- workers will work on several hits in multiple tabs in a browser
- workers examine how quickly a requestor responds to questions
- workers avoid majority rules grading practices -- probably used in ML labeling tasks
- turkers use a consensus scheme to assess requestors in the forums
- turkers follow rules of requestor good practice
- turkers seek to know how often a requestor is online, quick to responsive he is to a task, how polite the person is
- turkers base trust upon several dimesions - competence of the requestor, concern of the requestor, and the intergrity (consistency) of interactions with the requestor
- some turkers scam, others try to solve this scamming through social governance
- in threads inidividuals are accused of cheating qualifications
- turkers suffer from fatigue ("i was not paying enough attention)
- turker practice reciprocity with requestors
- turkers label individuals as flamers
- turkers can be overly sensitive to a rejection
- workers might loss work due to a bad connection if work is saved on the cloud.
Martin, D. & et al. (2014)
- "how to motivate better, cheaper, and faster performance... without paying much"
Stolee, K. & Elbaum, S. (2010)
- to eliminate worker variability
- to control variability
- researchers need "some understanding of the system capabilities and constraints"
- the system needs to direct to other services in the business space that are more apt for the task at hand.
- requestors need to have the ability to control the presentation of their work, by priority, sequence, preference, iterative, random, importance...
Needs identified during the worker-requestor interviews
- to identify patterns of requestors
- to be able to receive work immediately
- to be be able to pause work demands
- to be able to SLEEP
- to meet a daily goal (how dos this daily goal shape decision making)
- to quickly test a hypothesis
- to write clear instructions
- to post a batch of questions
- to receive information on how to improve task design
- to engaging in conversation with workers
- to how to monitor approval ratings
- to gauge the level of threat a requestor is towards approval ratings
- to scatter work across requestors
- to select those who provide HITs with good background
- to have a name and email to contact people before HITs
- to post hits without much interaction
- to know people are on the other side
- to label X of this item
- to optimize worker speed in job design an shape UI
- to avoid rejecting workers
- to maintain quality assurance of workers
- to send out sample hits to test task
- to gauge preferences of requestors
- to be able to manage interactions with workers at scale
- to manage worker correspondence
- to rate workers fairly
- to automatically clarify and challenge qualification based rejection
- to meet face to face with others
- to manage time better as quantity grows (the challenges and methods change)
- to connect with outside services that expand one's professional capacities
- to have strong best-worker relationships (10 batches)
- to pre-schedule work at intervals
- to assure quality, truthful and honest responses
- to connect with worker communities
- to have a hit within an iFrame
- to be able to build one's own efforts
- to look at the hit and estimate the value of effort... (paper: Estimating Charlie's Run-time Estimator)
- to manage the expectations of work involved and run times tied to HITs
- to know the other worker's personalities and how it affects work
- to communicate with others the reality of a job
- to match job type with work preferences and skill sets
- to earn as much profit (revenue) as possible
- to match typing tasks with those who like typing tasks
- to match button clickers with button clicking tasks
- to estimate the value of a day
- to have a buffer of work demands over time
- to avoid random responses
- to accept as work that is better than chance
- to avoid rejections on the reputation system
- to have a constant contact ability with the requestor
- to have a buffer of points for reputation management
- to have buffer time that accommodate interruptions and disabilities
incredible enablers of scientific research avoids: anonymous, frustrating, complicated, tricky absolutely unpredictable, great, terrible power, frustrating,empowering, lonely, isolated, tiring diverse and amazing futuristic dynamic humans
what rules of thumb do people use? to try 2 minutes of work and then retry
types or requestors: volume, income