Worker-Requester Panel: Meeting Notes by Dcthompson & Willtchiu
This is a compilation of notes by @dcthompson and @willtchiu for the Worker-requester panel meeting held on January 19, 2016. Youtube link
(Q for question, A for answer)
"This is my life, and I have to make this work for me.” (or something very close to that …)
Q - What is the daily rhythm? What do the hours look like?
A - I get up with my kids, get breakfast for self and children. Turn on scripts, working through TurkerNation Daily Thread (cooperative forum). Engage with requesters and amongst workers in an open and transparent way.
The Thread contains good work. Set PANDA (Preview and Accept) link. Work through Thread, do work, revisit update queue (clearing queue, revisiting thread). TurkerNation chat window is open throughout.
(How long do you work for at a time, what’s the pulse?)
A - Don’t schedule breaks. If my favorite HIT is up, or if something well paying is up, I will work on those. Don’t know about when HITs will post. No schedule, not sure when something will post. “Do I not eat lunch?” This is a 24 hour cycle, always something going on. Could be something lucrative. Lots of systems, snapchat groups to alert when work is available. No typical day or time that you can plan for work to be available. For people who rely on a planned schedule, this doesn’t work. Unpredictable.
A - Having a daily goal, and being able to achieve this despite the unpredictability.
Q (For requesters) What’s the process for designing and iterating on HITs. What works well, what really aggravates you?
A - Varies a lot. Many HITs are designed on the academic side to make a point in a paper or to explore a hypothesis. One of the nice things about MTurk is that we can quickly design and deploy for rapid live feedback. Where it becomes important to put clarity in is regarding the task and what we need to clearly convey the work that needs to be done.
Q - When have you thought you were clear, and weren’t?
A - Good question. Oftentimes workers are awesome about providing feedback. Would be good to have a better way to manage that engagement. Perhaps ramp up engagement on TurkerNation. Importance of conversation.
A - Never risk going below a 99% Worker rating. Uses a script to ensure that even if everything fails, worker rating stays high. (elaborate?)
A - Not putting too many eggs in one basket. Worker approval rating high. Using requester reputation as a proxy for trust. Does try new requesters. Uses communication to probe and guide. Simple communication with requester seems to be important to building trust.
Q - Given probable increase of platforms, how are we thinking through, and enabling, the management of new managers?
A - Assumed good quality of work. Automatic approval, but had some bad eggs. Then moved to automatic provision of qualification, with removal of qualification if poor quality found.
A - Prof. outsources crowdsourcing requests and conversation with workers. Uses an UG. How are you managing your own reputation as a requester to ensure good work, with good people? In ideal work, there would be automatic adjudication process … Adjudication happening through peer-review (seems like making more work for other ‘gold standard’ workers). Emphasizes importance of fairness. Shown as serial numbers - how can the interaction be made more empathetic? How can requesters treat them as people?
Q - (to requesters) What can’t you do currently? How to engage smart folks from around the World?
A - As a typical survey requester how do deal with the attention span of workers? [Completely didn’t answer the question]
A - Doesn’t use any of the built in templates. [Didn’t answer the question, really - instead shared the fix]
Q - How do you estimate payment?
A - Look at HIT and make an estimate (usually for batch hits - i.e. tagging). Time estimates for survey are typically off (over/under estimate on time to complete). Guinea pig and then sharing time estimates on forums (not just taking, but sharing also - management of commons).
A - Try 2 minutes worth of HITs, calculator and spreadsheet. You can’t regulate an hourly rate, because individuals are so different (choppy audio).
A - Leverage individual skill sets. Fast typing is good for data entry/transcription.
Q - How variable are your earnings?
A - All over the place. Hundreds of dollars in one day, couple the next. Depends on how much time you can devote. What’s the best use of your time? Earning variation is huge, and unpredictable. Difficult to budget.
crowd-workers.com - Chrome extension useful for evaluating time / task
Q - What does the negotiation around rejection look like?
A - Requester effort against task versus clarity of task. Can you assess random behavior? If you can evaluate or assume good outcomes then all good. The rub (of course) lies in the middle. Tricky subject.
A - Worker Even just a few rejections will prevent participation in high paying work. Every survey/every batch have to ensure that there’s a path to contacting the requester. Taking multiple screenshots - if rejection, use the ‘paper trail’ to have a conversation with the requester about why. The importance of ‘backup’ to inform the negotiation and conversation with the requester.
Q - If you could use 5 words to describe your work what would they be? A - (R on W) Incredible enablers of scientific research A - (W on R) Anonymous, frustrating, complicated, tricky (very tricky) A - (W on R) Absolutely unpredictable, great, terrible A - (R <-> W) Power (imbalance), frustrating, empowering, lonely, isolated, tiring A - (R on W) Diverse, passionate, futuristic, dynamic, humans
- platform that encourages ethical behaviour in workers, good at teaching new turkers the proper way of doing HITs
- advises requesters on appropriate qualifications needed
- Problem: HITs posted in TurkerNation chat room, sometimes not visible in threads.
- On tackling HITs: Chooses HITs that do not risk approval rating (i.e. 99%)
- Kept track of HITs, payment, and time it took.
- Problem: Gauging time spent on HITs are hard, because different people have different skill sets (i.e. some are slower, faster, more conscientious)
- Suggestion: Some sort of hourly wage prediction?
- Problem: New worker’s reputation is very volatile, takes only 1 HIT to hurt < 100 HIT workers
- Idea: Streaks of accepted HITs give an extra boost to a worker’s reputation; makes it more forgiving if failing a HIT, yet more rewarding for consistently doing well
- Problem: Unpredictable when HITs are posted
- Prioritize higher paying HITs
- HITs are never scheduled out, edge of the seat waiting for HITs
- Need to make decisions between life and doing HITs
- HITs are posted at all times of the day
- alert groups that send out messages when new HITs are posted
- Prioritizes worker approval ratings first, and requesters who have a good reputation
- Suggestion: Wants better interaction with requesters to clarify HITs
- Suggestion: People on forums give feedback on HITs; especially time estimates are important
- Problem: Fluctuation of wages are very unpredictable; hard to budget income or spending because of this.
- Need: Tools similar to http://crowd-workers.com
- Workers are awesome at emailing requesters if HITs are unclear
- Suggestion: It’ll be awesome to have more interaction between workers and requesters.
- Problem: When scaling HITs to much larger volume, requester’s time to respond to HITs is limited and they outsource, in effect, the job of answering Turkers’ emails.
- Needs: Automatic judiciary system that will handle disputed cases for approval or disapproval.
- Idea: Turkers who have a very high approval rating can opt to be judges of cases of dispute
- http://crowd-workers.com (Tracks hourly wage, time completion, and HIT specifications)
- Posted HITs on labels on bookshelves
- Concern: We care about quality, and if a worker falls below a certain percentage of approval then we reject them.
- Need: A HIT inside an iframe, customizable interface for HITs
- Posts survey type of HITs
- Problem: Hard to gauge attention span of survey takers, ensure truthful answers
SpamGirl (Kristy, Worker and Requester)
- Need: Built-in workflows capabilities
@Dcthompson and @Willtchiu