Milestone 1 Burbankers

From crowdresearch
Jump to: navigation, search

Experience the life of a Worker on Mechanical Turk

As we had issues getting approved, we decided to focus more on the other aspects of Milestone 1, including the papers.

Experience the life of a Requester on Mechanical Turk

I signed up to be a requester on Mechanical Turk, but found the experience to be confusing. It took me a while to know exactly what I’m posting for my HIT. Thought the examples helped to some extent and showed me the variety of tasks that can be posted to Mechanical Turk, I found myself navigating back and forth between different links to understand how to complete the next step. I ended up trying to create what I thought was the simplest task listed in the examples (filling out a survey), since I got frustrated trying to figure out how to configure the more complicated tasks. Even doing this, I still made quite a few silly mistakes. One particular gaffe was mistakenly published the same HIT multiple times without realizing it.

In addition to the problems, I ran into creating my task, I had trouble understanding how to best optimize the different inputs (reward per assignment, time allotted, etc.) so that once my tasks were posted, workers would actually accept my tasks and complete them.

Perhaps I was being a little dim-witted and making the process more complicated than it needed to be, but if I were to actually post HITs to mechanical turk, I would have gotten frustrated very quickly and would have tried to find alternative ways to get the task done (or drop the task altogether).

Explore alternative crowd-labor markets

I spent some time looking through the TaskRabbit website to see what it was like to create tasks on their platform. It was a rather pleasant experience, as I felt they have done a good job streamlining the task creation process. On their homepage, they simply list out four main tasks someone might help with, and provide more detailed tasks below the fold. Once you begin creating a task, they continue to keep it simple by giving you a small set of options for you to choose from. And I especially appreciated that they kept all the different steps of the form on the same page, so that you can easily refer back to a different step in case you want to make a change. Lastly, they allowed me to see the different rates of the different taskers, so I didn't have to worry about optimizing for a certain price or duration to get my task done, or adjust my parameters in case there were not any taskers available.



  • Illustrates how a relatively more complex system like Mechanical Turk can be modified around its core principle of microtasking to reach an entirely new user base. This is a non-trivial task due to User Design differences/requirements as well as unknown user demand, and it sounds like the Authors of the paper were able to successfully overcome these barriers.
  • Ultimately, they were able to create a new method of microtasking that is more amenable to the 10 usability heuristics ( when applied to their designated market (Indian workers with basic cell phones).
  • Overall Strengths
    • Based on the reported quality metrics, it appears that the results were of very high quality while requiring relatively few workers (<2/<3).
    • We would ask whether this is due to the tasks being smaller (1-2 words) or whether it is due to the quality of the workers. This would be interesting to explore further.
    • Would ultimately raise the income by allowing workers to work during complementary parts of their day where they might otherwise be unable to earn any income. In effect, the ability to earn more without a "marginal cost" on the worker's / resources is profound.
  • Overall Weaknesses / Improvements to be Made
    • The system could be improved by understanding which tasks are best suited to being micro-tasked. It could be that OCR is particularly well suited for these phones/users while transcription might not be.
    • The simplicity of the tasks lends the tasks themselves to being automated in the near future as OCR platforms improve. The challenge will be maintaining the simplicity of tasks so that the users in this market can complete them while ensuring a decent payment amount for each task, all while providing enough value to the customer that uploads the tasks.
    • If you ultimately have "power users" on the platform that regularly complete tasks at the highest quality, there might be the opportunity to grantee them a wage to a certain degree, lending some stability to an otherwise unstable platform.


  • The usage of the SMS protocol for small images is both clever and supports the user in their current state. In addition, designing the transliteration component to allow for multiple different "transliterations" to count as correct shows a comprehensive understanding of the task flow and user experience.
  • Their initial experimentation in regards to sending messages was insightful, as it allowed them to design a system around SMS which the users ultimately received for free in most cases while also increasing the ultimate accuracy and ease of use of the system.
  • The fact that regression analysis of word length can predict the number of responses required to translate that word could ultimately better inform the compensation scheme.
  • Overall Strengths
    • The system's resilience, from mobile phone outages to compensation schemes to user interaction was incredibly impressive.
    • Despite high fee %'s built into the system, it was able to deliver a market competitive rate for its services while still competitive compensating users.
    • The ability to work with languages without font support could be more broadly applied than just mobile phones.
  • Overall Weaknesses / Improvements to be Made
    • Incredibly reliant on existing systems and processes to work on mobile carriers and free SMS plans, resilience of the system could be improved with partnerships in place and fall back methods of submitting tasks in the case of service outages.
    • Introducing more comprehensive and integrated gamification elements could likely further improve user engagement and task completion on the platform.
    • There was brief discussion around the cost structure of competing translation services, and there may be additional opportunity to pay workers on mClerk more while continuing to remain competitive. If the goal is the greatest good, then providing higher wages alongside partnerships that could be achieved as a non-profit might be worth exploring.

Flash Teams

  • Overall an incredibly insightful paper examining the polar opposite of traditional task management platforms which tend to focus only on the most simplistic and least cognitively difficult tasks. This could form the backbone for "continuous" development on complex projects whereas previously, teams had to wait for individuals to sleep. Similar to continuous deployment in software, if executed properly, we could eventually have continuous development of ideas.
  • Strengths
    • The insight is similar to cloud computing, in that you can instantiate or "spin up" an expert team based on the inputs and output goals. The initial structure of input/output allows less structure over "who" specifically needs to do it.
    • Resilience of the system is high, perhaps even more so due to the ability to slot in experts with different but complementary skill sets. This may automatically introduce idea diversity and structure that would otherwise not be present, potentially avoiding "group think" from the get go.
  • Weaknesses
    • Algorithmic matching of input to goal could lead to more steps than necessary. How do you find optimality?
    • May have a cold start problem of needing to populate the taxonomy of blocks and ensure they can accurately and successfully map to outputs.