WinterMilestone 1 Team 1

From crowdresearch
Jump to: navigation, search

Experience the life of a Worker on Mechanical Turk

Reflect on your experience as a worker on Mechanical Turk. What did you like? What did you dislike?

Mechanical Turk seems to be a platform in which there is a strangely difficult admission (a lot of application from the outside the U.S. got rejected), a set of repetitive and low-profile tasks awarded with very low and unfair wages compared to "western countries". It seems like it is a service for people who reside in an area where one or two dollars more can make a difference, even if sometimes the job is not even paid due to lack of technical binding to money flows when the job is done. Doing what probably most users would do, after playing around with a couple of image tagging and confirming website links (<0.05$ per HIT), we started looking what this platform can offer. Arranging everything by potential pay, everyone had to go through literally dozens of pages in hope to find a task that pays decent. After about 20 minutes of scrolling through pages we gave up and went to finish tasks that seemed easy enough to be paid for. Plus, one can't know when he/she is going to receive the approval or rejection of the hit. Also, a user approach of this platform doesn't seem to be easy and intuitive.

When it comes to the things our team liked, it was a capacity of sorting the hits based on many choices (creation date, expiration date,reward amount,...) and search bar where one can enter keywords of his/her target hit (survey, image and so on...).

Experience the life of a Requester on Mechanical Turk

Reflect on your experience as a requester on Mechanical Turk. What did you like? What did you dislike? Also attach the CSV file generated when you download the HIT results.

While working as as requester, out teams liked these next features: possibility to choose a suitable category with examples in project creation, the system is enabled to edit the hits if one would want to change anything and a page called manage hits, which enables one to see number of submissions he/she received and if he/sheI want to expire the hit early.

Which problems have been met? Posting tasks can be quite daunting and extremely limiting in terms of the actions your workers can take. Not only that but the User Interface lack pretty much everything except probably the functional aspect, but which without intuitiveness loses its purpose. We faced some issues to put many images in single project, so MT must clarify the way to do that. As a requester one has the right to reject workers’ submission whatever they did and this pretty unfair.

We think the evaluation of the job done could be a very difficult point in the flow of the worker/client supply. If the job is completed in its form but wrong in the contents it will be hard to proof the job is ok and get paid by the worker or get refunded by the client prooving the job isn't well done at all. We suggest the creation of a channel for result evaluation in the crowdsourcing process.

File:Requester results.csv

Explore alternative crowd-labor markets

Compare and contrast the crowd-labor market you just explored (TaskRabbit/oDesk/GalaxyZoo) to Mechanical Turk.

Overall opinion on MT: This platform can and does work; however, going through and finding a task that pays decent and may not need approval, where the objective makes sense and it stays away from subjective bias that may result in the requester “thinking” you have not accomplished his desire makes this platform a torturer in its own right. However, coupled with some smart tools and enough perseverance to go through with repetitive small (but certain) tasks I believe this platform offers a great deal of opportunity to workers in countries suffering from economic inequality.

Microworker platform seems to be similar to Mechanical Turk. Only small, poorly-paid job can be performed at this platform and the relation cost/hour doesn’t seem to be respected. To request a job, you need to have 10 dollars on your account which is not useful when you want to have only a small job done and you can’t request to complete too many tasks in a single campaign which limits us to create small jobs only.

CrowdFlower platform has more intuitive interface then MT and Microworkers, giving straight idea of what one need to do in order get people do the job.

TaskRabbit seems to be a fully developed platform that allows users to apply and offer jobs in the full spectrum of human capability. After spending 30-45 minutes on the application, they thanked for applying to become a Tasker however they were unable to move forward with the application at this time. But during this application one was required to fill in the fields and we have to admit we were impressed by the breadth of fields where one can apply. Their approach was quite simple: select hourly pay, personal pitch and level of expertise (4 levels: complete beginner, did it around the house, part-time and diploma backed expertise). In the latter, we found that the 4 levels were just not enough, having to put my knowledge (outside webdev and data processing) mostly between did it around the home and part-time. In either cases, our team ended up feeling like we were either jacks of all trades who learned and did everything around the home or someone that has had too many part-time jobs.

upWork This is fully developed platform that allow users to apply and offer jobs in the creative or developer industry (mostly). We have only tried it as a freelancer. This seems to be the most popular platform for freelancers. We liked creating and profile and with all the skill-evaluation it feels like this is something that can truly reflect one’s level of expertise. However, often you’d have to compromise when focusing on your main abilities as there was only one field of selection. In the end, the profile is very important in terms of matching you to employers based on both skill and personality but one can get lost in all the processes with no real guarantee of finding specific work based on these two. The rating system is one that is understandably good however not flawless. In the Daemo article we think you explain better than we could and we assign this to their incredible no of users (they boast 3 million). This pushes good and probably professional users to inflate ratings in order to maintain (or strive for) better visibility and changes to find work/workers.

ClickWorker is a developed platform that aims for users to be able to complete their tasks only by clicking. (therefore the name) We tried it as a Worker, however it is not intuitive at all - when compared to others platforms, such as Mechanical Turk or upWork, it feels clunky (takes a while until we realised we need to enter our further details to be able to do a job) and not fully functional - after clicking SEE ALL JOBS, no jobs were shown OR after trying to save one's details (such as Language, Education - which are needed to actually complete jobs), it requires multiple tries to get them saved, often ending with an error message. We liked that the site offers assessments in order to improve the Worker's performance (and unlock further jobs), nonetheless it said no assessments were available at the moment (not even the base ones, which should be accessible by every new Worker registered at the platform). Overall we believe ClickWorker as a platform has a lot to improve, both UX/UI- and functionality-wise.



  • What do you like about the system / what are its strengths?


Just like as in the case of the mTurk, we think this platform tackles economic inequality beautifully however as opposed to it, it actually makes sense because of the limiting technologies. Also the article was probably written in 2011, when the mobile technology was just starting to bloom (or at least capacitive screen technology). Nowadays, perhaps a lot more can be done in terms tackling more complex tasks (not necessarily more complicated, perhaps multi-media wise). The Web-Application focuses on mobile devices. Mobile devices are far more distributed than Desktop-PCs in India. The people of the so called "Bottom of the Pyramid" have a lower education rate and therfore more complex tasks could have an effect on the accuracy so that the focuse on just doing OCR tasks could be a benefit to this platform. The design of the UI is very functional and it has been developed for the most devices of that time.

  • What do you think can be improved about the system?

Today, the mobile OS distribution has changed in India. More than 50% of the devices are based on Android and have bigger screens and faster CPUs (Statista, 2015). This is has an impact on the UI core concept of MobileWorks. Bigger Screens mean e.g. doing more tasks at the same time but also to have additional functions which could support the worker. To offer higher wages the quality of work has to increase (in terms of accuracy and different task models like audio transcriptions etc.).


  • What do you like about the system / what are its strengths?

The primary strengths of Daemo are the Boomerang rating system, which is probably the most real and very-difficult-to-trick raking/reward system and the prototyping of tasks. Both offer a innovative approach to generate trust without social boundaries. Open Source and the core technologies (AngularJS, Python, Node) offer the possibility of a wide source of developers and a big community to maintain and extend the project organically. The 3 studies speak for themselves.

  • What do you think can be improved about the system?

The balance of supply and demand is an indicator for the success of a product. There are different target groups for Daemo in terms of geography who have all two common goals a) doing a task for a good rate and b) let a task done to save resources in terms of money. However the technological differences in the so called third world make it necessary to design the frontend based on their minimum technical requirements (e.g. mobile OS, average screen size, browser distribution etc.). So Daemo could offer a multi-lingual platform with different frontends and interfaces which are based on the technical requirements of the geo targets. Also, one of the main concerns was that very good users (and professionals of course) would sweep everything into one’s net however this is difficult to predict and as long as novice users won’t be completely downgraded with one bad rating the most problematic issue is nicely tackled.

The frontend could offer real time KPIs like current finished tasks vs. open tasks, accuracy rate etc. but also historic KPIs like GEO of workers, fastest worker (paying him a bonus?) etc.

A REST-API coulld offer employers to transfer tasks directly via their own internal systems to integreate Daemo as a standard service within their business processes

Flash Teams

  • What do you like about the system / what are its strengths? What do you think can be improved about the system?

Again, results speak for themselves. The flash teams seemed to require less coordination and managed to finish the work in almost half of the controls which translates into less money required of an employer which could of course translate and fuel a popular platform. However, some success may be credited to an agile-like team build, which I doubt is a coincidence as it would have been the best choice for scalability. One thing I was also intrigued by is the apparent waterfall workflow when tackling the overall project because this allows potential changes at the team level.

Milestone Contributors

Slack usernames of all who helped create this wiki page submission: @seko - Sekandar Matin @amdp - Alessandro Merletti De Palo @ahmednasser - Ahmed Nasser @vlado - Vlad Omete @kamilamananova - Kamila Mananova @jermenkoo - Latal Jaromir