Difference between revisions of "Milestone 1 pixelpals"

From crowdresearch
Jump to: navigation, search
(Experience the life of a Requester on Mechanical Turk)
(Experience the life of a Requester on Mechanical Turk)
Line 21: Line 21:
 
*Minimum no of workers-15
 
*Minimum no of workers-15
 
Challenges:
 
Challenges:
Designing simple and intuitive survey was a big challenge. So therefore , first we had built a sample survey of CAPTCHA and which we circulated around our peers and then we took feedback to understand  the level of difficulty of questions on survey.Out of 37 responeses, 62% found survey easy, 28% found it moderate and 10% found it difficult to solve.On the basis of feedback , we again designed the survey so as to make it intuitive and simple.
+
Designing simple and intuitive survey was a big challenge. So therefore , first we had built a sample survey of CAPTCHA which we had circulated around our peers and then we took feedback from them to understand  the level of difficulty of questions on survey.Out of 37 responeses, 62% found survey easy, 28% found it moderate and 10% found it difficult to solve.On the basis of feedback , we again designed the survey so as to make it intuitive and simple.
 
Then we uploaded the task as a Campaign (requester) on Microworker.Microworker took approximately 2 days to review and accept it.
 
Then we uploaded the task as a Campaign (requester) on Microworker.Microworker took approximately 2 days to review and accept it.
 
Finally we received 15 pass hits and 2 fail hits within 1 and half day.
 
Finally we received 15 pass hits and 2 fail hits within 1 and half day.

Revision as of 23:23, 4 March 2015

Template for your submission for Milestone 1. Do not edit this directly - instead, make a new page at Milestone 1 YourTeamName or whatever your team name is, and copy this template over. You can view the source of this page by clicking the Edit button at the top-right of this page, or by clicking here.

Experience the life of a Worker on Mechanical Turk

Our experience as a crowd worker was new and intriguing. We replaced Amazon Mechanical Turk with Microworkers since we were not able to make a verified account on Amazon mTurk. The skill set required for working on jobs on Microworker is minimal as the tasks were very rudimentary. Some of the jobs included searching a particular product on Amazon and taking screenshot; installing and reviewing a mobile App; commenting on blogs, Youtube etc. There was no restriction on the time taken to complete a job and for each job an approximate time to complete the job was mentioned. The description of the job and what was required as the proof of job completion was clearly stated. This helped us greatly in getting a clear idea of the project before we accepted it. Completing light weighted and diverse tasks was easy and fun. We liked the minimal user-interface that was easy to learn so we could start the tasks quickly. For each available task they mentioned ‘Time to Rate’ which is the maximum time within which the assigned task will be verified. Payment was made into the account after the task was reviewed. We liked that there were no delays in reviewing the tasks we did. The amount of money paid was very meagre but considering the skill set required for the tasks it was justifiable. The site also keeps a track of the success rate of the worker. This would motivate workers like us to keep a tab on their submission quality. However, we felt that the platform could have been more interesting, and challenging if requesters posted some tasks which required some specific skills. For instance, designing a website or writing blogs/articles. This would give us more value for our time (in terms of the payment) Also, sometimes, the instructions and wants of a job were unclear until we did the actual job. So, a sample task would help workers in understanding the deliverable proofs better and not lose out on accuracy because of unclear instructions (such as on CrowdFlower). The number and variety of jobs at any given time was limited. We didn’t have many options to diversify our jobs. There was no classification of workers on the basis of their experience on the platform (such as on CrowdFlower). We as new workers had the same job availability as any other. This further highlighted the absence of skilled jobs on the platform.

Experience the life of a Requester on Mechanical Turk

As a requester, we tried to focus first on choosing appropriate crowd-sourcing platforms.Initially Amazon Mechanical Turk declined out login credentials as it only accepted US citizen to sign-up.So therefore we explored various non-AMT platforms like crowdflower,click-worker etc and finally we landed at Microworker crowd sourcing platform.We first explored various available tasks/jobs on microworker and then decided the tasks

  • Tasks- Complete the suvey based on understanding the usability of CAPTCHA.
  • Total Questions- 7
  • Time to complete - 3 Min
  • Reward assignment- $0.01
  • Minimum no of workers-15

Challenges: Designing simple and intuitive survey was a big challenge. So therefore , first we had built a sample survey of CAPTCHA which we had circulated around our peers and then we took feedback from them to understand the level of difficulty of questions on survey.Out of 37 responeses, 62% found survey easy, 28% found it moderate and 10% found it difficult to solve.On the basis of feedback , we again designed the survey so as to make it intuitive and simple. Then we uploaded the task as a Campaign (requester) on Microworker.Microworker took approximately 2 days to review and accept it. Finally we received 15 pass hits and 2 fail hits within 1 and half day.

Explore alternative crowd-labor markets

Galaxy Zoo

Brief: GalaxyZoo is a platform for classifying galaxies that have been imaged according to their shape and other features. Due to the accuracy shortcomings of computer programs to recognise these patterns, human recognition was used. But classification of millions of galaxy images made it a perfect crowd market problem.

Comparison with Microworker: On Microworker, a lot of the tasks are based on data collection, such as, collection names of authors of particular articles. GalaxyZoo also collects data to create a database of classified images that can be later used as a ready and structured source by professional astronomers or scientists. Both platforms have tasks that do not require any specialised skill set and have detailed instructions for the worker to follow. While micro worker allows tasks of varied nature (such as searching particular information on sites, commenting on YouTube/Twitter), GalaxyZoo is solely dedicated to classification of galaxy images. Thus, the final use of worker contributions will be varied according to requester in Microworker but focused on research in astronomy in GalaxyZoo. A big difference we noticed was the no remuneration for the tasks in GalaxyZoo. The only incentives to a worker were his/her willingness to contribute to research in this field or keen interest in astronomy. While in Microworker, even tasks finished within 3 min had a payment of $0.1. For an actual crowd labour worker, Microworker provides more opportunities, varied problems and better payment. Since workers are paid, there is a system in place for verification of task done (through different kinds of proof mentioned in problem) before the payment is made. Galaxy Zoo needs no such system-it a simply for data collection. To ensure accuracy data collected Microworker sets a benchmark for accuracy that worker must meet in order to continue with more tasks. This accuracy is decided by the requester’s/employer’s review. In GalaxyZoo to ensure quality of results, classification is decided by comparing answers of multiple contributors for the same problem.

Readings

MobileWorks

  • What do you like about the system / what are its strengths?
  • What do you think can be improved about the system?

mClerk

  • I basically like that it gives less skill full people a chance to earn from crowd sourcing. This is beneficial for developing regions. The technologies and simplicity are combined in a nice way for e.g. the sms sending graphical and digital messages which makes it highly usable. They have explained the different factors in their experiment with mclerk. The biggest advantage certainly is that it is a cheaper option for digitising local languages. It looked highly scalable with many persons joining and referring others.
  • The one thing to improve may be is mode of payments. Also not everyone may be knowing English (even broken ) so the way they accept input seems difficult for someone having no idea of the language. The accuracy rate of 90.1% as they themselves have said is quite less as compared to market standards. Too much influence of mobile networks on result, They give example of Airtel dropping messages which halt their progress for 3 days. During peak hours the result might also be delayed. Unequal weightages can be given to different users for increasing accuracy level.

Flash Teams

  • What do you like about the system / what are its strengths?
  • What do you think can be improved about the system?