Milestone 1 sanjosespartans

From crowdresearch
Jump to: navigation, search

sanjosespartans' submission for Milestone 1

Experience the life of a Worker on Mechanical Turk

Experience - I have been a worker on Mechanical Turk (mTurk from here) for nearly 5 years.


1. mTurk shares many of the benefits of other crowd sourcing sites that is generally work is available whenever workers need , the requirements are having an internet connection and a decent computer in order to work. mTurk's use of qualifications is commendable. It is difficult to game this system, provided it is set up properly, which protects both turkers and requesters in the end.

2. The site's set-up is quite simple, easily navigated, and has barely changed in the years I have worked on mTurk. I like this because it is preferable to a site that is redesigned frequently, which could slow users down as they learn to navigate it.


1. Overall, the lack of arbitration is a major issue. Amazon uses a very laissez-faire attitude in its dealings, and workers who are wronged, either by requesters or Amazon themselves, have no chance of arbitration, potentially spoiling years of hard work. This also breeds an attitude of uncertainty; if Amazon suspends a worker's account, they may never find out why and could potentially lose their main source of income. This is also a major issue for requesters, as they can experience the same lack of support if something goes wrong.

2. It was discussed on the first G+ Hangout that mTurk is geared towards lots of small tasks rather than a few large tasks. Making money on mTurk ranges from somewhat difficult to very difficult. There is a steep learning curve, and making a liveable wage requires hard work and dedication beyond what is typical of a comparably-paying job. This is, of course, highly individualized; some naturally fare better on mTurk than others depending on skills. I do not believe I can comment on whether or not that is ultimately fair. For example, if I am a very fast typist and am comfortable working with computers, I can do better than someone who is a slow typist and uncomfortable on computers, but I would do worse than someone who has my skills plus other skills (eg, they can code).

3. While the qualification system is important, mTurk's highest qualification, Masters, is a source of discontent for many mTurkers. Theories on how to get Masters have been discussed on forums, with workers posting their account statistics, but other than a high approval rate, there is no common thread among Masters. Personally, it is frustrating to see workers with fewer completed HITs and less time on mTurk receive Masters. There are currently three Masters qualifications: Data Categorization, Photo Moderation, and general Masters (without either of these two specialties). If Amazon expanded the range of qualifications so that more workers could earn a similar qualification for one type of work (eg, one showing that they have excellent accuracy for transcription work), I strongly believe that this would improve worker morale and requester satisfaction. However, this would also require an extra step, because requesters would have to give their tasks a category before posting; and again, especially since Amazon dislikes moderation, this would only create more work for them.


1. mTurk itself is very isolated. There are no areas to chat with other workers. This is in line with Amazon's hands-off policies: a large forum is hard work to police and they could weaken their site, potentially, if turkers there began talking about surveys, codes, etc (although I have, in my experience, not found this to be the case). However, this is also a benefit--workers must join forums to be social and to get help, which leads to strong communities where a worker can complain about a problem they are having without fear that Amazon will see it.

Improvement Areas

1. Better communication platform should be provided with respect to transparency and grievance redressal of the people.

2. Learning curve for the same should be made less steep by publishing online content , books or E-Books for the same.mTurk is not known commonly eve to many IT Professionals. If word about it spread more then there can be input of many ideas that can take place and steps for improvisation can be taken in this area.

3. Transparency among the Master's level should be made and certain policies for promotion to Master's level should be rolled out,so that there is not a feeling of dissatisfaction among those who are not promoted quickly.They must be able to know of their weak areas or other things so that they know what they are lagging and how to upgrade themselves.

In Conclusion

The two features that a new crowd sourcing site needs, in my opinion, are:

1. An arbitration system for workers and requesters.

2. Permanent (provided quality remains high) qualifications for workers based on category.

Experience the life of a Requester on Mechanical Turk

Experience - Although mainly a worker, I have also casually posted some requester tasks.

What did you like?

1. There was little delay between posting my tasks and having them completed.

2. mTurk does offer a very small amount of guidance to requesters, eg, what it would take to make a fair wage.

What did you dislike?

1. There aren't really any resources available to help requesters. My tasks were completed quickly. However, it must be noted that as a worker, I am familiar with the sort of tasks the average worker would do and the sort of tasks they wouldn't. My tasks were clearly labelled and fairly priced. A new requester might not have these benefits and might not know what's fair and what's not.

2. The requester interface is not simple to use and its layout is confusing. The layout seems to be geared partially towards those who are only basically computer literate, and those who are very computer literate. The result is muddled. For example, Masters is set by default in an out-of-the-way panel, so it would be easy to keep the Masters qualification on all tasks without being aware of it.

3. As was noted for turkers, there is no arbitration system. This is a poor choice for a site's longevity.

The CSV file of my results may be found here.[1].

Explore alternative crowd-labor markets

Compare and contrast the crowd-labor market you just explored (TaskRabbit/oDesk/GalaxyZoo) to Mechanical Turk.

Pros and Cons of TaskRabbit :


1.The core philosophy of Task Rabbit is to outsource small jobs and tasks to other people in the vicinity. Examples include like getting daily groceries our launderies picked up from certain place and be delivered to the home,setting up Wifi Modems,getting plumbing service done etc.

2. It works on the concept of Demand and Supply. A person who wants a job done posts the requirements and also specifies the conditions of budget.On the basis of this it goes on to participants based on the location as preferance.Bidding is done and the best match gets the task done.A small cut is taken by the Website which is a nominal fee.

3. It is helping in promoting the unemployed or individuals who are willing to make money being mostly around their homes to get short jobs and earn livelihood from it.Professional services are not taken here,which in a way helps at reducing the idle time of many people residing in a community and the task can be done in relatively less time by the vicinity workers than experienced professionals coming from distant places.Plus the labor charges and other factors are also less using this medium.

4. There is also an interview and a training that TaskRabbit gives to the people it hires to complete the task posted,plus background check and other things are also done to ensure safety.


1. It although offers help to individuals of a local community in getting jobs but also proves to be harmful for established business that are operating in the area as it reduces the labor rates and also affects their business economy.

2. Not all types of jobs can be outsourced to general people,there is also some level of professionalism and edge required to complete a task that sometimes only a skilled professional can do and layman can't do it.

3. Even though sufficient background check has been done but still cases of forging such type of data may arise and the workers can have ulterior motives and can be burglars,kidnappers etc also.

Pros and Cons of oDesk:

Pros :

1. It takes us to the next level for making money without having to worry about where to find clients. If the job is done then both employer and freelancer get to rate each other and this helps in maintaining a level of good feedback mechanism used for rating which is transparent.

2. It emphasizes on work life balance.It is a home based system,we can make our own schedules.Start working on a project as per our own convenience and there is a feeling of responsibility and ownership of the tasks unlike other companies where we work full time and still don't get work according to our qualifications.

3. The freelancing section includes Web,Mobile,Designing,Writing ,Accountants and Consultants also.It spans a plethora of services that are offered by it and people having defined skill set can work as free lancers and get paid for jobs that they like and according to their comfort level.There is no pressure of working 24*7 every time,unlike in some companies where you need to slog every time no matter what the situation be.

4. There are many companies tied up with oDesk and they can hire freelancing individuals from there in record time and save money and other resources , some of the daunting tasks that the companies face are made easy to resource and manage by the website,thereby appearing to be a boon for the industry.


1. Most of the jobs present in the site are low paying and lot of effort is needed as mTurk to get a decent paying job.

2. The main factor driving down the pay rates is availability of many freelancers,so in this case competition proves to be a curse for the people themselves as some are ready to work even at low rates,just to land up with an offer.

3. Sometimes the parties on either side may be lazy,have ulterior motives and they may not provide valid feedback which is the driving force behind this system and such type of factors reduce the trust and in the end affects both parties.

Pros and Cons of GalaxyZoo:

Pros :

1. It is recently launched astronomical Website that allows the public contribute to the project that aims to help astronomers understand the large scale structure of the universe.It has adopted the model of SETI@home in having volunteers analyze huge amount of data in search for extra terrestrial intelligence. Here the participants are called upon to help classify a million or so galaxies on images which are taken by Sloan Digital Sky surface.

2. The need is that each participant must have a good eye and a willingness to learn things. The participants are first subjected to simple and straight tutorial and then they have to pass a quiz,after which they are required to use the skills they have developed i identifying pinwheel shaped spiral galaxies and whether there rotation is clockwise or anti-clockwise.

3. Main reason behind making this was the presence of millions of odd galaxies inthe universe made the task of professional astronomers daunting and they had to call in public help in this task ,so as to reduce the burden on them and urges the users to make their best guesses in identifying the galaxies.


1. Although images that are displayed in the tutorial are fairly easy to classify ,many of the pictures that are provided are not distinct and a person faces difficulty in correctly identifying them.

2. For a professional astronomer its easy to classify whether a galaxy s spiral or elliptical given his plethora of knowledge but its very difficult for a layman given any set of pictures as a reference and then classify what type of galaxy he/she is viewing.

Comparison with mTurk:

1. TaskRabbit if compared to mTurk offers wider range of facilities as its main aim is in service industry and provide work to the individuals of the community where the need for completion of the task has originated from. mTurk aims at earning money online by answering some questions,finding out discrepancies in the data that has been collected previously,reporting some issues.It is not an onfield work.

2. oDesk is much broader than TaskRabbit and eventually mTurk as it has a wide range of services to be offered.Essentially it is a freelancing community where professionals/experienced people in their relative field have some free time or have a different mode of operation which is more concentrated as being part time or work from home and caters to their individual preferences. The process is completing the task which is offered by a company in stipulated time frame and then move on with the next job.The compensation offered here is more but the expertise level required is also more than average.

3. GalaxyZoo is a website aimed for helping out the astronomical community with the tiresome task of identifying various galaxies and have common people help them in this tough task of identification on voluntary basis.



1. What do you like about the system / what are its strengths?


1. Easy accessibility, almost everyone has a mobile phone which makes the plan more reachable. Internet is also becoming easily accessible at affordable price rapidly and hence MobileWorks can reach a larger target crowd.

2. The platform is suitable for OCR tasks.

3. The Historical Accuracy and Quality concepts keep the workers motivated to maintain or attain certain standard in terms of quality of the output.

2. What do you think can be improved about the system?


1. Currently the tasks that the workers can serve are of a very singular type. More diverse tasks can be added to provide greater range of tasks to choose and work upon.

2. User experience can be adapted depending on the type of the phone (smart phone/ non smart phone) . The technology can aim to make the tasks more relevant and personalized to a particular worker.

3. The technology can easily be made available in local language using either translation algorithms/ English transliteration platforms. Enabling audio content can increase the platform accessibility so that even blind or less abled people can use it.

4. Task prioritization should be done to ensure the critical tasks are completed and the workers gain the most from the available workload.


1. What do you like about the system / what are its strengths?


1. Partnering with local business to overcome the hurdle of auto payment is a good way to tackle auto payment crisis. Although this might not ensure an ideal money flow but in areas where the required technology is yet not available it will allow the workers to get paid on a periodic basis.

2. Using mobile phone as a platform for crowd sourcing. Almost everyone in India have a mobile phone these days so mClerk can reach out to more workers.

3. The user diffusion network described looks very effective. I have encountered similar plans have worked well in India.

2. What do you think can be improved about the system?


1. There is a huge restriction on the number of SMS that can be sent in developing countries like India. This will be a hindrance in the number of tasks that can be served.

2. Very limited number of tasks can be served via SMS. Also limits on SMS text will not allow the system to perform well in certain scenarios. The user experience will not be very good.

3. In recent times with increasing Internet accessibility, methods of auto payment can be set up. The incentive to earn money via Internet will motivate people to get Internet plans and then the money can be distributed. Also the platform can be redesigned to reach workers via Internet for better user experience.

4. The accuracy rate can be improved by having an algorithm/ logic which would validate answer only if certain number of people give the same answer. Its crucial for a crowd sourcing platform that a standard quality of work is maintained.

Flash Teams

1. What do you like about the system / what are its strengths?


1. It is suitable for complex tasks like software development, prototype development.

2. Has the potential to bring together best workforce over the world to solve the task in hand.

3. The elasticity ensures that Flash is able to scale resources depending on the demand.

4. Dividing actions in blocks allows development to proceed in independent modules which enhances re-usability and design.

5. It’s a platform where paper napkin design can become tangible with the help of collaborative expertise.

2. What do you think can be improved about the system?


1. Peer review mechanism can be set up to ensure healthy competition and quality.

2. Extra incentives can be given on completion of certain milestones to ensure that workers are motivated to complete the task on time.