Milestone 2 Akali

From crowdresearch
Jump to: navigation, search

Welcome Folks!!

This is Team Akali, with our 2nd Milestone submission.

Attend a Panel to Hear from Workers and Requesters

Observations from Panel Meetings

  • Main motive of people joining AMT solely for seeking extra money that they can earn during their free hours. Most of the Turkers also consider AMT as a full time job.
  • Beginners in AMT usually have to face a lot of trouble in figuring out the website, since there are no tutorials for startup and the interface is also too complex to understand.
  • Forums help the workers and requesters a lot as it provides a means of communication between them and also to find the views of both demographics.
  • It isn't easy to find work on AMT. So Turkers use the following techniques to find hits,
    • Workers use tools like scripts and scrapers to find them hits which cater their requirements.
    • Workers experiment with the work hours for getting more hits. Searching for HITs during unconventional hours can fetch more work.
    • Turkers use a tool called Turkopticon which helps them to watch out each other. It shows the review of the requesters which help the workers in finding hits form a generous requester.
  • AMT is a bit tricky for requesters to use as it requires the proper explanation about the task they want to get done. But a vaguely instructed hit can also be explained using Forums.
  • Community of Turkers is strongly bonded to each other by the forums, where they help each other to figure out the hits and also help each other to find work.
  • Many Requesters reject the works and this is the cause of discouragement among workers, which takes a lot of time to deal with.
  • Setting up qualifications like Masters cost more than the average, but sometimes this also does not provide the high quality result as promised by AMT.
  • It is always difficult for requesters to filter the given hits, as workers give a lot of fake information on surveys.
  • Requesters not always get the work done from the demographic they need. Like Requester targets a group of people from a particular region, but also get the works from workers of different region.

Reading Others' Insights

Worker perspective: Being a Turker

Observations about Workers

1. It motivates better cheaper and faster work performance, to get a good idea from workers to improve the performance. The primary reason for turking is to earn money. There has been much less qualitative work into what it means to be a Turker

2. It is been contrasting on the point that the collaboration between requester and worker on the basis of information should be proper. Which was a result in developing of fairer relationship based crowdsourcing. This kind of system helps in rating requesters.

3. Workers have been worrying about the interference of other sections such as government which may affect their jobs. Basically the workers do the hits for earning the wages no matter how worst the work be.

4. Workers mainly look out for a union of workers who can easily identify the unauthorized requesters so that they can expand their duties.

5. Turking is perceived as a free labor; workers calculate their effort and pay of a hit. They share their comments whether they think the pay is fair or not.

Observations about Requesters

1. Amazon is still a big Requester; AMT has been deployed as a platform and connects a wide variety of Requesters with up to 500,000 Providers. Research has been primarily concerned with the employers perspective has had far reaching consequences.

2. On joining the forum, members have limited access to various areas and threads. With increasing activity, members can increase access rights based on their usefulness to the community. The majority of the public areas are devoted to issues directly related to being a Turker and doing AMT HITs.

3. Turkers on TurkerNation, the primary concern is to find good Requesters and avoid bad ones. Requester is bad because of their ‘mass rejection’ and this is aggravated by their ‘demeaning’ comments.

4. Turkers are understandably offended when Requesters reject HIT submissions for reasons they do not understand. This not only deprives them of money they believe they have rightly earned, but it has a damaging effect on their approval rating.

5. Objectivity is not only fair towards Requesters, but Turkers also cite the benefit to the community. If Requesters are ‘unfairly/blackballed’ it is not only damaging to them, but may put off other Turkers from working for decent Requesters. As such, SeerKRap proposes it would be beneficial to inform the community that they have been hastily unfair.

6. Practically, better understanding and engagement with Turkers can help design better HITs, and technologies to support Turkers and their relationships with requesters. Important avenue of design is tools to help reduce and manage all the invisible background ‘work to make Turking work’ since that is work for which Turkers are not paid but takes up a considerable amount of time.

Worker perspective: Turkopticon

Observations about Workers

1. Turkopticon, an activist system that allows workers to publicize and evaluate their relationships with employers. Building and maintaining the system, as well as communicating about the system with workers, has offered a distinct vantage point into the social processes of designing interventions into large-scale, real world systems.

2. Turkopticon is a browser extension for Firefox and Chrome that augments workers, view of their AMT HIT lists with information other workers have provided about employers

3. Turkopticon grew out of a tactical media art project intended to raise questions about the ethics of human computation. it as micro labor marketplace draws attention to pricing mechanisms.

4. Crowdsourcing platform draws attention to the dynamics of mass collaboration among workers, the aggregation of inputs, and the evaluation of the crowdsourced outputs.

5. Framing workers on AMT as computational services is more than just rhetorical flourish. Through AMT, employers can literally access workers through APIs. Though a web form-based interface is available, the API allows AMT employers can put tasks into the workforce and integrate worker output directly into their algorithms.

6. AMT brings together crowds of workers as a form of infrastructure, rendering employees into reliable sources of computation. As established organizations develop and publicly release tools for the system, they embed computational microwork firmly in existing technological practices and systems.

7. Turkopticon attempts to prevent employers from retaliating against workers writing reviews by obfuscating workers’ email addresses.

Observations about Requesters

1. Several requesters have noted that a thousand to one worker-to-requester ratio makes responding cost prohibitive.

2. Turkopticon is nothing without users and their reviews. Like many CSCW systems, it requires a “critical mass” to serve users at all. AMT model often has workers doing HITs from a large number of employers in a session; they needed to offer workers a quick way to assess employers.

3. Priority with prominent requesters as well as a source close to Amazon who wished to remain anonymous collects money for task volume; Amazon has little reason to prioritize worker needs in a market with a labor surplus

4. In the logic of massive crowd collaborations, dispute resolution does not scale. Turkopticon has been designed to offer workers a way to dissent, holding requesters accountable and offering one another mutual aid.

5. Automated approaches are difficult to implement in practice because they cannot account for community-specific and emergent norms. Within the space of social moderation, broad-based community moderation is susceptible to vandalism because our users are potentially from two competing classes with opposed incentives.

6. Turkopticon has succeeded in attracting a growing base of users that sustain it as a platform for an information-sharing community. In part because of its practical embeddedness, it has drawn sustained attention to ethical questions in crowdsourcing over the course of its operation. This attention comes not only in the crowdsourcing community, but also in broader public forum.

Requester perspective: Crowdsourcing User Studies with Mechanical Turk

Observations about Workers

1. The workers prioritize the tasks if the standards of the tasks are specified, this leads the requesters in high paying of the worker and the task is completed in less time without compromising on the quality.

2. There are many technically skilled workers who are unable of finding jobs offering a considerable salary, hence the use of the online market will help in motivating the workers and boosting their skills.

3. Workers benefit from standardizing of crowd sourcing as they get paid more for a task if no one is taking up on them from a long time.

4. With the help of standardizing the tasks and selling them in the market as any other stock or commodity it will attract many new workers who are un aware of this crowdsourcing platforms and thus increasing the number of workers for that particular crowdsourcing platform.

Observations about Requesters

1. The requester of a particular task is allowed to provide feedback to its workers in order to get more precise and accurate answers for his/her requested work.

2. Requesters have the ability to set the number of workers that can perform their particular task; they can even select a worker (from the pool of workers interested in doing that task) for doing their task.

3. Requesters can also decide which answer should be validated and paid this is quite important for avoiding irrelevant answers and this can save their money.

4. Workers belonging from various parts of the world is actually a disadvantage as well because some Turkers must be un aware of the task which includes the demographic information, this leads to improper answers due to lack of sufficient knowledge.

5. A requester cannot check on the generality of each and every worker, this at some point of time will lead to some problems.

6. Requester must be very careful in designing of their task and also in setting the amount which is to be paid to the worker after the completion of the task.

Requester perspective: The Need for Standardization in Crowdsourcing

Observations about Workers

1. As we very well know about the interrelated problems standardization of crowd sourcing will be beneficial to workers as this will give them more work opportunity.

2. The workers if being trained on tasks could complete the tasks very easily and then can jump over to other tasks, this will be helpful for the requester as his/her job will be completed within the deadline.

3. With the help of standardizing the tasks and selling them in the market as any other stock or commodity it will attract many new workers who are unaware of this crowdsourcing platforms and thus increasing the number of workers for that particular crowdsourcing platform.

4. The workers can easily know that which task should be done first with the help of the queue which reveals the amount of time a task will stay in the system.

5. The financially poor workers gain access to interact with the significantly rich buyers (work buyers) in various countries.

Observations about Requesters

1. The online labor markets currently work in a very unsystematic order, by standardizing the crowdsourcing the requesters will be limited to certain boundaries in generating their tasks, this will assure that no worker will be paid less nor the other will be paid more.

2. Because of standardization, the requesters now will be able to concentrate more on their task rather than on building appealing user interfaces.

3. In order to complete their work first among the others, a requester can pay more than the current price so that the workers give higher priority to their tasks over the other.

4. The requesters mainly should focus on creating such tasks that are best applicable in the environment and the worker working on it should be benefitted by doing the task.

5. Not only the workers but the requesters also can interact with highly qualified workers who can do their task with in less time and within the amount that is being to them.

6. They not only help in doing the assigned task but they are capable enough of creating new creative products beneficial to everyone.

7. Thus creation of new job will invite more new workers and the system gets expanded.

Both perspectives: A Plea to Amazon: Fix Mechanical Turk

Observations about Workers

1.Worker's need trustworthiness guarantee for the requester, they should not be afraid of working for a particular requester.

2. It should be ensured that workers are getting paid quickly and also care should be taken to avoid unfair rejections

3. Volume of the posted work and the lifetime of the workers should be recorded, workers can use this info in choosing a requester.

4. Interface should be developed in such a way that the workers can interact with the requesters efficiently

5. Apart from the interfaces, Systems, search engines should be improved. Using which worker can reduce their overhead work and can work efficiently.

Observations about Requesters

1. Requesters need a platform to make it easier to post tasks and reduce the complexity in posting the work.

2. Using of tools like Iframes should be encouraged instead of workers building their own interface from scratch.

3. New methods like public qualification tests, rating of workers, maintaining worker's records should be introduced.

4. Rating should be done to workers alone but not requesters. As, Requesters need no qualifications to give work.

5. Payment should be done according to the caliber of the worker. This way, good workers can get a payment according to their caliber.

Do Needfinding by Browsing MTurk-related forums, blogs, Reddit, etc

1. Worker needs to set goals for themselves so that they can actually attain what they want. Here is an example of a worker who is a beginner and wants help in setting his goal.

2. A Turker who wants to use automated search for Hits needs a set of scripts to start on with. In MTurk Forum we have found a great article article for How to Script for Newbies.

3. Workers need Qualification for getting good hits. This post on MTurk Grind can help them out in attaining qualifications on AMT

4. Workers need to be rewarded (not rejected) for the task they have conscientiously done. This study is about how dedicated wokeres are

5. Requesters need to have some sort of basic knowledge about creating a meaningful hit. Here is the wiki page of WeAreDynamo, where they give out basics of how to be a good requester.

6. Need for Security on such Crowd Sourced Websites. Here is a question on Reddit about the safety on AMT.

7. Requesters need to know about the demographics. This post about the Demographics on Mechanical Turk can actually be helpful.

8. Turkers need a well-defined UI so that they can search and wok according to their preferences. This post points out the defects in the search engine of AMT

Synthesize the Needs You Found

Worker Needs

  • Workers need to be rewarded (not rejected) for the task they have conscientiously done.
    • Evidence: This guy(Who is a newbie) have done a Hit, but got rejected because in the survey asked by requester, there was missing measure to be added.
    • Interpretation: It is actually the fault of Requester, he/she could have closed the Hit and could have posted another Hit by updating the previous one. These kind of acts demoralizes the worker.
  • Workers need Qualification for getting good Hits
    • Evidence: Here is a post where a person who dose not have any knowledge about qualifications, wants to know about it.
    • Interpretation: Since AMT doesn't provide any kind of assistance to the newcomers, people like the above guy finds it hard to understand the basis of the website. Qualifications are actually not well portrayed by AMT and for seeking assistance, workers should visit forums.

Requester Needs

  • Requesters need to have some sort of basic knowledge about creating a meaningful Hit.
    • Evidence: Here is a post about How a Requester didn't know how to set amount of reward for his Hit, which caused inconvenience to a worker and requetser also rejected his work.
    • Interpretation: The requester was not aware of the settings in AMT, and the interface also doesn't come to any help. This can lead to a bad reputation of the Requester, as the worker said that he will stay away from the requester because of this.