Milestone 2 researchinprogress

From crowdresearch
Jump to: navigation, search

Attend a Panel to Hear from Workers and Requesters

Deliverables

Observations: The morning session had requesters and workers on MTurk who talked about their experiences in their roles and the problems they faced. They also talked about how they dealt with those problems. Workers felt that it got difficult to find suitable jobs initially. It was observed that workers communicated with each other in communities to help each other and solve worker related issues. Workers used scripts to know which hits are newly arrived or which ones would be suitable for them. Requesters used time constraints for the job. They would sometimes pay higher amount to get jobs done fast. AMT restricted usage to US citizens only and implemented strict rules to implement that.

Interpretations: It gets difficult to get good jobs initially. This is because of lack of recommendation of good jobs for a new worker and it also seems that the user interface of AMT is not very quick and easy to use/learn. This appeared to be one of the biggest hurdles for the new workers. Workers form communities which helped them raise a voice, show power in case they were not being paid rightly for the work. Running scripts to get new hits help workers stay updated about the new jobs coming in and helps them get a greater chance to get jobs suitable for them. Setting up of time constraints helps the requesters get the job done faster. However, paying higher for some jobs might result in only attracting money-minded workers and thus quality work may not be guaranteed. Some requesters feel the US citizen restriction on AMT has increased quality of task submissions.

Needs: One who is interested in crowdsourcing would want to know the pros and cons of it and truly be aware of all the risks involved when one is planning to further explore this. The panel discussed about how different workers need the right to be able to question when they are rejected. There were solutions like mailing the concerned authorities. There were also other needs of the workers such as the right to know the question properly when it's not correctly given. Some worker's also felt the need to get paid higher in certain jobs. Requester's needs were to get good quality work and to ensure authenticity in the work. It was discussed how they met this need by setting time limits and keeping a check on the ip addresses. Requester's needs to get certain work done quickly was satisfied by enterprising specific tasks. However it was difficult to satisfy the need of getting work done by specific workers or get a guarantee of work completion by a certain time.

Reading Others' Insights

Worker perspective: Being a Turker

Mainly US based, but a growing percentage seen of Indian and other nationalities in recent times and belong to lower income strata. But recently they have changed the account settings for the workers, and only those workers who are above 18 years of age and are residents of US can work on mTurk.

If we look very closely and try to analyse then we will conclude that the main reason for workers to work on AMT apart from entertainment, past time is MONEY. Workers at times paid less compared to amount of effort that was put in by them.

The work as a Turker may be a part-timer for some while for some others it might a full-time job because of various job issues that they may be facing. The earnings per year can range from $50 to $15k.

Worker perspective: Turkopticon

Might get cheated by employers, frauds who do not pay them after the work is done. Requesters are not rated, only Workers are, this may lead Workers being exploited. Turkopticon gives the Workers a forum where they can view the ratings of Requesters at a larger scale and thus can be saved from exploitation since the ones who have acted fraud in past would be at a lower rating than others.

The HITs are dependent on answer that the majority gives, not on the individual opinion of a Turker. The answer that is approved correct because majority answered with that may be right, may be wrong. Thus, a worker might get rejected even after he/she has answered correctly.

Workers generally get rejected and create a view that unfair means have been used and feel that amazon on their behalf are acting irresponsible towards the workers.

Requester perspective: Crowdsourcing User Studies with Mechanical Turk

Requesters might be misled about their request for the workers, if majority answers incorrectly. The incorrect answer is then considered to be the correct one, and the workers that answer correctly might be rejected and the ones answering incorrectly might get a HIT.

With the introduction of Turkopticon, requesters have to be more generous, communicative, fair and prompt towards their workers; otherwise they might be down rated by the workers which would lead to a great loss to their work, since in future not many workers would take up their work.

Many rapid prototyping tasks and surveys can be completed with the help of mechanical turk. It can be easily compared to a micro-market where the labour is equivalent to the workers and requesters are the demanders in the market.

Requester perspective: The Need for Standardization in Crowdsourcing

Requester has to pay a minimal amount to get his/her task done in a given interval of time with a good probability that task will be delivered efficiently.

Requesters have to take up the sole responsibility of judging whether the task delivered buy the worker is acceptable or not. If not then he/she has to be accountable to the worker about their work.

Both perspectives: A Plea to Amazon: Fix Mechanical Turk

Workers might not get exposure to all the tasks but are made to view filtered tasks by priority queuing. This is also a problem because if a worker wants the work of a particular requester it is difficult to find them unless he/she is aware of the requester’s username.

It is difficult for the requesters to post tasks on a non-user-friendly interface, which uses command line. It is tedious for requesters to break down the work into tiny groups and divide it into suitable tasks post them, then reassemble their work and so on, this can only be done by serious requesters, thus small tasks might not be given that much importance.

It gets highly challenging for the requesters to distinguish between the true workers who would actually work whole-heartedly towards the task and the ones who are just doing for the sake of it. They require a better understanding of the workers and hence a better mechanism to view the profile of a worker who has applied for the task.

Do Needfinding by Browsing MTurk-related forums, blogs, Reddit, etc

It appears that people have varied queries and doubts starting from questions as basic as how much does a HIT cost. []http://rh.reddit.com/r/mturk/search?q=flair%3ARequester%2BHelp&sort=top&restrict_sr=on&t=all[] As it was discussed in the panel too, there isn't much help available as such and thus one relies on various blogs and sites to seek help to get used to sites like AMT.

While it appears that some follow the system seriously/religiously, some people also seem to be posting questions/queries out of helplesness: []http://www.reddit.com/r/mturk[]. There isn't much which can be done about slow payment procedure. It would be more advisable to mail the concerned authorities.

While someone asks questions about 'When crowdsourcing used to be cool' []http://www.reddit.com/r/mturk[], it appears there are people who are dissapointed with the 'evolving' changes too.

The first rejection disappointments are shared and celebrated together by the people.[]http://www.reddit.com/r/mturk[]

There appear to be serious discussions where people talk about new developments which should take place. For example, mobile apps for workers: []http://www.reddit.com/r/mturk/comments/2wkrdu/what_would_your_ideal_crowd_work_platform_be_like/[]

It was also quite evident at various places that people do struggle in finding relevant and suitable jobs easily, specially when they start working. []http://www.reddit.com/r/mturk/comments/2wkrdu/what_would_your_ideal_crowd_work_platform_be_like/[]

Synthesize the Needs You Found

Worker Needs

Needs: Workers need an easy way to search jobs suitable for them and easy usage of the website. Evidence: This was discussed in detail in the panel discussion, and we also found it here: []http://www.reddit.com/r/mturk/comments/2wkrdu/what_would_your_ideal_crowd_work_platform_be_like/[] as an issue. Interpretation: A good way to get suitable jobs saves time, increases efficiency and leads to better performance of the whole system.

Needs: Workers need methods to search HITs which match the skills they have. Evidence: []http://www.reddit.com/r/mturk/comments/2wkrdu/what_would_your_ideal_crowd_work_platform_be_like/[] Interpretation: If worker's are able to find tasks according to their skills, then they will be able to perform better in it as they know what they are doing in more detail.

Needs: Workers need portals to easily communicate with the requesters. Evidence: []http://www.reddit.com/r/mturk/comments/2wkrdu/what_would_your_ideal_crowd_work_platform_be_like/[] Interpretation: Workers sometimes have questions as to why their task was rejected or maybe to request an increase in the amount. There doesn't seem to be an easy way to get answers to the issues of the workers because of which they have to rely on their communities and the mTurk authorities for the 'wrong' done with them by the requester.

Requester Needs

Needs: Requesters might want to pay to workers who have not completed the task within the time limit. Evidence: []http://rh.reddit.com/r/mturk/search?q=flair%3ARequester%2BHelp&sort=top&restrict_sr=on&t=all[] Interpretation: There is no way to pay the workers who complete task out of the time limit. There might be a possibility that good quality work is obtained with some little extra time. But credits for that cannot be given.

Needs: Requesters need a method to increase/change payment for a HIT later. Evidence: []http://www.reddit.com/r/mturk/comments/2wkrdu/what_would_your_ideal_crowd_work_platform_be_like/[] Interpretation: The requester might realize that his job is much in demand or not as much as he expected and change the HIT money accordingly. However, it appears that one needs to delete and put the HIT again which will make the job even lesser known once it is deleted previously.