Milestone 2 singularity

From crowdresearch
Revision as of 21:02, 11 March 2015 by Rishibaijal (Talk | contribs) (Deliverable)

Jump to: navigation, search

Attend a Panel to Hear from Workers and Requesters

Deliverable

Topic. Sources of finding work

Observations

(drawn from "SpamGirl") 1. Primarily through massive forums which are dedicated to sharing opportunities. 2. Daily hits thread which let "altrustic" people share hits and opportunities. 3. Scripts that can automate the task. 4. Websites and tools such as i. Turkalert ii. Crowdworkers iii. Hitscraper iv. Turkmaster 5. Chatrooms dedicated to this purpose 6. Be a part of the community and get involved with the people in it. 7. Give back to the community by helping others out.

(drawn from Manish): 1. Usage of the above mentioned sources varies from the seriousness of the worker. One could rely on these platforms for serious expenses to casual ones.

Topic: Appealing characteristics of hits and jobs. The factors that a worker looks for when searching for hits.

Observations:

(drawn from Manish) 1. Workers often use tools and scripts that can filter jobs and tasks based on the criteria set by the worker. The quality of jobs can be roughly estimated using these tools. 2. For example, a worker might use a tool to set a criteria to filter out all jobs that pay less than a certain amount.

Topic: Motivations for workers or requesters to use a crowdsourcing platform and stick to it or abandon it.

Observations:

(drawn from "SpamGirl") 1. The primary reason seemed to be money. The financial aspect appears to be the biggest factor that draws people into these platforms. Examples for this might include people who learnt about these platforms where one could make money through advertisements, friends, relatives etc. 2. Money is the biggest factor when it comes to "starting" with these platforms. 3. The motivations to go BEYOND that and help others out stem from the following factors: i. Altruism ii. A sense of belonging to the community iii. Forums, blogs, chatrooms often emulate watercooler talk where different members come together to empathize and help each other out. iv. Helps deal with the tediousness and frustration that can emerge when having a bad day.


Topic: What one can do to collaborate with other mturkers and requesters. What sort of interaction occurs between turkers.

Observations:

(drawn from David) 1. By getting involved with the community. 2. Using scripts, tools and other websites.

(drawn from "Spamgirl") 1. Interaction with the community ranges from helping others out to talking to other workers about opportunities, discussions about quality of work, type of requesters, possibilities of collaboration and general chats. The community thrives on forums, facebook pages, blogs, subreddits where anything could be a matter of discussion or conversation. Some examples of things that are usually discussed include: i. What tools are good. ii. How to be more efficient. iii.Which requesters to avoid. iv. Where to make money 2. Interactions with the requesters happens primarily through emails although they are often invited to join the community too. Workers often help requesters with setting jobs appropriately by providing suggestions such as possible mistakes, improvements in the pay structure, setting the appropriate criteria or skill levels based on the jobs being assigned. Requesters can also be asked for suggestions on how to go about a task or improve as a worker on their tasks. 3. Workers may also reach out to requesters if they believe that their work has been unfairly rejected (inadvertently or not). 4. These interactions are a lot like the ones that can happen on any other forum dedicated to a certain thing.

Topic: Comparison of different platforms such as oDesk, mTurk etc

Observations:

(drawn from David) 1. oDesk seemed to have a lot of competition on it with very little work. 2. One has to often reach out to individuals, blogs in search for work which might be scarce. 3. MTurk was highly consistent when it came to looking for work and fared much better in comparison to oDesk and other platforms when it came to finding work and dealing with competition.

Topic: Process when designing and iterating over jobs and pricesthat a requester has to put on a crowdsourcing platform

Observations:

(drawn from Serge) 1. Pricing depends on the amount of time it takes for one to complete the work. 2. Requesters have to assess this and sett wages appropriately which often border on the minimum wage (8 or 10 dollars an hour) 3. Requesters might refrain from pricing tasks to high as it might attract cheaters or casuals looking to make easy money. (Gordon would pay from 10 to 20 dollars for certain work) 4. Another thing that requesters need to do is assess the quality of the submission which a researcher like serge could do by keeping open-ended questions at the end which might have a subjective answer. This lets the requester guage the seriousness of the worker and the diligence with which they performed the task. It becomes apparent if the worker hasn't even read the instructions.

Topic: Restrictions on crowdsourcing platforms. For example some require SSN to register.

Observations:

(drawn from Serge) 1. The SSN restriction is purely a tax-issue. US Laws require declaration of income and this is primarily why SSN is asked for. 2. Another restriction according to Serge is that a requester can not ask a worker to download any content. This probably exists to ensure that malware is contained and does not affect the worker but Serge believed that it severely limits some of the tasks that a researcher might want to assign.

Topic: Comparison of hits and their duration with the quality of those hits.

Observations:

(drawn from Serge) 1. Workers have finite patience, attention and energy. 2. Requesters usually try to make hits as short as possible due to the above reason.

(drawn from Gordon) 1. Keep hits short. 2. Acknowledges a condition "Bubble Hell" where a worker moves from one hit to another like drone and zones out which might decrease quality of hit.

Topic: Alternatives to these platforms. What would one do if these disappeared.

Observations:

1. Several companies have their internal mini workers who are hired and can work on such tasks. 2. Google and twitter have such divisions too but have differing sizes.

Topic: Major hurdles for new Turkers.

Observations:

(drawn from David) 1. The biggest hurdle for a new turker is finding work. It requires lots of digging and finding worthwile tasks. 2. Another hurdle is filtering out jobs from requesters who consistently underpay or unfairly reject hits.

(drawn from Serge) 1. Working on crowdsourcing patforms has a steep learning curve when it comes to figuring out how to earn enough. 2. A new worker might face a lot of dejection and disappointment on realizing that they managed to earn very little in the first few weeks.

Topic: Actions a worker can take on getting their work rejected.

(drawn from "Spamgirl") 1. Initially, a worker used to be usually helpless and they had no choice but to suck it up and move on. 2. Now, a worker can reach out to the requester via professional emails and ask them about it. There exist templates that a worker can use to draft emails enquiring about rejections. 3. If they believe it has been unfair then they can reach out to the community to seek validation or check if anyone else has gone through the same experience. 4. People in the community spread awareness about unfair requesters who rejected hits when they shouldn't.

Topic: Methods of controlling quality in hits. Criteria used be requesters to reject or accept work.

Observations:

(drawn from Serge, Ranjay and Gordon) 1. Serge preferred approving almost all hits. 2. Rejection occurred only when it was extremely apparent the worker was very lackluster or did not bother to read even the instructions or chose random options. 3. He kept open-ended questions at the end to guage the worker's seriousness and diligence. Some non-serious workers or cheaters would type gibberish or plagiarize answers to get past such questions easily. 4. Some requesters keep timers on pages to check the amount of time a worker is putting in.

Topic: Alternatives for workers outside US (example Indians).

Observations:

1. Alternatives for workers outside US are few. 2. Examples of some include Clickworker (which does not offer a lot of work) and oDesk (which David criticized for having little work and high competition).


Other general observations:

1. Gianluca was believed that the quality of submissions for his academic research had increased of late. 2. He was discontent with these platforms for two major reasons: i. There is no guarantee of the time by which a task will be completed and the requester will be returned the results. ii. If a requester needs a task to be done by a specific worker then he/she has to contact them individually through emails or other means which complicates things.

Reading Others' Insights

Worker perspective: Being a Turker

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

Worker perspective: Turkopticon

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

Requester perspective: Crowdsourcing User Studies with Mechanical Turk

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

Requester perspective: The Need for Standardization in Crowdsourcing

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

Both perspectives: A Plea to Amazon: Fix Mechanical Turk

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

Do Needfinding by Browsing MTurk-related forums, blogs, Reddit, etc

List out the observations you made while doing your fieldwork. Links to examples (posts / threads) would be extremely helpful.

Synthesize the Needs You Found

List out your most salient and interesting needs for workers, and for requesters. Please back up each one with evidence: at least one observation, and ideally an interpretation as well.

Worker Needs

A set of bullet points summarizing the needs of workers.

  • Example: Workers need to be respected by their employers. Evidence: Sanjay said in the worker panel that he wrote an angry email to a requester who mass-rejected his work. Interpretation: this wasn't actually about the money; it was about the disregard for Sanjay's work ethic.

Requester Needs

A set of bullet points summarizing the needs of requesters.

  • Example: requesters need to trust the results they get from workers. Evidence: In this thread on Reddit (linked), a requester is struggling to know which results to use and which ones to reject or re-post for more data. Interpretation: it's actually quite difficult for requesters to know whether 1) a worker tried hard but the question was unclear or very difficult or an edge case, or 2) a worker wasn't really putting in a best effort.