Milestone 2 Team Innovation

From crowdresearch
Jump to: navigation, search

This will continue to be a work in progress there is a ton of information to cover, and three days isn't enough time to do it justice. We've included our "good copy" at the top and our "rough notes" for future work at the bottom.

Worker Needs

Workers need to feel like they are achieving something meaningful. MatrixMan92 struggles to justify crowdsourcing as a viable means of employment in this thread on reddit: http://www.reddit.com/r/mturk/comments/2wn4ri/you_know_you_have_to_get_a_real_job_right/ “Man, sometimes I dont feel like doing this anymore just because of how frowned upon it seems to be by people in my life. Anyone else go through stuff like this? How do you stay motivated?”

MatrixMan goes on to lament the difficulties in earning money here: http://www.reddit.com/r/mturk/comments/2x1589/what_am_i_doing_wrong/ “I see even newcomers only work a few hours a day and make 30+ dollars. I have been working since 8 AM this morning, non stop, and have made 15 dollars ..it is now 1 PM.”

Workers need to believe there is a viable future in crowdsourcing. It is very easy to think of all of things that we might want to enhance our experience and add to it, but there are also many things that we might take for granted and/or not even realize we have. Personally, I believe in the idea of crowdsourcing from both perspectives; worker and requester alike. I believe that crowdsourcing will be around for a long time to come and overall I believe that mTurk is a great platform for this type of work.

Workers need to be able to trust the requester. Lepi expresses his frustration here: http://www.mturkgrind.com/threads/what-would-your-ideal-crowd-work-platform-be-like.27462/#post-497406 “I think my biggest thing would be accountability. The fact that Amazon itself has NOTHING to tell workers if the person they are working for is any good, but has piles and piles of statistics about workers for requesters to look at, bothers me to not end. Ideally it should be a two way street.”

monkeeeeee publicly questions the integrity of requester on reddit: http://www.reddit.com/r/mturk/comments/2xp7tg/is_crowdsurf_legit/ “It just seems to me that it's entirely too faulty, unfair, and unethical for paying sweatshop wages.”

Requester Needs

Requesters need to be able to trust the worker: Flipper3 recounts his experience will poor results on reddit: http://www.reddit.com/r/mturk/comments/2yemqj/requester_why_am_i_getting_so_many_workers_trying/ “At first everything was going well, but I am currently on my batch of 50-100 and am having to reject a lot of workers due to them faking their responses. “

Requesters need to have their work completed in a timely manner. khusocial writes here: http://mturkforum.com/showthread.php?28580-Why-nobody-submitted-my-hit-Could-you-help-us-to-figure-out-the-problem

“Our research group has published a hit recently: Reading, imaging and investigation, take about 7-10 minutes. The payment is 0.25 dollar.

After several days, no submission at all. We don't know where goes wrong of the hit that nobody answers it.”

This demonstrates a larger, underlying need to be able to understand the needs of the workforce.

Requesters need to have control over their projects: flipper3 struggles to find balance on reddit: http://www.reddit.com/r/mturk/comments/2yemqj/requester_why_am_i_getting_so_many_workers_trying/ “I initially started off with only the Master's qualification as that is the default set by Amazon, but I could not fulfill by first 50 so I read and switched to a minimum of 5,000 completed HIT's and 98% acceptance rate.”

Overall, these needs lead to a theme of trust and collaboration. For the Requester and Worker to come to terms on expectations and results.

Rough notes:

Workers:

Need a mechanism to get their accounts verified : http://ai.reddit.com/r/mturk/search?q=verify&restrict_sr=on http://ai.reddit.com/r/mturk/search?q=suspend&restrict_sr=on

Need to socialize

Need to prove themselves to Requesters, to bring more Requesters to the platform, to get more quals, etc.

Need to get started.

Need to earn. http://www.reddit.com/r/mturk/comments/2x9m41/the_fabled_elusive_batches/ Today marks 2 weeks of mTurk, and I've managed to average $15-20 a day with $315 total made. This sounds great, but I'm working almost 12 hours a day EVERY DAY to achieve that. Granted, I only reached 1k approved HITs the other day because I've been on the survey train to make as much money as possible.

http://www.reddit.com/r/mturk/comments/2x1589/what_am_i_doing_wrong/ I see even newcomers only work a few hours a day and make 30+ dollars. I have been working since 8 AM this morning, non stop, and have made 15 dollars ..it is now 1 PM.

Need to find HITs that match their skills and abilities.

http://www.reddit.com/r/mturk/comments/2vr30e/coming_from_clickworker_i_need_some_help_here/ So im used to doing these hits on clickworker where you rate "how to guides, Search results, and whatnot" I found one that looks sort of like what i am looking for on mturk "Categorize these products from amazon.com" But i need a ClassificationAccuracy score of 85, ive got 80. How do i raise this? Need to interact with requesters.

Need to be protected from advantage-takers. http://www.reddit.com/r/mturk/comments/2xp7tg/is_crowdsurf_legit/ It just seems to me that it's entirely too faulty, unfair, and unethical for paying sweatshop wages.

Need to keep track of HITs and payments.

Need to advance their career. If we want to keep workers long term, advancement opportunities must exist. http://www.reddit.com/r/mturk/comments/2wn4ri/you_know_you_have_to_get_a_real_job_right/ Man, sometimes I dont feel like doing this anymore just because of how frowned upon it seems to be by people in my life. Anyone else go through stuff like this? How do you stay motivated?

http://www.mturkgrind.com/posts/497406/ I think my biggest thing would be accountability. The fact that Amazon itself has NOTHING to tell workers if the person they are working for is any good, but has piles and piles of statistics about workers for requesters to look at, bothers me to not end. Ideally it should be a two way street.

Need… http://www.mturkgrind.com/threads/positive-feedback.19831/ It is very easy to think of all of things that we might want to enhance our experience and add to it, but there are also many things that we might take for granted and/or not even realize we have. Personally, I believe in the idea of crowdsourcing from both perspectives; worker and requester alike. I believe that crowdsourcing will be around for a long time to come and overall I believe that mTurk is a great platform for this type of work.

Requesters

Need access to the platform http://rh.reddit.com/r/mturk/#rh

Need to post HITs quickly and easily. http://www.reddit.com/r/mturk/comments/2yemqj/requester_why_am_i_getting_so_many_workers_trying/ I initially started off with only the Master's qualification as that is the default set by Amazon, but I could not fulfill by first 50 so I read and switched to a minimum of 5,000 completed HIT's and 98% acceptance rate.


Need to get HITs completed quickly. http://mturkforum.com/showthread.php?28580-Why-nobody-submitted-my-hit-Could-you-help-us-to-figure-out-the-problem “Our research group has published a hit recently: Reading, imaging and investigation, take about 7-10 minutes. The payment is 0.25 dollar.

After several days, no submission at all. We don't know where goes wrong of the hit that nobody answers it.”


Need to receive technical help with their HITs. http://mturkforum.com/showthread.php?32445-Need-help-with-a-simple-survey&p=1062701&viewfull=1#post1062701 “I went into the code and copied the first yes/no a couple times and changed the questions, but one user reported that they could only select ONE yes or no for all 5 questions. Can someone help me with what I figure is a command to put between each question to separate them? “


Need to attract workers http://www.mturkgrind.com/threads/how-to-improve-my-hit-and-attract-more-workers.27461/ “I'm guessing the main thing that keeps people from doing these HITs is that it requires you to record your screen and voice and they probably think that is hard to do. But there's a free Chrome extension that can be installed in less than a minute.

http://mturkforum.com/showthread.php?15237-How-long-until-I-should-get-responses Hey I've recently started using mturk for surveys with the goal of crowdsourcing a startup. It may be that I don't have the patience but I've found I keep sweetening the pot / removing restrictions to get turkers to go through my survey.

Any advice?”

Need to transfer payments to workers.

Need to account for money

Need to maintain integrity of HITs. Need to limit how many times a worker does a HIT, if necessary. http://www.reddit.com/r/mturk/comments/2yemqj/requester_why_am_i_getting_so_many_workers_trying/ At first everything was going well, but I am currently on my batch of 50-100 and am having to reject a lot of workers due to them faking their responses.

http://www.reddit.com/r/mturk/comments/2wknl9/frustrated_there_are_no_more_available_hits_in/ So this morning there have been about 10 HITs in a row that I've clicked on and get the message " There are no more available HITs in this group. See more HITs available to you below. " SO FRUSTRATING!!!!! This seems like a worker complaint, does it need to be moved upwards? YEP

Need to sort people Will there be "qualifications"? How will they be handled? How transparent will they be?

Need to filter workers to meet research subset.

Both

Need to have the ability to learn/train easily (in ref to below) How do you set up an account? What sort of payment types will exist for both requesters to pay workers, and for workers to withdraw from their worker account? How is a worker account set up? How is a requester account set up? What information is going to be needed for both? Will you allow foreign workers? How about foreign requesters? What information do you need to make an account? How are taxes handled?

Need efficiency in their work/save time/save money What addons that are used by workers are needed for this new platform? How do you plan on implementing them? Will there be a way to set up what addons you as a worker, or requester, use?