From crowdresearch
Jump to: navigation, search

Experience the life of a Worker on Mechanical Turk

Amazon's site seems adequately designed (at least for an engineer, although I suspect the average user might have difficulty), but the 48 hour waiting period has taken longer for me, so I'm not happy about that. They just approved me. I think they need a mechanical turk doing their approvals! :p The few projects I was able to see seem to have better pay than some of the other sites, but it still low. I also performed some tasks for project members on the sandbox, which had some design issues (like the submit button hiding until you scrolled down, as well as having to remember to click accept hit before filling out the fields).

Experience the life of a Requester on Mechanical Turk

I was only finally approved for the formal MT site a few minutes ago, so I wasn't able to properly request a project. I did submit a project but was unable to find the HIT URL to post to wiki, perhaps because my account wasn't yet approved.

Explore alternative crowd-labor markets

Crowdflower is one of the worst designed websites I have ever used, with lots of vague clickthroughs and multiple clicks and windows required for a simple, common action or to reach a required place on the site. Hierarchy and order doesn't exist here. The sign up process was confusing and annoying, and later required a facebook be linked to it. The page also implied that the fb email and the original signup email used for crowdflower needed to match, but this wasn't true. The jobs available to me were all either one or two cents, and the first few I clicked were't even available. The one job that I was finally able to attempt to execute involved 10-20 minutes of confusingly labeled work and when I finally clicked submit the system gave me an error, even though I'm quite sure I'd done everything correctly. After this I gave up on crowdflower.

Microworkers seemed to be the best of the three, both in terms of design/ease of use and ease of signing up/available jobs and compensation rates (from my limited browsing). I took a university research job that paid one dollar for about 10-20 minutes of work. The research site/survey could have been better designed and was unnecessarily confusing, partially because of the English errors throughout. I believe the designers/researchers messed up in a few sections due to their ESL status, and this will likely skew their results. It also made my work harder, and more much confusing/frustrating. Once I ascertained that the confusing parts were likely due to language errors I got the project done rather quickly, though the dragging portion of the adjustable bar graph (to gauge my opinion on the subject) could use a more intutive approach. Finally, the actual university site told me I had finished the job at the end, and would be compensated in a week. Unfortunately the Microworkers site gave no such indication that I had completed the job, as my list was completely empty. This leaves me worried I will not get credit and I would imagine this lack of sync is bothersome for many workers.

Compare and contrast the crowd-labor market you just explored (TaskRabbit/oDesk/GalaxyZoo) to Mechanical Turk. Taskrabbit is cheerfully designed, relatively intuitive, and I didn't hate using it. By far the best designed of the sites I explored. I can see why American workers might be attracted to this site, not only for the higher pay, but also for the simplicity and ease of use.



  • What do you like about the system / what are its strengths? I like that it enables "poorer" workers. The wages may not be exploitative in India, and paying them more may actually result in undesirable social problems in a given community (raising the price of basic good, for example). The ability to use relatively primitive technology to work and complete moderately advanced tasks is also a clever use of simple mobile phones and possibly mobile primitive networks.
  • What do you think can be improved about the system? It seems like the technology is self-limiting, with user accuracy and ability being limited by the nature of the phone. However, the user may not have enough computer literacy to currently use a more complex device. On the other hand, an average person could like learn to use a tablet within a few hours, but tablets need more mobile data and access to a reliable power source.


  • What do you like about the system / what are its strengths? I largely agree with the statement at the beginning of the abstract:

"The success of a crowdsourcing platform depends on a strong foundation of trust between workers and requesters. In current marketplaces, workers and requesters are often unable to trust each others’ quality, and their mental models of tasks are misaligned due to ambiguous instructions or confusing edge cases. This breakdown of trust typically arises from (1) flawed reputation systems which do not accurately reflect worker and requester quality, and from (2) poorly designed tasks."

I've come to similar conclusions during my own crowdsourcing/funding research.

I think design will be important for a site like Daemo, both in terms of interface, UX, and projects. My experiences on Microworkers caused me to unnecessarily lose trust in the platform, primarily through poor design and UX decisions that left me wondering if my jobs were actually recorded properly.

  • What do you think can be improved about the system? I need to log into the site (haven't been able to do so yet) before I can properly answer this question.

Flash Teams

  • What do you like about the system / what are its strengths? Pipelining sounds good. A common complaint of workers in teams is having to wait for other workers to finish their tasks before starting their own. This is particularly common in complex tasks with multiple skilled workers.
  • What do you think can be improved about the system? I'd love to see a simple invision prototype of this system or a more detailed flowchart/storyboard of the systems described, so that I might have a better idea of how they work.