WinterMilestone 1 PierreF

From crowdresearch
Jump to: navigation, search

All contributions : PierreF_Space

Experience the life of a Worker on Mechanical Turk

Unfortunately, I was not been able to subscribe to Mechanical Turk : my application was rejected by Amazon without much an explanation. Anyway, some browsing makes me aware of various obvious facts:

  • An interface particularly unattractive
  • Fees appear mainly particularly low (I cannot imagine anyone making a living from it, some urgent jobs get better paid but I imagine they are not that frequent).
  • On the task side, the required time for completion seems quite poorly evaluated

This appears clearly a seller market with a very strong deflationary bias. People are not buying talent but unloading boring and time consuming tasks for the less possible money. On the reflexion, one particular disturbing fact comes from the fact I had to master some non trivial information to get started while, at the end of the day, the pay-level is clearly uncorrelated with this capacity. I guess on the long run, it should be possible to augment productivity, but it is clearly not an attractive prospect.

I succeed better with registering in UpWork (formerly oDesk). The experience was quite different (comments in part 3.)

Experience the life of a Requester on Mechanical Turk

The above reflection on MTurk is reflected by the request interface. From the beginning, the expected jobs are expected to fit in predefined categories: classification, ratings or data cleaning. One can see why Amazon was positioned on this offer: they simply extend a platform that was clearly develop first and foremost for their inner needs. The dismal user experience makes me thinking this will remain a side project for them. From my perspective, using the platform clearly implies specific needs that result in possible cross-validation and massive very repetitive micro-tasks. The way to get satisfying level of quality is simply to ask for concurrent answers and run for comparaison.

This said as I am not living in the US, I was not in a position to go forward in submitting a project.

Explore alternative crowd-labor markets

The experience with UpWork was rather more engaging : First, you are put in the shoes of a business owner: the whole subscription is oriented to help you in selling your skills. On the minus side, this asks for a lot of personal information but I guess we cannot have the best of both world. The available works are also of a totally different level. I choose to present myself as a technical writer. Most of the open propositions consisted in long term relationship (several months) and ask for rather creative production. I can think to be in a position to create recurring business relationships with a customer. This is overall a completely different value proposition and much more of a real marketplace for skilled workers. Some basic accreditations can be achieved for free and enable one to validate its profile. Of course, the fact that we are facing a modern interface helps in the global experience. The pay-level appears still not enough to sustain a living but in not ridiculously small (in an international context, I am not sure to completely understand how tax issues are managed as that can change some perspectives).

When entering the system as a customer (posting jobs), you are presented possible job applicants: I cannot say if profiles are fake but the population seems on average consisting in young graduated students who are making a first run in their professional career. Skill is the key word there. You clearly can access cheap talent with taking the risk of working with people that are still in their learning curve. Something that you cannot expect from Mechanical Turk.

As a conclusion, we are comparing two very different business situations :

  • in one case (MTurk), I am mainly trading in people time. There are no way (and no need) to differentiate between workers. The task are really micro and have no value per se, it is only from a massive accumulation of small results that a value is achieved. As such, there is virtually no real limit how much small a task can be paid.
  • on the other case (Upwork), we have a freelancer market place. The value of the platform is in playing the middlemen in creating the relationship between the offer and the demand. Of course, this value proposition is more attractive to the less experienced workers and the less fortunate buyers, but can be a great way for people to build experience and customer portfolio. Workers are indeed rated and can differentiate one with another. In this case, tasks are not so much micro (we are talking more about one-hour a piece tasks).

We are thus looking at a complete range from very micro- unskilled job distribution to elaborate market place. More dedicated platforms (I think for example at sites like Photolia or even Quora, but Amazon reselling business come to my mind too) present characteristics that make them part of this spectrum that has been dubbed Wikinomics (Is it not what Internet is all about?). I cannot think of one solution to solve all these very different profiles. Then if targeting precisely micro-tasking, it would be interesting to understand what kind of jobs are suited to fit this model. I will certainly see a parallel between this landscape and the analysis of the evolution of the automobile industry as made in "The Machine that Changed the World".

This opens a vast field of research regarding the economy mechanisms underlying wikinomics but I am not sure this pertains to the scope addressed here (but will result in exposing the fundamental mechanisms for pricing and competition in this economy, so answer some interesting open questions for the functioning of the platform).



The article describes a system clearly dedicated to unskilled workers.

  • What do you like about the system / what are its strengths?

The system aims at lower the hurdles of task processing. Wages are immediately made apparent to workers.

  • What do you think can be improved about the system?

It remains to be proved that the system can be extended beyond the very specific situation of text or audio recognition. The wage remains incredibly small (but they are dimensioned for people earning 2$-a-day). No reward seems to be more particularly directed to people with higher accuracy scores.


Daemo is a research platform aiming to solve two observed problems: inaccurate rating between users; and misunderstandings between task provider and workers concerning task objectives.

  • What do you like about the system / what are its strengths?

An elegant way to solve the problem of rating users is proposed with aligning ranking of job proposal with the scores attributed to the previous interactions between players. A prototyping mechanism helps validating task definition before these tasks are published to the full worker community.

  • What do you think can be improved about the system?

Team building mechanisms would be an important enhancement in my opinion. This could lead to new social dynamics inside the platforms. An inspiration can be guilds found in MMPORG systems. In the same inspiration source, virtual trophies and properties have demonstrably real social values and can provide rewards to members at no cost, while contributing to reputation building. On the task design side, pattern-based approach can be adapted to enhance the current task design system. A reflexion on parallels between agile methodologies and task definitions can be also a fruitful way to enhance this crucial step.

Flash Teams

The article tackles the subject of organizing micro-tasking in a context of complex project involving highly-skilled workers. It rightly underscores the fact that main crowd sourcing platforms target very simple tasks for the moment.

  • What do you like about the system / what are its strengths?

The described automatic planning engine is a really impressive aspect of the system. The ability to infer team skills from one application context to another is a differentiating capacity.

  • What do you think can be improved about the system?

As mentioned by the authors, it would be interesting to introduce something to develop further group dynamics. In particular, a scoring of proficiency would help detecting groups that work better together. The teaming system can then associate them more often. The provided examples show only very simple task splitting. In terms of organization of complex projects, it is almost certain one should go in more details in the description of the structure of tasks. Pattern-oriented generic structures would be useful to help in the design of project.

Milestone Contributors