WinterMilestone 2 PierreF

From crowdresearch
Jump to: navigation, search

All contributions : PierreF_Space

Previous contribution : WinterMilestone_1_PierreF

Attend a Panel to Hear from Workers and Requesters

Above all, I found a palpable anxiety from workers to maintain their acceptance ratio. It is clearly critical for them to maintain access to interesting jobs and must be maintained at all cost. This leads to elaborate strategies to make sure an untested requester will not destroy this position as described by Laura. As a consequence, building direct contacts with requesters appear to be essential to help ensure that any possible rejection could be amended. This is essentially opposite to the initial presentation of Mechanical Turk by Jeff Bezos where MT is meant to hide outsourcing to people behind the system and its API. It is also essential to understand that for some people like Laura, this source of revenue is essential and MT represents one of the only way to make a living while staying at home. The capacity to found help in community like Turker Nation that ensures a certain level of moderation and get information about requesters. Fair relations are a repetitive subject we will also find in the article and it is easy to see that the absence of direct interactions can lead to frustration and excessive reactions. Also of interest, Rochelle stressed the difficulty to plan for activities. You do not have advance alerts for job arrivals and everything run real-time on a 24h-a-day basis. This is also true concerning the revenue that can be generated. This leads to a very insecure situation as well as a strong barrier for using the platform for short windows of activity.

On the requester side, the main point seems to me that if one wants to really manage its relationship with workers, it becomes a full-time job (Chris feedback). At the end of it, organizing tasks, managing interactions with workers constitute hidden costs due to lack of support from the platform. Obviously, unless someone is going to have a lot of recurrent work, the upfront investment is not negligible and everything requires a real organization on the requester side. Requesters are also concerned with their reputation to be able to attract the best workers. So trust issues run both ways, and both sides are in the constant stress to manage this aspect.

Possible bias : all the attendees are US-based. Requesters have all an academic background. In particular, requesters appear to be quite restrained in their rejection policy, it would be interesting to have more general information on this particular subject.

Reading Others' Insights

Worker perspective: Being a Turker

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

The first finding confirms the key attraction of the platform that is making money. Again the article underlines the fact that the forum used to conduct its analysis (Turker Nation) is mainly used by US residents. It is interesting to consider that the large majority of examples in the article consists in women. If we need to infer a archetype from this type, we could (perhaps wrongly) deduct that a lot of the population of workers consist in at-home mothers. The capacity to remain at home is then critical. The article stresses the fact that MT platform plays a role of social nets for a lot of workers. The article informs us that MT can be the primary source of revenue for some of them, but also that some employees use the platform to complement their wage, apparently even with the support of their employers.

Being part of aa group becomes essential to understand the best ways to make money and which requesters are best to work for.

The fact that people targets a level of revenue can be paralleled with the complaints about the inability to plan revenue. Communication between workers help also in this respect. The fact that people want to achieve a revenue equivalent to the federal minimum may be linked to some sense of minimal acceptable value of their work. On the other side, it seems difficult to achieve better results. The article mentions quite rightly that different standards in cost of life represent a difficult challenge for defining acceptable wages.

The mix reaction towards unions is remarkable : people want to remain free and the choice of MT against a more stable job can result from this preference of independence. The rejection of regulation is certainly part of the same mindset. All these are compounded with of course the fear that additional constraints could scare away requesters. The organization of direct relationships between requesters and workers could in fact potentially lead to reevaluation of the legal framework associated with the relation between workers and requesters depending of the concerned countries (I am not a specialist regarding US laws).

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

The article mentions the behaviour of some requesters and the strong reactions from workers to work rejections. Mass rejections jeopardize a worker statute in the system and could take a long time to correct. The quality of the interaction is an important factor : workers want to be considered and respected as human being in a system that tends to market human as a ressource.

Worker perspective: Turkopticon

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

This article mainly concerns the experience of creating a system to help workers rate requesters. The article addresses the main troubles that can distort it :

  • protection of workers developing the reviews against retaliation
  • fake reviews posted by reviewers

The need for moderation is natural conclusion of the mentioned problems. It raises the question of the economics of a sustainable long-term mechanism. The boostrapping process by paid crowd sourcing is an interesting concept.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

Not much

Requester perspective: Crowdsourcing User Studies with Mechanical Turk

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

Not much

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

The article mainly demonstrates the importance of task definition in order to achieve usable results from crowd working. The experiment tested subjective questions but, even in this case, achieved efficient results as long as tasks are defined correctly. Obviously such an expertise is not simple to acquire and require a solid investment in task design.

Requester perspective: The Need for Standardization in Crowdsourcing

The article advocates for a modularization and standardization of tasks definition.

The most interesting point to me is this could lead to an innovative price fixing solution. Financial markets give us the example of efficient price fixing and the author proposes to adapt this example to find the right prices in MT.

Of course, standardization of elementary tasks also gives us an easy way to develop efficient development framework for task definition.

I share the point of view of the author with the useful parallel between the evolution of the auto industry as mentioned in my previous post, but as this evolution has been thoroughly analyzed I think that the full implications between the two situations must be analyzed further.

Both perspectives: A Plea to Amazon: Fix Mechanical Turk

This blog article examines the defaults of the current state of development of MT.

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

On the worker side, the author describes correctly the fact that MT is not develop with the workers in mind. No help in task selection is provided in the interface. More generally, no tools to ease the life of the workers has been developed by Amazon.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

On the requester point of view, it underlines the difficulty for a requester to be a mass provider without having already a well-established reputation. As workers fear above all mass rejection, the requester faces a lack of good workers and recruits only the less experienced ones or scammers.

The lack of good task definition environment is also underlined as well as the limitation of the available API for integration. The lack of worker rating system is noted but we have seen that approval / rejection rates currently play this role by default. On the same subject, the author proposes the introduction of qualification tests to help the assessment of workers skills, but such an introduction will introduce a whole new field of potential fraudulent misrepresentation that is not mentioned in the article.

One could infer from these proposals a balanced rating that could take into account the attitude of reviewers in general to help in balancing and weighting reviews according to the relevance of the reviews.

Synthesize the Needs You Found

Worker Needs

  • Need to get access to fair-priced jobs - Evidence : Laura interview, Martin & al. article, Ipeirotis blog entry, Interpretation : solution by efficient price fixing, tools to match expectation and jobs
  • Need to make sure their acceptance rate remains high - Evidence : Laura interview, Martin & al. article Interpretation : job acceptance is not the correct rating system, definition of a better way to detect wrong doers (not necessarily rating)
  • Need to be able to plan revenue generation - Evidence : Rochelle interview Interpretation : intermediate mechanism for job distribution and task sharing, specific reservation system
  • Need to humanize the relationship with requesters - Evidence : Laura interview, Martin & al. article Interpretation : trust building mechanism, conflict resolution process, third party arbitrage, in relation with point 2/, workflow of interaction management between workers and requesters, clarification accesses.
  • Need to exchange between co-workers : Rochelle interview, Irani & al. article Interpretation : introduction of intermediate group dynamic in the platform, add-on tools for inter-workers communication and organization.

In general, provide a plug-in capacity for scripting extension and tool addition to the platform.

Requester Needs

  • Need to provision micro-tasks on limited budget. Evidence : Chris interview, Kittur & al. article. Interpretation : framework for task definition and pattern-oriented task definitions, standardization of available basic actions.
  • Need to ensure a sufficient quality level. Evidence : Chris interview, Kittur & al. article, Bernstein & al. article. Same interpretation as previous point.
  • Need to integrate requesting in their proprietary workflow. Bernstein & al. article, Ipeirotis blog entry. Interpretation : advanced API for integration in proprietary external workflows,
  • Need to structure response management and organize relationships with Turkers. Evidence : Chris interview. Interpretation : part of the tool for managing the interaction between workers and requesters, see also introduction of intermediate group dynamics
  • Need to build quickly worker crowd. Ipeirotis blog entry. Interpretation : Introduction of an initial monitored intermediation by the platform for new users (requesters are less numerous than workers in principle, the idea of sponsorship or mentoring could be introduced regarding the worker point of view).

Milestone Contributors

@PierreF