View source for WinterMilestone 2 UE Team One


== Attend a Panel to Hear from Workers and Requesters ==

=== Deliverable ===

During the meeting, we made the following observations about requesters and workers: 

'''1. Workers '''

* Workers engage with the requesters and other workers through the cooperative forums like Turker Nation, etc. They also receive when new HITs are posted on thought the Thread/Chat Room on these forums. 

* Workers do not schedule their work day, they work directly with what they have. 

* Workers can't predict their earning from one day to another due to the unscheduled posts for task. 

* Worker's reputation is very fragile * even few rejections will not allow workers to do highly paid tasks.

* Most of workers choose highly*paid tasks, HITs that do not risk approval rating (i.e. 99%) and requesters who have a good reputation.

* Sometimes workers provide feedback to help requester improving the HITs

* Most of workers has no specific time for work it changes from day to another

* Workers prefer HITs where they can find a way for contacting requester (in case of rejection)

* Workers do effort to make their HITs accepted not only for getting paid but also for keeping their success rate high and be able to apply for high paid HITs


'''2. Requesters'''

* Requesters mostly do simple tasks in order to receive better work. 

* Requesters use built in templates to make the HITs easily

* Tasks can be in the Academic scope like exploring a hypothesis

== Reading Others' Insights ==

=== Worker perspective: Being a Turker  ===

1. The workers need the crowdsourced work environment to have a narrative, a social world, they need it to be easy and to be an adventure.  * in verbs: they need to enjoy, to find an easy context, to live stories, to find a social environment

They also need to fulfill their own missions, and not only the labour market ones, in a context that makes working fair and not leading to endless and static cycling. They need a labour market that constantly improves their lives and their development desires. * They need to evolve, to feel trust, to feel free. They need to desire and accomplish; to recognize and identify themselves in what they accomplish.

A possible idea of improvement is to make requesters give the inner motivations why a work should be done and to show what the job has led the requesters to accomplish, in a way to make workers part of a common adventure.

''(OBS #1.
''Tim Worstall described AMT as the pure form of a market, where mass individual activities of a requester/turker falls into a fair equilibrium * I think this was proven to be false, otherwise there wouldn't be so many other platforms to aid the Turks in their mission. Also I believe it's very important to remember that TurkerNation is a US based platform where many <individuals that are usually overly*exposed to technology> actively participate in order to improve '''their''' experience, this can be seen throughout the article especially in the sections regarding the wages, where it was concluded that a 7$/hour was the minimum wage * reflecting the min wage in the US at the time, even though 14k in earning clearly reflects outliers w/n the platform.
''The versatility of HITs makes it impossible to determine no of hours worked <& hourly wage implied>, which in the end could translate in frustration, especially since some tasks take longer <or forever> to get approved.'' 
'''''Need''': A platform that approaches this transparently so that new users can clearly make an idea of their earnings and that makes it easier for existing users to attain a desired wage; also for this to work, work '''needs''' to be objectively assessed which will further drive up users' trust. 

''OBS #2.''
''Despite most being part*time Turkers, many see themselves as workers, some doing it for fun & education <INT: this is imperative as it acts as a positive conditioning to their turking>, which often seems to be directly proportional to their wage, but bringing obvious advantages. '''Needs''' to address this more as gamifying the boring microtasks can drive up engagement at that level.''

''OBS #3.''
''Turkers seem to be against the media & academia as they believe AMT can be regulated by all the additional platforms, while this may be to an extent true, it's only complementing the experience and it's only doing it for a very specific part of the population using this platform <US turkers> that also are getting this improved experience in return of their active engagement on the other platforms. This would '''need''' standardisation, but the regulation they are fearing may be true indeed, and I am curious how this will be tackled by Daemo)''

2) Requesters need good quality and less risks. They also need, as the workers, to fulfill their own missions, thanks to a context that assures reliability, both of the workers and of the results. They dream of smart, knowing*all workers, that know their problems better than they do. They need, just like as the workers, a labour market that constantly improves their lives and their development desires.

In verbs, they need to achieve good results, they need to trust their workers, to feel secure and to collaborate with experienced people.

''OBS #1.
''Requesters sometime don't understand the task well enough themselves, therefore there's a huge '''need''' that needs focusing on <despite Daemo addressed this with the prototype tasks>, so that it allows the requester to hypothesise what and how the task should be done, process throughout which the Turker can comment and communicate towards a common goal.'' 

A possible idea of improvement is to have, along the labour market, also a parallel labour market that has as a job to explain better what the requesters want. ''(Fully agree, another way of doing this is TurkerNation's areas where Requesters are actively engaged with Turkers, hoping to improve HIT design. This, of course, I imagine would be a central pillar to any new platform aiming to tackle crowdsourcing, considering the chances of non*English speakers it will attract and the difference in efficacy and scalability a well designed HIT makes.)''

=== Worker perspective: Turkopticon ===




1) What observations about workers can you draw from the readings? 

Workers do treat HITs as their normal jobs and might leave the platform if they are not accepted. (especially now, when they have no actions to take against this). They want faster payments, as the delays often cause frustration.

With one hour of work on AMT workers could gain few dollars we can look at workers as a resource can be integrated into a human computation they have multiple tasks and they have to choose among them, some tasks has qualification test so the worker must pass the test before accepting this task, other tasks require that the worker has minimum success rate.
some workers do tasks to increase they wages level but others do it just for fun or time killing.

we observed that workers not satisfied with requester because of many things:
* They feel of injustice as many of their work got rejected without any reasons
* They see that the wages are not enough
* They wish more faster responses as AMT allows requester 30 days for evaluating the work and pay which is too long time 


Observations after using Turkopticon
Workers are able to rate requesters based on 4 things: 
* How fast they respond 
* How fair they accept or reject 
* How generous the requesters are
* How the communication is going 

Turkopticon made workers able to build groups of them for example when they share the same interests, also saved some workers' rights and protected them against requesters' cruelty.

2) What observations about requesters can you draw from the readings?

* employers can put their tasks maybe with some constraints like (success rate, some qualifications) or without 
* employers add some instructions to help the worker finish the task
* employers have the full right to accept or reject the tasks without texting the worker

=== Requester perspective: Crowdsourcing User Studies with Mechanical Turk ===

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

* Workers want to save transaction costs by saving time for searching and performing the tasks.

* The quality of their results decreases as more requesters don't implement mechanisms to identify and reduce spam (e.g. by implementing verifiable questions)


2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

* Malicious users have a significant impact on the quality of HITs. For example, "extremely short task durations and comments that are repeated verbatim across multiple tasks" are signs of suspicious users. Requesters are indicators of suspect edits. 

* Requesters should implement explicitly verifiable questions as part of the task to increase the quality of the results.

* Requesters should design the task such that the correct accomplishment of a task should take the same amount of effort like doing the task randomly or malicious.



=== Requester perspective: The Need for Standardization in Crowdsourcing ===

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

=== Both perspectives: A Plea to Amazon: Fix Mechanical Turk ===

'''Workers'''

When it comes to workers, we can observe following ''actions and behaviours'' that are taking place:

* Experienced workers tend to be wary of HITs from new requesters * they rather do few small tasks for them to verify if they do not reject valid work and pay promptly. 

* Workers use Turker Nation and TurkOpticon to learn more about the requesters * which requester pays on time, does not reject a work without obvious reason, etc.

* Work can be rejected even though it has been completed with respect due to requester's constraints, resulting in withheld payment, and they can't appeal the decision of the requester.

* Workers do not know how fast the requester typically releases payment, resulting that they do not know when (and if) they will receive remuneration.

* 'Bad' and 'good' workers receive quite the same payment.

* Workers are not uniformly good in all types of tasks, they have different set of skills.

* Workers do not know whether requesters post only a one*time order or they tend to post several tasks. 

* All this results in that workers actually worry about an unreasonable behaviour of the requesters. 

* At the moment there is no system to choose HITs according to interest / categories workers want to work on. Also, they cannot search for a particular requester (if only requester doesn't put his name in keywords).

* Inability to predict the competition time * most workers choose tasks by most recent HITs / groups with most HITs (have their priorities).

These observations of activities are made in the ''context'' of using and working on Amazon Mechanical Turk (mTurk)  with the recorded ''interactions'' between workers and requesters (mostly through the HITs posted), workers with other workers (through the forums such as Turker Nation and TurkOpticon and etc...), and workers with mTurk (''object, man made artifact''). ''Users'', i.e. people that we see in our observations, are mTurk workers. 

Based on the observations we can deduce the following suggestions to improve the situation of workers:

'''Reputation of a requester'''

Workers should have the ability to appeal in the case if they think that a requester has acted unfairly. 
The reputation of a Requester should be based on a set of of objective characteristics:

* Show speed of payment

* Show the rejection rate for the requester

* Show the appeal rate for the requester

* Show the appeal rate for the requester

* Disallow the ability to reject work that is not spam

'''A Better User Interface'''

* The improvement of the User Interface will reduce frustration and increase motivation of workers

'''A Better Task Search Interface '''

*The categorization and better indexing of tasks will reduce response times of workers and will lead also to more specialized workers per tasks because the workers will choose the tasks which they can do at best.


'''Requesters'''

While reading the article, we may notice following ''activities'' of requesters: 

* Requesters are responsible for providing high*quality and well*specified tasks so that they can reap highest benefits. In order to get a high*quality result they spend a lot of time while creating micro*tasks to be done.  This often  leads to them having to build their own systems to manage the workflow and interfaces, which in the end may result in hiring a person just to design tasks for the requester.

* Requesters are omnipotent * they choose whether a HIT is accepted or not and thus whether the payment is successful.

* Requesters can not make their own qualification tests available to other requesters for a fee in order to test their skills.

* Due to the lack of working history of a certain worker, requester may find difficult to trust a worker to get the task done well. 

* Requesters often assume that workers are bad (healthy assumption based on the lack of reputation system), thus they leave one HIT for multiple workers  just to get accurate results.

These observations of activities are made in the ''context'' of using and working on Amazon Mechanical Turk (mTurk)  with the recorded ''interactions'' between workers and requesters (mostly through the HITs posted) and requesters with mTurk (''object, man made artifact''). ''Users'', i.e. people that we see in our observations, are mTurk requesters.

'''Suggestions to improve the conditions for requesters:'''

'''* A Better Interface to Post Tasks'''

There is a need of a more flexible interface to reduce complexity and the necessity of third party tools. This will save transaction costs.

'''* A True Reputation System for Workers'''

The reputation of workers should not be based on a payment connected rating system. Requesters should get a detailed overview of the reputation and skills of a worker which can be based e.g. on the following set:

* Public qualification tests (Language Skills? Proofreading Skills? etc.)

* Working history (past requesters, average earnings etc.)

*Type of submitted work (Transcription? Image tagging? Classification? Content generation? Social Media activities?)

These are just a few examples which could increase the valuation possibilities of worker skills. An API could help to automate the process of selecting the best workers for a specific task.



== Do Needfinding by Browsing MTurk*related forums, blogs, Reddit, etc ==

It seems some e*workers prefer e*jobs and they want to make their e*jobs real ones:

“My SO makes $20*$40 a day, but treats it like a "real" job. I'm a casual Turker (my "real" job pays way more) so I'll make $100 every few months.” 

“I used to be able to make upwards of 50/week with a couple hours a day. Now I have an actual job, so my effort towards mturk has really dipped. I now make about 10*20/week.”

== Podcast People inside your machine ==

OBS #1.
Isac (one of the guys at Amazon who originally worked on the product) described AMT as a computer metaphor designed with an open*ended nature. He also said that rules are still not there implying (I think) that many requesters have a fluid approach when posting HITs.

INT: For example, cases in which jobs disappear in a matter of seconds (sometimes even during click load*time if your internet isn't great) may give requesters an idea of the popularity of the job. This in turn has a domino effect on the price (which automatically goes down) and the workers' pace & frustration (which automatically go up). This highlights that there's a '''need''' for regulating the prices of HITs while focusing on the worker's advantage, as this will build stability and act as a trust foundation, while keeping frustration in its place.

== Synthesize the Needs You Found ==

=== Worker Needs ===


[[File:WorkerNeed.jpg|900px]]

In the picture above we can see some needs of workers that we deduced from the observations made. 
In general, firstly workers need to trust their requesters. 

- Workers need to be respected. With respect comes trust.  

a) Need to be paid on time and fairly. Evidence: a lot of workers are anxious about the mass-rejection of their work and the time-pool for the payment. Interpretation: workers want them to be respected, even though their work may not explicitly meet requester's wants, they still spent time on doing this. They want to be able to trust their requesters to be paid on time because they want to count on this money. 

b) Need to be judged fairly. Evidence: again, worker can spend a lot of time doing the job but it can either be accepted or rejected. Interpretation: Due to the lack of communication between requester and workers, requester may think that worker is bad, when just the requester's explanation of tasks wasn't clear. Workers need to be able to appeal the rejection in order to keep their reputation.

- Workers need to communicate more with their requesters. Communication is building trust between requester and worker, win/win situation because requester gets the quality work and worker gets payement and good reputation.

Secondly, workers need to stay motivated and trust the site in a long term. 

- Workers need to easily find tasks for their skills set. Evidence: workers can't navigate through site properly and find interesting task. Therefore we can derive :

a) Workers need have a qualification test for their skills. Evidence: every worker has different set of skills. Interpretation: having qualification made by requester or something similar may economise a lot of  time for workers to do the job and for requester to get a quality work. 

b) Workers need to have better API. Evidence: for now, the interface is not very user-friendly and workers have difficulties to find different/ interesting tasks. Interpretation: the difficulty of using interface can be directly influencing motivation of workers. Better API may ease the work on the site and economise work-time. 

-Workers need to be able to plan their budget. Evidence: workers do not know if their job's going to be accepter and when the next task will be available.  Interpretation: Minimum payement should be introduced, because worker still spent some time working on the HIT (see the link with another need). 


* In "Being a Turker" it is observed how big of a role transparency plays when working on low*paid microtasks, sometimes even half*way across the globe. I could almost feel the tension between the two classes as you probably have a huge discrepancy between the two in terms of how money is perceived (some working for 4*6$/day, albeit as part*time most often than not). There is a need for activity to be carried out with utmost transparency, for the purpose of easily tracking and predicting the income. 

* In "Being a Turker", some users have reported (despite being deeply correlated with the wage) the "turking" activity as sometimes having a fun and educational side. A need for gamifying the experience could drive users excitement for microtasks and perhaps even future engagement if it successfully acts as a positive conditioning.

* In "Being a Turker", Turkers seem to be against the media & academia as they believe AMT can be regulated by all the additional platforms. While this may be to an extent true, it's only complementing the experience (which any platform should keep internal) and it's only doing it for a very specific part of the population using this platform (US turkers) that are only getting this improved experience in return of their active engagement on the other platforms. This would need standardisation, but the regulation they are fearing may be true indeed, and I am curious how this will be tackled by Daemo

=== Requester Needs ===

[[File:Bildschirmfoto 2016-01-25 um 05.03.55.png|900px ]]

The map illustrates the interpretation process of the observations.
 
The main needs of requesters which were observed during the panel and in the readings are:

* Reduction of costs (HR, Development of custom software, "bad quality results needs re-runs" etc.)
* Flexibility (They need to create custom tasks, integrate results and tasks into existing systems)
* Time savings (Better UI, redundancy of workflows, no repetition because of low quality results, fast selection of specific workers)


Reducing of costs can be achieved by reducing the transaction costs e.g. by ensuring proper allocation of qualifications, offering an API to gather specific information on workers (e.g. qualifications, history of work) and to select specific workers based on specific selection criteria. 

To offer more flexibility during the processes of task generation, the requester needs an intuitive and functional interface. Also the option to use the capabilities of an API to substitute e.g. old technologies like iframes would increase the flexibility of a requester.

Time savings can be reached by increasing the quality of the results. Low quality results need longer evaluation periods, the repeating of the main processes leads to general suspicion on both sides. To generate high quality results, the tasks should be executed by workers who are familiar with similar tasks. Therefore the automation of the processes to select workers based on a set of specific criteria via an API would increase time savings and quality of the results.

The observations imply that a) requesters need a true reputation system for workers to increase time savings and quality in the long term. b) The requesters need a REST API for automation purposes but also to increase flexibility and to decrease spendings for "dirty"/hacked solutions (e.g. iframes, scraping etc.)

A higher-quality result 

Also observed happy cases that decrease adversarial tension between the two camps can be found in "Being a Turker" article, when workers helped requesters in the creation process of a HIT. A need for a constant collaborating between requesters and workers could improve HITs drastically through design, making them more efficient and scalable (however I suppose this was one of the biggest findings since Daemo developed the prototyping episode where workers give feedback).



Team Members:

@seko - Sekandar Matin

@amdp - Alessandro Merletti de Palo

@ahmednasser - Ahmed Nasser

@purynova - Victoria Purynova

@vlado  - Vlad omete

@jermenkoo - Latal Jaromir

@kamilamananova - Kamila Mananova