Difference between revisions of "Milestone 2 ZSpace"

From crowdresearch
Jump to: navigation, search
(Browsing Blogs)
(Browsing Blogs)
Line 117: Line 117:
 
* Requesters often ask turkers about technical know-how of designing a task.
 
* Requesters often ask turkers about technical know-how of designing a task.
 
* Requesters use these forums to analyse the needs and efficiency of the turkers.
 
* Requesters use these forums to analyse the needs and efficiency of the turkers.
*
+
* Turkers decide reputation of a requester based on reviews on the forums.
 
+
* Turkers outside of USA are willing to work but it is difficult for them to get approved.
 +
* Both turkers and requesters are not happy with the MTurk interface and standardization can be helpful.
 +
 
 
[http://turkrequesters.blogspot.in/ requester turker blog]
 
[http://turkrequesters.blogspot.in/ requester turker blog]
 +
 
[http://paulgoodman67.hubpages.com/hub/How-to-avoid-MTurk-scams how to avoid scams]
 
[http://paulgoodman67.hubpages.com/hub/How-to-avoid-MTurk-scams how to avoid scams]
 +
 
[http://www.reddit.com/r/HITsWorthTurkingFor/ links worth turking for]
 
[http://www.reddit.com/r/HITsWorthTurkingFor/ links worth turking for]
 +
 +
[http://www.turkalert.com/ turkalert]
 +
 +
[http://brokenturk.blogspot.in/ broken turk]
  
 
== Need synthesis ==
 
== Need synthesis ==

Revision as of 20:20, 11 March 2015

Attend a Panel to Hear from Workers and Requesters

Deliverable

The following realizations and revelations were brought to light in the Panel that happened in the morning:

  1. For a full-time Turker, finding the right HIT itself is a full-time work. There are a lot of tools, scripts that have come to a workers aid. Forums like MTurklist,crowd worker, crowd alert. There are tools like hitscraper listed all the HITS that appear real-time and meet a certain criteria. The full-time Turkers go much beyond casual browsing for HITs, to the extent of having multiple screens to be able to track all the HITs.
  2. For most of the Turkers the basic incentive to Turk is money. But the community there allows Turkers to stick around and help others, share HITs, provide support. "You become a part of a community. The ability to have people who understand what you are going through allows you to stick on with Turking"
  3. The requester-worker relationship is much beyond task completion. To contact requester, email them. The subject can be as varied as - "Why was I rejected, your request is unclear, you are paying less". Further, Turkers suggest ways to frame a better request. Requesters are welcome by the Turkers to forums and they have conversations with them. Turkers help the requester from start to finish, even with handling APIs and everything because Amazon doesn't have that detailed support.
  4. The requesters are well-connected. It is like a facebook group, where they talk about almost everything under sun: their highs and lows.. how to be more efficient..drama and issues.. It is literally like any other forum.
  5. The steep learning curve of AMT makes it the biggest hurdle for a new Turker to stick on. A newcomer simply cant imagine making money using the platform. One needs to know working with scripts. Drop off rate on MTurk is severe due to lack of support and the new platform should provide support service and documentation, things that retain good workers. Horrible interface. MTurk doesn't seem to care about it.
  6. For people not from USA, finding jobs is tougher and they (Indians) have to settle for really underpaying jobs like transcribing receipts. Ultimately they are frustrated and they quit. Since 2012 an Indian cant get a new MTurk account.

From a Requesters point of view:

  1. As much as a Turker wants the Requester to be active on forum, a requester prefers to be communicated via Email. They are preoccupied with their own research or job.
  2. The one restriction that may bug the requester is its requirement that you cant ask the Turker to download anything (software) for security purpose against malwares. It is not doing much it is affecting the good actors because the bad ones are going to do it anyway.

Reading Others' Insights

Worker perspective: Being a Turker

Crowd sourcing has aptly been defines as "the act of a company or institution taking a function once performed by employees and outsourcing it to an undefined (and generally large) network of people in the form of an open call" by Jeff Howe. The attempt made by the researchers is to represent the concerns of this network of people and their relationship with the requester and with each other.

Statistics about Turkers

The following statistical information has been collected by the researchers:

  • 80% of the work on AMT (Amazon Mechanical Turk) is done by the most active 20% of the Turkers.
  • 56% of Turkers are US nationals and another 36% are Indians.
  • Primary motive is money. Though low wages on AMT tell a paradoxical tale.

Turkers' concerns

  1. Unfair Rejection of Work - The instances when the Turkers feel that their HITs are unfairly rejected are highly frustrating, as it not only deprived them of the earning they should have made for spending the time and effort on task but also it spoils their reputation and a future chance of work.
  2. Slow Payment - Many Turkers turk for fast money and slow payment tend to defeat the purpose.
  3. Low Wage - Few HITs end up paying less than the official minimum wage.
  4. Poorly Designed tasks by the Requester - This results in misunderstanding the task and rejects that could have been avoided.

It has been felt that the design of AMT largely supports the needs of Requesters over those of Turkers. Suggested solution is a plugin to rate the Requesters. This is similar to the services provided by Turkopticon.

Reason for Turking:

Money remains to be the primary reason. Few Turkers turk to meet their ends while others do so to supplement their primary income. Some state that Turking is fun and gives them joy, though it is debatable if they stick to Turking had it not be monetarily rewarding, albeit really less.

Earnings of a Turker

The annual earning is a highly variable figure from as low as $50 to as high as $15000. But it is dependent upon the number of hours put in by the Turker. The fair pay according to Turkers (on TurkOpticon) is largely according to the current minimum wage in US($7.25 per hour). But even $6/hour is not such a low rate for AMT in general. There are some HITs that pay a rate closer to $1 per hour. The reasons for choosing such a variable and minimal wage is assumed as follows: In many cases their primary source of income is much less than the minimum wage and Turking helps them manage their bread and butter.

Turker-Requester Relation

Relationships are more than the sum process of doing HITs and receiving pay. Many posts show that direct, open, polite, and respectful communication is highly valued. An engaging requestor is appreciated. Turkers are thought of as having a significant amount of untrustworthy members among them. Though the views are highly negatively skewed, some turkers do resort to cheating and deception. mturkforum.com is misused, where answers, cheats, scripts and workarounds are posted. But work ethics are prevalent among Turkers, there are instances where Turkers accept the responsibility for genuinely rejected HITs

Worker perspective: Turkopticon

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

Requester perspective: Crowdsourcing User Studies with Mechanical Turk

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

Requester perspective: The Need for Standardization in Crowdsourcing

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

Both perspectives: A Plea to Amazon: Fix Mechanical Turk

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

Turkit: Algorithmic human computation using imperative programming style

TurKit

TurKit is a Java/JavaScript API for running iterative tasks on Mechanical Turk. One can safely re-execute TurKit programs without re-running costly side effects on Mechanical Turk, like creating new HITs, but still write your program in a straightforward imperative manner—there is no need to unravel the program into a state machine.

Iterative tasks versus parallel tasks

  • MTurk is capable of doing several similar tasks in parallel as well as iterative sequential tasks such as iterating on an image description.
  • Iteration is helpful because instead of making different workers do the same task and picking the best solution a worker can review and add to the task done by the previous turker leading to refined results at lower cost.
  • Merit of iterative tasks over parallel recursive tasks is also presented in the model of mClerk. This can be done by the composition of primitive human computation tasks such as soliciting content, voting, and improving content.

Multi-phase task decomposition

Complex human computation algorithms where workers build on each others’ work can be designed conveniently by TurKit. Applications: blurry text recognition, iterative writing.

Crash-and-rerun

  • Crash-and-rerun programming is the backbone of TurKit Script.
  • It is a method for allowing a script to be re-executed without re-running costly side-effecting functions. It generates imperative programs for the user and then crashes (forget their state) and reruns on request or automatically on polling.
  • The server preserves state between requests in a database. Turkit instead of storing the current state stores the trace leading to the current state of the program. This leads to the ability to modify programs between reruns to fix bugs.
  • In order to avoid the cost of rerunning human computation steps, we look up the previous result in the database. The programmer has control over whether a previously executed line is retrieved from the database or evaluated afresh. This control is primarily embodied in the TurKit Script language feature once.

Crashing versus blocking

  • Crashing is better than blocking because the program does not have to store all the states and wait for the blocking function to return.
  • Storing all the states can be cumbersome in tasks involving many workers and tasks.
  • Crashing stores selective states and reruns other steps much like polling.
  • Blocking is not very useful when parallel tasks are being assigned.

TurKit script

  • TurKit provides a thin wrapper around these basic features, and also provides crucial higher-level calls not part of the MTurk API. The most important of these functions is waitForHIT,which allows a script to wait until a HIT is completed.
  • TurKit Script also provides several generally useful functions: prompt, vote, and sort. Supporting common subroutines helps make writing human computation algorithms easier.
  • Prompt function shows a string of text to a turker, and returns their response. Prompt takes an optional argument specifying a number of responses to be returned as an array.
  • Additionally, TurKit supports fork and join features for more easily implementing parallel algorithms.

Once function call

Once accepts a function as an argument, and calls this function. If it returns without crashing, then the return value is recorded in the database, and returned back to the caller. Subsequent runs return the recorded value without calling the function again. Once also records which function was passed to it so that it can ensure that the same function is passed again on subsequent runs of the program.The following type of calls should be listed in once :
  1. Non deterministic calls such as random functions should be ran with once so that they give the same value on rerun to render the whole function deterministic.
  2. Costly calls which involve human computation should also be called once and stored in the database for further runs
  3. If functions have side-effects, then it may be important to wrap them in once if invoking the side-effect multiple times will cause problems. For instance,approving results from a HIT multiple times causes an error from MTurk.

Limitations

The crash-and-rerun program will need to rerun all previous iterations of the loop every time re-executes, and eventually the space required to store this list of actions in the database will be too large.In some cases it is better to poll inside the script, rather than rerun it. But ,the model of polling inside the blocking script as opposed to rerunning the entire program does not allow the script to be modified between reruns. We can think of an approach where we don’t have to rerun the entire program but we can still change the script between reruns.

Browsing Blogs

  • Turkers have strong community forums where they discuss everything from popular HITs to fixing water coolers. These forums are a learning platform for beginners. Even experienced turkers look up to forums to get information about requesters and type of HITs.
  • Requesters often ask turkers about technical know-how of designing a task.
  • Requesters use these forums to analyse the needs and efficiency of the turkers.
  • Turkers decide reputation of a requester based on reviews on the forums.
  • Turkers outside of USA are willing to work but it is difficult for them to get approved.
  • Both turkers and requesters are not happy with the MTurk interface and standardization can be helpful.

requester turker blog

how to avoid scams

links worth turking for

turkalert

broken turk

Need synthesis

Worker Needs

Needs Of the workers are summarized as below:

  • Workers need to be need better- more intuitive and easy to use- task interfaces.

Evidence: Requestor-Question Blood Rayen lists out various interface needs that facilitate turkers like-

  1. Survey should have progress bar to no the remaining time.
  2. All qualification for filling any part of the survey need to be listed before hand.
  3. Turkers should be able to finish their HITs with minimum scrolling.

Interpretation: We need to standardize the interface used by turkers for tasks.

  • Workers need to be appreciated in form other than monetary benefits. They feel better to have done something of importance.

Evidence: [1] necoya says he would prefer remarks such as Good Job! in the feedback.

Interpretation: Turkers appreciate a closing sentiment acknowledging their contributions.

  • Workers need an easy and faster search mechanism for HITs as many turkers spend a lot of their time searching for appropriate HITs. This can be more useful for newbies. Mechanical Turk still does not have such functionality, a lot of APIs are coming up for the same but there is a lot of scope in the area.

Evidence: [2] this is an example of an API designed to cater to the mentioned. This [3] [4] lists various demands of the worker like :

  1. Workers want to differentiate spam tasks from authentic one
  2. Workers need to find task that suit their skill set easily.
  3. Workers need to find tasks with greater hit and completion rate.
  4. Workers need tasks that they can be complete with minimum disclosure of their personal information.

Interpretation: Workers can speed up their performance if they can efficiently search for tasks.

Requester Needs

Needs Of the Requester are summarized as below:

  • Requesters need to have tasks that require the worker to download software this widens the scope of type of tasks but MTurk currently prohibits this.

Evidence: As discussed in the hangout with requesters.

Interpretation: Requesters feel that the bad people are already asking turkers to download malware and the restriction affects only those with good will.

  • Requesters need more assurance of completion of the task.

Evidence: As discussed in the hangout.

Interpretation: There is no guarantee that if a turker accepts a HIT he/she will complete it correctly. At times such unpredictability can lead to losses.

  • Requesters need easy to use interface to design tasks which help them write more complex human computation algorithms with ease without much expertise. [5] [6] [7]
  • Requesters need to appropriately set time and amount to be paid for a task. If the time assigned for a task is too less it can lead to unnecessary rejection of task. Similarly, appropriate money paid enhances chances quality participation.