WinterMilestone 5 westcoastsfcr

From crowdresearch
Jump to: navigation, search

Deliverables

Task Authorship Method Section

Study Introduction

We begin by comparing the variance of workers responses between two different conditions of task authorship in order to find out whether worker's quality of responses depend upon the requester's quality of task authorship. The two conditions we will analyze are if all of the requesters are given templates versus if they are not given templates in which to author tasks. After describing the experimental design, designed to indicate in which condition the worker's responses are more homogeneous, we show how different workers responses vary across both conditions. Previous studies have used a similar method to study the laziness of workers, however this study will focus on the quality of work posted by requesters.

Study Method

This study's experiments reported in this paper were conducted using Mechanical Turk, a crowdsourcing Internet marketplace that outsources crowd work to workers. In this system, workers choose from task listings posted by requesters and requesters post tasks they need completed.

Experimental Design

This study presents workers with similar tasks under two different conditions imposed upon the requesters. Requesters will be given five different tasks to post. The tasks they are given will not provide any instructions on how to format the task. One group of requesters will be given a set of templates to use and the other group will not be given templates. The tasks given to both groups of requesters will be the same. Then there will be two groups of workers. Each group of workers will consist of workers with varying levels of experience with Mechanical Turk and similar sites. Also both groups of workers will contain enough workers for each task to be completed by multiple individuals. Each group of workers will complete tasks from one of the requester groups. So one group of workers will only complete tasks from the group of requesters without a template and the other group of workers will only complete tasks from the group of requesters with a template. The results of the workers will then be analyzed. Specifically we will look to see the consistency in the work produced from tasks posted with and without templates.

Measures

For this study we will compare the work produced for each task under both conditions, tasks posted with a template and tasks posted without a template. Each task between each condition will be completed multiple times by different works with varying levels of familiarity with Mechanical Turk and similar sites. This way we can see how answers vary both between each condition and between workers knowledge and familiarity with Mechanical Turk. Since each task was completed by multiple workers in each condition, we can analyze the difference in responses under each condition. We would expect less of a variance in response to questions posted with a template than those posted without a template.

Analysis

We want to analyze the difference in task results between the two conditions. Specifically we want to analyze the difference between requester's answers in each condition to see if there is less variance in task answers for the workers answer tasks that were using templates versus those who were not using templates. If there is a larger variance in answers between a particular group, we can see whether using a template or not made the variance in worker response decrease or not. And we could compare this variance to how different the responses were for the worker group's tasks who were answering questions of tasks without a template. If there was a significant difference in each group's variance in answers, then we could prove whether or not requester's quality influences worker quality.


Task Ranking Introduction 1

There are many aspects to task raking that need to be taken into consideration when developing a crowdsourcing site. One of these being that there need to be thoroughly thought out placement exams as well as some sort of metric that allows for workers to be accurately assigned tasks. Using placement exams would allow for workers to be properly assigned to tasks that fit their skill set while enhancing quality of work as well as the requester's experience. Many workers do not utilize their skill set, but instead try to maximize profits. But what if a task feed was implemented that gave workers faster access to HITs that utilized their strongest attributes. Many times people will go after a career that does not suit their talents, but instead either 1) maximizes their income, 2) gives them a false sense of fulfillment due the title they receive, or both (we need to find a study that backs this up). Now say a task feed used specific metrics as well as placement exams that are taken at initial sign up that help point workers in that right career path, or in the case of crowdsourcing, right hit path. Now some may argue that this won't work cause it will result in workers not maximizing their income, thus they'll leave. For that very reason this study proposes to prove that these metrics could accurately predict the tasks that are best suited for the workers, therefore providing requesters with a consistency of good quality results. This would give the requester a sense of trust and confidence in whatever worker picks up their task. If these requesters are guaranteed high quality work they should have no issue forking over a little extra cash per task if our results show that our method of task ranking almost always guarantees them high quality work.

Mock Abstract

Task Ranking Introduction 2

Contributors

@tgotfrid7, @alexstolzoff, @dmajeti