Milestone 6 UWI
Implementations of Profile, Review Mechanism, and Workathon on Crowdsourcing Platform
This paper describes problems relating to empathy and trust with current crowdsourcing platforms and then discusses our approach toward building a system that solves these problems. The primary motivation behind building the system described in this paper is the frustration that workers and requesters face with current crowdsourcing platforms such as mTurk and oDesk. The system we introduce in this paper is a publicly available online User Profile for each worker. With social features such as profile pictures, names, followers, testimonials, etc, the system helps build empathy and trust with requesters. In addition to the User Profile, we also introduce a worker-requester meetup through workathons, where requesters can sit with workers as they perform tasks. We also discuss methods that we used to evaluate the system and proposed changes that came out of these evaluations.
By observing turkers and their working conditions in panels, research papers and in the forums they attend, we found one of the main problem in the crowdsourcing platform is that people do not trust each other.
Workers worried about whether they get paid fairly or on time or whether they would get a rejection reasonably. Requesters are not sure whether they will get their hits done completely and correctly and they don’t know whether they can get the workers that are qualified for their hits. Requesters do not trust workers and workers don’t feel that they are treated fairly and respectfully.
We think that why the trust are broken in this platform is that people don’t know each other and people can hardly talk to each other in this platform. Most of the users in this platform is anonymous and the platform requires little personal information to create an account. Since the AMT is a requester-oriented system, so workers don’t have a voice in this platform. It is hard for both requesters and workers to get feedback on their work. Moreover, they don’t feel empathetic with others, since workers don’t know how requesters work and vice versa.
It would have a significant negative effect on the quality and quantity of the results which requesters would get because when workers don’t trust the requesters, they just get the hits done and don’t check the results. On the other hand, requesters would keep rejecting the results since they don’t trust the workers. In the long term, the lack of trust in this platform will damage the confidence for both requesters and workers of working on this platform and keep new talents away from the platform.
Workers have made a lot of effort to build varieties of platforms and forums so that they can communicate and share information with others. They also contact requesters on the forums and provide suggestions about the design and the payment of the hits. In addition, they have built communities and do social networking on these forums.
However, these activities are far from enough for building relationships and trust among workers and requesters for several reasons. First, workers and requesters have to get used to each forum and network on them. There is no obvious connections between accounts on AMT and accounts on the forums, so people still don’t know who they are talking to and cannot build direct relationships with others. Second, as these forums don’t provide personal or professional information about the workers and requesters, people still cannot know about others and their skills and qualifications. Third, there is no way for people to directly leave comments or suggestions on a particular hit, a worker or a requester, so there are gaps between the actual hits or people and reviews about them left on the forums.
The ideas we generated is to assist workers and requesters to develop better relationship through both online or offline activities. For online activities, we designed user profile and review mechanism; so that, users can get to know each other and the task better. On the other hand, we designed Workathon event for workers and requesters to meet offline for having a direct interaction opportunity. All foundational ideas are complementary to each other; therefore, by using our design solution package, the relationship between workers and requesters can be enhanced in even higher degree. The existing method used on AMT for assisting communication between workers and requesters is allowing them to email each other; but obviously, it is not an effective way because both workers and requesters are lacking trusts and information of each other. Workers voluntarily initiated forum for them to share opinions and reviews. However, this platform is isolated from requesters.
All foundational ideas that we generated meet the user needs of empathy. To implement these foundational ideas, we designed several features.
The profile and review mechanism include features listed below: Name & Photo: this profile encourages a user to provide a photo and a username. The user doesn’t have to upload his/her real photo and provide a real name. But we encourage the user to do so by providing him/her with points. Whether the photo and name are real or not, this information would show real life personal interests, which will enable others to get to know this person better.
Location & Event Attended: this provides the location of the person. By clicking on the location the website will lead people to the “Meeting” section where the workers and requesters in this location organize in-person meetings. In the “Event Attended”, people can see the events this person has attended in this location.
Follower & Following: workers and requesters can follow each other and build relationships with each other. Requesters can invite the workers they follow to do their tasks. Workers can set notification for the new tasks from the requesters who they choose to follow.
Experience & Task History: The experience shows how long this person has worked on the platform and task history shows how active this person is. In the worker profile, stars will be added to those tasks that requesters invite workers to finish.
Rank and Points: the point system is shown on the right of the image. The points represent the reputation and working ability of this person. Once a worker has enough points, his/her rank will upgraded and he/she can access higher-level tasks. Workers in each rank can only access the task created by the requesters in the same or lower levels. If a requester has enough points, he/she can create more complex tasks. Entry-level requesters have to create test tasks for their tasks. The entry-level workers have to do then perform these test tasks. Test tasks will provide requesters and workers more points as well as prove their skills and abilities. For workers who are at a higher level and want to do entry-level tasks, they can choose to do test tasks or not. In addition to the hits list, the requesters name for the hits will be in different colors, representing different levels.
Achievement & Leaderboard: The rank shows the quality of one’s work and the achievement shows the quantity. Workers can get achievements when they have finished a certain number of tasks and requesters will get achievements when they have created a certain number of tasks. The system also provide leaderboards for each week, each month and each season; the top workers and requesters who get the highest points in a week, a month or a season will be on the leaderboards. People will also get achievements if they are on the leaderboards for a certain number of times. Skill: in the skill section, not only the person can a add skill tag for himself/herself and the requesters can add tags of the skills that this worker exhibited while completing their tasks. Requesters can also vouch for worker’s skills to prove that a worker has a particular skill. So it’s more convincing than just allowing workers to add their skills for themselves.
Testimonials: workers and requesters can leave reviews to each other if they have worked together before. In order to encourage people to leave reviews, a point will be given if they leave one review. In addition, the system only provides a “like” option for people if they really like others’ work but doesn’t provide a “dislike” button because we don’t want the workers and requesters to build bad relationships with each other. Other workers can learn about this person from others’ testimonials. Moreover, workers can leave reviews for every hit they have done and requesters and other workers can reply to these reviews.
The workathon activity is designed for allowing workers and requesters to meet offline. By meeting in person, workers can gain trust of requesters by showing their skills and talents; at the same time, requesters can not only control the time that workers spend on the tasks, which is 24 hours, but also evaluate a worker’s capability by seeing the work process. Each time when the platform administer announces a workathon event, workers and requesters who are willing to join needs to register for the event. During the registration, each requester need to fill out a survey to explain his/her task, including the type, description, payment, approximate time cost, etc.; workers need to fill out a survey to indicate the top three choices among all tasks. After the period of registration, the platform administer will match the workers and requesters. During the workathon, there are four sections for tasks, and workers can work for different requesters for each section; four social breaks is offered in between, providing opportunity of the communication for both workers and requesters.
In order to evaluate our system we used multiple techniques including surveys, interviews and usability testing. We created a set of metrics for workers and requesters so that we had something to compare against when our system was being tested. These metrics included: How fast can workers and requesters get accustomed to the point system? How easily can workers and requesters understand User Profiles and followers? Do the worker profile images build empathy with requesters? Do the Achievements and Leaderboards motivate workers? Do requesters trust the skill rating system? Do requesters actually trust the testimonials given by other requesters? How much do the above features affect the system? Do the workathons help requesters empathise with workers? Do requesters find trustworthy workers at workathons?
We interviewed workers and requesters based on these questions and recorded their answers. We found that experienced workers thoroughly enjoyed the points system and felt that this motivated them to work more and in turn earn more money. The amateur/new workers however felt that they were not being given enough recognition.
We also performed a set of usability studies aimed to test how easy it was for workers and requesters to use the system. We found that more than workers following other workers it was requesters that followed workers in order to see how often and how quickly they complete tasks. We even interviewed participants at the debrief of the usability tests and found similar results as earlier. Our survey was sent out to all users of the system and so the sample size was much larger. The survey results were interesting and showed that not many requesters actually take the time to write testimonials for workers. Workers expect more appreciation for their work and sometimes assume that they will receive a testimonial for a job well done. Unfortunately, requesters don't always have the time to write a testimonial and this can cause issues.
Based on these studies and tests, we have proposed some changes to the system including adding a “request testimonial” button for certain kinds of tasks. Also, results from our interviews show that the major problems of trust and empathy have been solved but these have in turn introduced other problems.
[Foundation Idea] http://crowdresearch.stanford.edu/w/index.php?title=Milestone_3_UWI [Feature Idea] http://crowdresearch.stanford.edu/w/index.php?title=Milestone_3_UWI_TrustIdea_1:_Profile http://crowdresearch.stanford.edu/w/index.php?title=Milestone_3_PixelPerfect_TrustIdea_2:_User_Rating_System http://crowdresearch.stanford.edu/w/index.php?title=Milestone_4_UWI_Empathy:_User_Profile http://crowdresearch.stanford.edu/w/index.php?title=Milestone_4_NotMathMajors_Empathy:_Workathons:_24_Hour_Working_Sprints http://crowdresearch.stanford.edu/w/index.php?title=Milestone_5_UWI_Storyboard:_Workathon http://crowdresearch.stanford.edu/w/index.php?title=Milestone_5_UWI_Mockup:_User_Profile