WinterMilestone 1 @alanaut24

From crowdresearch
Jump to: navigation, search

Team Alanaut24




The proliferation of technologies aimed at shortening the digital divide between developed and developing countries, the rich and the poor, and technology beneficiaries is fundamentally changing how people connect and benefit from: remote work, crowd-sourced research, and online learning. This disruption in what was once a formidable brick and mortar approach to working has now opened up the doors allowing key stakeholders to learn and work with little to no regard to geographical location. Because of this, individuals now have the ability to choose where to work and for whom.

Similarly, organizations can now focus on sourcing quality talent from one of dozens of online crowdfunding platforms whose purposes varies from research to online learning. With this said, the benefits of crowd-sourcing does not come without limitations - many of which test pedagogical theories which stymie growth into verticals where traditional academia refuse to explore alternatives to classical education and research.

Note: Platform and system are used interchangeably throughout this Milestone.

Scope & Methodology

In choosing which platform to review, I chose to focus my initial efforts on platforms where I had little to no experience working with. These include, Amazon Mechanical Turk and CrowdFlower. The latter platforms, Odesk, Elance,, Guru, Upwork, and 99 Designs I use daily. For the sake of brevity as I tend to be verbose, I ultimately choose to review: Amazon Mechanical Turk and Upwork.

In undertaking the optimal way to present my findings, I choose to follow the Like / Dislike format as presented in the sample outline.

Structure of this Review

  • Introduction
  • Platforms I Tested
  • Comparison of AMT to Upwork / Elance
  • Readings

Platforms I Tested

Experience the life of a Worker / Requester on Amazon Mechanical Turk

I had never heard of Amazon Mechanical Turk (AMT) until I began participating in this Crowd Research Collective. I signed up for AMT earlier this week with no problems. When I began working on HIT's (Human Intelligence Tasks) I was informed via email that my account had been suspended. Unsure as to when the problem would be rectified, I signed up for a Sandbox account and got started. Note: A quick email to Customer Support resolved the issue within one day after which I began working on my first HIT after I had already earned well over $1.00.

My overall experience was great. I found the learning curve to be short and it was extremely easy to sign up for and complete HIT's.



  • Knowledge Management: Ease of disseminating static information aimed at sharing knowledge between two systems: Workers and Requesters.
  • Page load: The site was responsive to my inputs and each page loaded quickly. I used Load Impact AB to test the load rate of AMT's main dashboard.


  • Simplistic Interaction: Although cluttered, the system allowed me to quickly search for and sign up for different HIT's.
  • Visual Hierarchy: The platform presents a clear beginning, middle, and end. Although dated, I found it easy to navigate around the site.


  • Forms and Data Entry: It was extremely easy to input commands into. Titles, website navigation, and page navigation all worked as expected.
  • Trust and Credibility: Unlike other sites I have used, I had no concerns when signing up for and completing HIT's. Perhaps it was in large part do to my brief interaction with the system, but I would use this site again.



  • Multimodal Interface: Given the lack of infrastructure around implementing tools which allow multimodal interaction, the benefit would allow Requesters to better simulate real world scenarios users encounter in varying environments.
  • Lack of interaction: There exists very little in the way of Requesters and Workers interacting. Because of this, Amazon is missing out on valuable feedback which could areas where both the Requester and Worker's needs are not being met.


  • Context Awareness: Given when the system was designed and launched (circa 2005), it is understandable designing for mobile was not a priority. However, mobile phones are ubiquitous and there exists a clear gap in the market for designing platforms that are both environment and context of use ready. Furthermore, the ability to swipe, all, push, etc. opens up a broad areas of research once localized to facilities, labs, and research rooms.
  • Open User Innovation: There is a clear lack of innovating beyond the platform itself. The user interface allows for vary little variability in its use. Many of the greatest insights stem from direct user involvement in the platform. Amazon shows a clear lack of innovating AMT.


  • Task Orientation: After using the platform, I believe there exists opportunities to enhance how each user interacts and uses the system. In particular, when creating a Requester account, it was not immediately clear how what certain objects were used for.
  • Affordances: The platform was easy to use initially. However, AMT lacks clear visual properties of the interactions associated with objects users encounter.
  • Fluff: AMT is full of dead HIT's with unclear job requirements and instructions.

Recommendation for comparing to other websites: Conduct a Heuristic Design Usability review using a Radar Diagram.

Experience the life of a Worker on Elance / Upwork

I combined the two as I have more years of experience using Elance and Upwork adopted many of features into it's platform. My first experience as a freelancer working on a crowdfunding platform was via Elance which is now an Upwork (firmly Odesk) company. Before Elance merged with Odesk / Upwork, I had been working as a freelancer for 4 years. I have experienced the good and bad of using the platform. From below 1MBPS wifi speeds up to 100MBPS. On the slopes of Mt. Kilimanjaro to the Himalayas. I have searched, browsed, and navigated using mobile hot-spots in more than 40 countries around the world.

Below, you will find my Dislikes and Likes of using the platform.



  • Information Architecture: Tremendous thought went into the accessibility, organization, and navigation of the site. So much so that it allows users to build meaningful information relationships that ultimately enhance both freelancers and companies interaction of the site.
  • Customer Support: One of my favorite things about using Elance / Upwork is customer support. The platform provides three different mediums for contacting support: Chat, Phone, and email. I most often used Chat as it led to the fastest response time.


  • Sign Up: It is extremely easy to sign up for Elance / Upwork.
  • Aesthetic and minimalist design: Elance / Upwork moreso does an excellent job of creating invisible design. There exists minimal use of unnecessary dialogues which could detract from the users experience.


  • Forms and Data Entry: It was extremely easy to input commands into. Titles, website navigation, and page navigation all worked as expected.



  • System Failure: The Upwork / Elance system is constantly down or under maintenance.


  • Trust and Credibility: It doesn't take long after using Elance / Upwork that you find how naive the systems creators were about the feedback / review process. Upwork, in my opinion, is in a class all of its own where the users (up workers) have no direct benefit from using the platform. Clients often lie, threaten, and manipulate freelancers into doing work for virtually nothing at the expense of dangling a bad review over their heads. Furthermore, Upwork implemented a new scoring system called JSS (Job Success Score). I will not belabor on the JSS. If you would like to read more about this score and what I believe to be Upwork's eventual takeover by another, more transparent platform, just Google Upwork JSS or Upwork Job Success Score.

Comparison of AMT to Upwork / Elance

In terms of UI / UX, Elance / Upwork is light years ahead of AMT. Specifically, Elance / Upwork does an excellent job of aggregating data to support the end users needs. The platform itself is still in its infancy and I suspect over time, will become more intuitive of the users needs and wants.

My Readings


MobileWorks presents a Crowdfunding platform for workers at the bottom of the pyramid focused on OCR tasks.

Likes / Strengths

  • Initial concept. I like the idea of creating micro-tasks for users to earn supplemental income outside of their normal daily work load.

My thoughts and Improvements

  • Technology: Technology solutions for the BOP should be appropriate. I define appropriate technology according to the following 8 metrics: simplicity, independence, human-centered, sub-sidiarity, liberation, pedagogical, affordability, and non-violence. Our approach was adapted from years of human-centered design research and designing product and services for the bottom of the pyramid. The importance of this is such that: technology, although product focused, should serve as a platform to lift key stakeholders out of poverty.
  • Quantifying Cost: I think more research needs to go into the net benefit of using the proposed platform. My main concern is central to costs associated with using the system via phone carriers. Because data transfer rates and using the platform are mutually exclusive, I envision significant barriers to entry in getting users to use the platform.
  • Suggestion: Use two groups. Provide the first group data cards to supplement the cost of using the system. The second group, being the control, will use the platform as normal.


Daemo presents a crowdsourced crowdfunding platform.

Likes / Strengths

  • Initial concept. I like the idea of creating a more fair crowdsourced crowdfunding platform. Of all the papers, this one is of most importance to me. Having spent the past two years working on Elance / Upwork, the platform has undergone significant changes. Namely, the implementation of the new JSS.

My thoughts and Improvements

  • Requesters: It is our assumption that "Requesters" are domain experts. Why? In my experience, they are often not. I would say this is a major flaw (early in the stage) of the design of the platform and needs to be revisited. Most often, tasks are sub-delegated to those who may not be well versed in the subject domain they are wanting to test / elicit workers to complete.
  • Credibility of Feedback: One of the main pain points of the Upwork / Elance feedback system is the credibility of feedback left. People will inherently abuse a system for their benefit. Whether the worker identifies the general nature of how the system feedback exchange works and then hacks it to increase his score and secure better jobs or, the Requester identities ways to be dishonest about feedback, effectively sandwiching workers into completing more work for the same amount of money. These are two instances that exists within the current Upwork / Elance platform.
  • Suggestion: Another concern of mine with Dameo is how Requesters could rank middle quality, lower paying workers higher than high quality, middle paying workers. This is a common problem on many crowdfunding sites.It is nearly impossible to compete from a cost standpoint with these workers. What ends up happing is you have a highly skewed platform (like where a large percentage of workers are providing low - middle quality work (depending on the job of course) at a fraction of the price. An option to hire per geographic location could enhance the overall value proposition of each stream ( a stream being a geographic location to find workers). e.g. US to US, India to India, etc.
  • Time: I am unsure as to the length of time each requester will need to put into ensuring a "Boomerang" type feedback / review system results in superior quality of output. As it stands, getting workers / requesters to leave a review / feedback is often difficult. What incentives can we create to ensure this feature of the platform is a success?

Expert Crowd Sourcing with Flash Teams

Flash Teams presents a a framework for dynamically assembling and managing paid experts from the crowd.


  • Initial Concept. I like the idea of assembling a distributed team under the same umbrella (whether it be a company, organization, agency, etc.)

My thoughts

  • Domain Expertise: My main concern with this concept is each expert has a varying degree of experience and thus, could result in a solution that is not cohesive nor does it meet the end users needs.
  • I will use Design as an example. As it stands, the example provided in the paper (Figure 1) can be done by one person. Why is this important? The designer needs to be designing with the end user in mind. When you distribute work amongst a group of key stakeholders, you are working under the false premise that each worker / expert will design with the end user in mind.

My (half) Workday as a Turker

This post articulates the real world experience of being a Turker for half a day.


  • How Jeff breaks down his experience into 1 hour chunks. I also enjoyed his final analysis of using the platform.

My thoughts

  • Technology & People: As with any system of this magnitude, algorithms can only do so much. You must have a support team (be it: engineering, HR, service, etc.) to monitor the validity of posts, feedback, and users of the system. This is one of the main pitfalls of Upwork's implementation of their JSS feedback system rating - no one actually monitors the validity of the feedback being given.

Milestone Contributors

Slack usernames of all who helped create this wiki page submission: @alanaut24