WinterMilestone 1 Crayons

From crowdresearch
Revision as of 04:33, 16 January 2016 by Sachinsharma (Talk | contribs)

Jump to: navigation, search

Mechanical Turk Sandbox version

Review : Worker

What do you like about the system / What are its strengths?

  1. Each HIT has been provided with instructions which explains the task to be followed clearly, making the task to be performed achievable.
  2. Easy to seek tasks according to earnings that the tasks provide, which is the main incentive for almost all the workers pursuing such platforms.
  3. The worker dashboard is pretty useful for workers to track their progress.
  4. As far as I could work on the tasks, I found that the reward per HIT was fair when compared to the work to be done for that reward.

What do you think can be improved about the system?

  1. The parameters to filter the HITs need to have a broader classification criteria, besides the one already present, i.e., creation date,availability,reward amount, expiration date.
  2. Each HIT should have more informative statistics like the acceptance rate of the task by the requester.
  3. Search feature did not prove to be very helpful, since the HITs are not tagged or classified under specific categories depending on the task to be done.
  4. This platform is not available globally, which is its biggest drawback. Since much of the online workforce hail from countries other that US, the requesters are not getting best of their investment, as they are not getting the potential workforce they could have at the same cost.
  5. Sandbox worker, don't have access to transferable payment.
  6. Cannot comment on the result of HIT, the sandbox is only but a simulation and requester may not be looking at the work closely for accuracy as may be doing so on the alternative.
  7. User interface was utilitarian at its best, and did not provide a very welcoming look and feel, and did not prove to be very navigable. Upgrade to responsive design, even a mobile app/m site could do wonders for the platform.
Screenshot of the Mechanical Turk Sandbox Version.

Review : Requester

What do you like about the system / What are its strengths?

  1. The project(HIT) from the requester had proper categories, like data collection, sentiment, survey, tagging of an image.
  2. For purposes of simulation, decided to go with tagging of an image since this was a straight forward task.
  3. For creation of project, details regarding the project had to be filled in, which was easy to navigate through, and supports all essentials from a requesters perspective.(e.g: Details including Description of the task, reward on the assignment, time allotment to carry out the said assignment, expiration of the HIT)
  4. The above details are followed by a window which is the design of the HIT in the window for the worker, which is a useful feature for the requester to control the aesthetic appearance for the worker.
  5. Manage Feature : Batches of task of HIT could be managed using the MANAGE tab on the dashboard, HIT as a part of a batch or individually.
  6. Developer version had pre allocated amount of money, so didn't have to unnecessarily provide sensitive details for capital purposes.

What do you think can be improved about the system?

  1. Will help with an android app, for the purposes of managing the tasks, and its progress in terms of attempts on the HIT.
  2. Since this is a developer version, cannot gain the actual experience of getting crowd-source solution for a given task, which in itself could have its own disadvantages.

Here is the link of the CSV file containing the responsesMedia : results.csv



Mobile Works is a mobile based crowd sourcing platform. The aim is to allow the weaker section of the society to take advantage of the growing crowd source industry. Simple Optical Character Recognition (OCR) tasks are sent to different people via a mobile application. Usually the document is divided into smaller pieces which contain one or two characters. These pieces are then sent to crowd source workers through the mobile app. A very simple interface is used so as to allow the people to easily understand and perform tasks.

What do you like about the system / What are its strengths?

  1. It very well captures the fact that in developing countries, like India, mobile penetration is very high as compare to desktop penetration. Hence, such mobile bases platform allows a larger section to be a part of the crowd sourcing.
  2. Simple user interface ensures that even those who are not very educated or technically sound are able to understand and use it.
  3. Each task(piece of document) is sent to two persons, and the answer is accepted only when both the answer matches. Hence, no compromise with efficiency.
  4. To encourage good performers, the function to calculate the cost of respective tasks also include the past performance/efficiency rate of the worker.
  5. It is not very time consuming. Workers may perform the task while travelling or even at home which watching tv etc.

What do you think can be improved about the system?

  1. To include more than just OCR tasks, for example audio recognition, picture classification etc.
  2. There is a rapid growth in the number of smartphone users. They are now easily accessible to even lower section of the society. Smartphones bring with them larger screens , better quality, better hardware, location facility etc which can be utilized to assign more complex tasks to the workers.


Daemo is a crowd-sourcing platform employing community driven, open source approach to work. Right from its design to underlying principle, Daemo challenges the problems faced by currently popular system, like Amazon Turks.

What do you like about the system / what are its strengths?

  1. A democratic system of policy making where requesters, workers, and researchers have equal say.
  2. A transparent and incentive driven rating system, aimed at providing genuine rating.
  3. To employ task prototyping, improving quality of final results, and help request view things from workers perspective.
  4. Development of trust through feedback, and subsequent acknowledge of feedback via ratings, task improvement and task feeds
  5. Leaving room for performance improvement, by giving weight-age to latest rating.
  6. A cleaner interface, which is more welcoming from workers perspective.
  7. Hassle free registration process.
  8. Community driven approach.

What do you think can be improved about the system?

  1. Build an intelligent system that helps detect discrepancy in rating.
  2. Work parallely on developing negotiation policies for decision making.
  3. Work on larger datasets of requester and workers to design quality.

Flash Teams

Flash Teams are basically computationally guided team of crowd experts supported bu lightweight reproducible and scalable team structure. Interactive systems can manage and manipulate by expanding and shrinking the the team as per requirement. Foundry is an end user authoring platform and run-time manager and it also serve our purpose by enabling the formation of flash teams. Crowd-sourcing system coordinates large group of people to solve the problem that a single individual could not achieve at the same scale. Crowd-sourcing system work by organizing work into micro-tasks. Foundry and Flash Team enable crowd-sourcing of a broad class of goods including design prototyping, course development and film animation in half of the time of traditional self managed teams(as per the paper). The aim of this paper is to provide the value of team structure, modularity and coordinating mechanism but in way that automatically creates team at the cost of increasing flexibility of the crowd.

What do you like about the system / what are its strengths?

  • Flash team composition:
  1. Flash enable us to gather and coordinate paid experts form the crowd to compute complex and independent task quickly and reliably.
  2. Flask teams afford dynamic recruitment and co-ordination of on-demand expertise that is extremely difficult in offline scenarios.
  3. Flash teams can take advantage of timezone differences that can potentially allow them to carry on uninterruptedly for days or weeks.
  4. Flash teams can be combined to create new types of organization with completely flexible boundaries.
  5. The basic unit of flash team is Block. Each block is represent one or more experts performing performing a task.
  6. To avoid confusion or diffusion of responsibilities for each block , the project specifies DRI(Directly Responsible individual) to manage the task completion.
  • Runtime and Coordination:
  1. Users either author a team from scratch or use the existing team with or without change in its composition.
  2. Workers can easily be recruited from oDesk for the particular job.
  3. oDesk provides rating that makes filtering of high quality experts.
  4. Requesters are provided transparency of the whole process so that requester can provide feedback answer and question through the chat system.
  • Computationally enhanced flash teams:
  1. Modular composition of the flash teams i.e small flash teams can be combined to get the bigger organization.
  2. Path Search support for team formation or authoring.Planning algorithm to find set of blocks that connect from original input to final output and provide automatically if such path exist.

What do you think can be improved about the system?

  1. Flash team since this paper are limited to to napkin desing , one can work with different kind of flash teams.
  2. There can be coordination problem and can originate conflicts within the team itself. There could be conflict resolution section for continuous work flow.
  3. Foundry can improved to predict the the time for the completion of particular work by a flash team.
  4. There can also be system to admire the best DRI by accessing number of job of quality that had been completed under his/her guidance.
  5. There can also be flexibility of changing DRI form time to time within the flash team from one job to other for the best result.
  6. System can also be improved to show best flash team available for the tasks according to the previous work done(work/cost can be point).

Milestone Contributors

  • Ameen Mohammed Khan @ameenkhan07
  • Farheen Nilofer @farheen
  • Sachin Sharma @beingcooper
  • Sarah Masud @sarah
  • Shivam Rana @tminima