Summer Milestone 4 Micro Macro Analysis rcompton

From crowdresearch
Jump to: navigation, search

Micro-Macro distinction

For most of the discussion that has occurred in our meetings, there has been a split between how micro and macro tasks have been discussed. Typically, we are considering them different entities and are trying to build a system that is adaptable to both. However to build such a system there needs to be a good definition in what is different between the two. The best definition I have been able to come up with is in terms of milestones. A micro task would be simply a task that has only one milestone while a macro task is consisting of multiple milestones. This is assuming that milestones are essentially singular in nature, in that they are the simplest possible task. With that mindset, what I am proposing is that the distinction between the two is less strict and more so a continuous scale.

Issue with Current Systems

My main issue with many of the systems in current use is that they are very heavily tasking for the requestor. They need to first need to set a goal, then create a workflow to accomplish this task. This is more so with smaller tasks that need a lot of workers, but experts can help accomplish the work flow breakdown as they have experience accomplishing this task (once again, big assumption that "experts" are true experts). So how can requestors take advantage of the experts to help create the workflow toward the goal? This proposal will also address this issue.

Main Points of proposal

  1. A macro task is essentially multiple micro tasks within a workflow
  2. Requestors set goal and workers find workflow toward that goal

Work Flow

  1. Requestor sets their goal
  2. Work Flow is created for this goal
    1. If the requestor has the ability to do so, then they create the workflow
    2. Otherwise the requestor posts the task of creating the workflow out to experts within the realm of the goal (Issue: How to find experts and see if they are experts)
    3. Validation of the workflow is conducted
      1. Either that be the requestor evaluating the work of workers (but assumes the requestors has knowledge of how the workflow should be)
      2. Or the validation is carried out through additional experts (if experts are rare, then finding more would be slow and difficult)
  3. Implement first of the milestones
    1. Implement qualification task
      1. If requestor has knowledge of which qualification would be best, then they can implement it
      2. Otherwise they can put it to the experts to find a good one
      3. Qualification through either inter reliability checks (compare work to others) or compare to a ground truth (if answer is already known)
    2. Qualified workers begin main work on milestone
    3. Evaluation of work
      1. Either through automated inter reliability checks or validation from others
  4. Repeat this workflow for additional milestones
    1. Qualification tasks may be needed if additional workers can be added on at this point
    2. However qualification may not be needed for workers who have accomplished previous milestones (however milestones are expected to be very different and workers may vary in quality from one milestone to the next)
  5. Evaluation of meeting the requestor's goal

Strengths and Weaknesses

A system as such is assuming to be micro task heavy and this system is focused on using the crowd to break down the macro tasks into micro tasks in order to rely less so on the requestor having knowledge on the task itself. Its biggest strength would be less work on the requestor without sacrificing quality. This system would help the requestor find the people who know about their problem and can help them reach their goal. Some flaws with this system are mostly within the assumptions:

  • All tasks can be broken down
  • All milestone take the same effort and time
  • Experts exists for all problems
  • Experts can be found for all problems

Other possibility

Another possibility would be once the workflow has been created, workers are recruited for each milestone and do the work in parallel unless the output of another milestone is needed then work is haulted until it is ready. This system is based on Flash Teams (http://hci.stanford.edu/publications/2014/flashteams/flashteams-uist2014.pdf).

Short Version

Crowdsource everything and have the requestor's only requirement to post a goal