Milestone 8 taskforce Foundation1

From crowdresearch
Jump to: navigation, search

Submission template for Milestone 8. Create similar one's for all four foundations, each foundation will have its own page. All four pages needs to be submitted to meteor voting+commenting website.

Foundation 1: Micro+macrotask market

For example, "Foundation name" = "Foundation 3: External quality ratings".

Challenge question 1 (macrotasks, microtasks)

  • We would include both micro and macrotask projects in the crowdsourcing platform.
  • We could have macrotasks divided into sequences of microtasks (when feasible). The main problems that we see with this is that the delivery of responses will be slower and that the tasks would need a bigger effort in terms of detailed instructions, task qualification management and workforce training. Some ossible restrictions that would make the split into microtasks unfeasible are:
    • dependencies between subtasks. For example we can have a macro project about painting something or writing an essay. Workers need to work on top of the work of others. So, we would exchange people on the task but without paralelizing the tasks. The drawback would be the time that this requires to finalize.
    • requiring people (for the macrotasks) with deeper skills, methods to find the experts and coordinate them.

Challenge question 2 (how would these macrotasks work)

  • We would ask for second opinion to other experts (might be paid or any other mechanism might be considered). The experts will evaluate the deliverables of milestones on high level, without really debugging the outputs, and try to detect critical flaws.

Challenge question 3 (high quality)

  • accept the person and establish milestones / deliverables (as it was said in the hangout session) and include a test phase in which the requester can decide (after communication) whether he/she would like the worker to keep on doing the work.

when intermediate results are sent we can open them to a small audience (crowd / requesters / public) to receive feedback which could encourage the person more

Challenge question 4 (trust that they are experts)

  • pre-evaluation the person
    • type of test: cognition test, skill based test, domain based test, real questions
    • response processing and assessment: gold responses, cross validation
  • asking the person to show previous accomplishments (extended profiles)
    • information: work history, results (bad and good)
    • certification / endorsement: platform, requester
  • trust the expertise of teams: once you know you have individuals that are very much experts and other individuals that are not so expert, assemble a teams such that the team will outperform individuals (e.g. the non experts will learn from the experts).