Milestone 0 Design

From crowdresearch
Jump to: navigation, search

This document is a collaboratively-edited "spec" for Milestone 0. Make it better!

The goal of Milestone 0 is to give requesters an idea of how to improve their task, and who to choose to complete it, in a way that holistically combines microtasks (short tasks) and macro tasks (long tasks).

How much?

After a requester describes the title of the task, we ask them to upload a dataset if they have one. If they’ve uploaded a dataset, we ask them to estimate how long each item will take and to indicate their availability??? (@ARichmondFuller). We then set the Milestone 0 limit to be the equivalent of ~10 minutes of work. Example: 2-minute task means 5 tasks is Milestone 0. If there is no dataset, we ask them to author a Milestone that is the smallest unit of work they can use to evaluate quality (e.g., a sketch if you’re asking for a logo, or your best portfolio item). If the requester cannot estimate the smallest unit of work or/ and the amount of time, they can refer to templates to estimate both of them. (Karolina)

They then set the maximum number of workers who do Milestone 0 before we show them.We recommend 3. They select the worker level they need (reputation), which guides who is able to submit to Milestone 0.

Launch!

Any sufficiently qualified individual via the reputation requirements can submit Milestone 0, first come first serve, up to the maximum the user chose. The task has an open feedback textbox to send suggestions to the requester about how to improve the task design.

  • @ARichmondFuller -- Possible issue with 1st come 1st serve: I'm wondering if workers will feel pressure to be online all the time to be able to "snag" the tasks they want to do. The tasks will always go to the workers sitting there waiting to pounce on a task. If we have a calendar of availability for people on the platform, could there be a way of allocate workers to tasks to keep the workers from feeling like they always have to be online to get the good work? I'm just thinking of how I felt when the comments in our Slack channel kept getting archived; I felt an ever present sense of panic when not online checking the channels. Also, what are the mechanisms for a worker to grab the task - submitting the completed work for Milestone 0 or just by indicating "this one's mine" somehow and then, in the case of Macro-tasks, taking a bit more time to polish up and submit Milestone 0.
  • Karolina -- suggestion to address ARF's concerns: a worker can subscribe to receive updates from the tasks to their level. So, when a new task for their abilities is posted, they receive an email alert, and can login and decide whether to pursue that job or not.
  • @claudiasaviaga -- For @ARichmondFuller's concerns: I imagine a possibility where the requester is not satisfied with milestone 0 results, i.e. if it's a logo design job, the design style of the M0 applicants/submitters weren't the type of style the requester was looking for. If this happens, could it be possible to allow the relaunch of M0 and also deliver updates to other workers who wanted to apply but couldn't because only 3 people were allowed??? Could this mitigate some of the pressure of being online all the time to able to "snag" the tasks they want to do?
  • Jsilver -- @ARichmondFuller: I think it's normal to feel some pressure. At the same time, let's remember that workers aim to work and clients aim to get some tasks done, sometimes immediately. In that regard, workers need to be online to be able to work on a task; if they don't, others will take up that task. There would be work for many: a group of workers won't be able to monopolize all available tasks since each of us only have 24 hours a day and could only do so much. As Karolina suggested, workers can get notifications via email, in-platform, and perhaps mobile/SMS (and have a mobile app) later on. Our platform should aim to facilitate job selection and completion. If we let tasks linger in our platform far longer than they should (uncompleted tasks), it might lose future clients.
  • Jsilver -- @clausaviaga: Sure, it could alleviate some of that pressure. In that case, I imagine the first come, first served approach would still be enforced. So if the first 3 workers failed to impress the client, the next 3 workers would then have their chance, and so on. However, the better alternative for a macrotask like logo design is using the selection approach (see below) where the client picks the best workers or those whose style fits the clients needs.... I imagine our platform wouldn't impose any restriction on workers to apply for a task (but the client would only focus on a certain group/early applicants only), unless we offer a feature where the client can set a task availability timer which the client can turn on or off. @ARichmondFuller asks how the payment would work for these 6 workers in the case that the first 3 do not work out. She sees benefit in allowing the requester to see more than just 3 at the outset and then selecting the 3 from a pool to submit Milestone0. So, anyone who's interested can say "pick me" or "I'm interested" - say 10 or so workers - and then the requester can select 3 (who will get paid) to submit milestone0.
  • Jsilver -- An alternative to the first come, first served approach is the selection approach: allowing clients to choose the suitable worker/s. This approach would be ideal for macrotasks.

Timeframe

Jsilver: Our platform should impose a time limit on both sides (workers and client). The client can set a time limit on his/her task. For example, the worker should be able to submit M0 within x minutes/hours/days, and the client should respond, hire, and pay just as quickly or within a specific time. Worker's responsiveness (speed of application, communication, and output submission) and client's responsiveness (speed of hiring, communication, and payment) would ideally be incorporated into their respective reputations and stored as part of historical data we could use later on.

Feedback

When Milestone 0 completes, we notify the requester.They look at the results. If this will be a large-scale (many person) task, they will likely want to make changes to the task design in response. If it's a small-scale task, it makes more sense to just message the workers and talk with them. Following this, the requester can either relaunch Milestone 0 (to iterate) or author the next milestone. Authoring the next milestone means qualifying some or all of the workers who did Milestone 0 to go ahead --- in other words, picking who you'll go ahead with which will also trigger a payment reminder to those who completed Milestone 0 (ARF).

Future work

Milestone 0 results and task design templates (Jsilver) can be used to create training data for onboarding new workers once the task launches to everyone.