Difference between revisions of "Winter Milestone 5 @niranga"

From crowdresearch
Jump to: navigation, search
(Brief introduction of the system)
(Blanked the page)
 
(10 intermediate revisions by the same user not shown)
Line 1: Line 1:
== System ==
 
  
'''Introducing the workers skill to the Boomerang Ranking'''
 
 
 
=== Brief introduction of the system ===
 
 
Boomerang, the rating system that uses in Daemo decide what are tasks that workers going to get in their future and what type of workers, requesters going to get for their task. Requesters' ratings of workers of tasks are used to give early access to workers that requester rates highly. However, Boomerang still can recommend a worker to the requester who are not familiar with the posted task because Boomerang doesn't consider the workers skill level related to that particular task when recommend.
 
 
For example:-
 
 
* Requester post two tasks to the platform (Image tagging, Translation)
 
 
* When the requester posts the tasks Boomerang recommend the higher rating workers first
 
 
* However, among those workers, there can be workers who are not familiar with those tasks because those workers previously have worked with the requester on some other type of tasks( Surveys, Bookkeeping, etc.)
 
 
* So, if the requester selects one of the higher rating worker not familiar with the task, there will be a quality issue.
 
 
=== How is the system solving critical problems ===
 
To motivate continued participation, Twitch provides both
 
instant and aggregated feedback to the user. An instant feedback display shows how many other users agreed via a
 
fadeout as the lock screen disappears (Figure 4) or how the
 
user’s contributions apply to the whole (Figure 5).
 
Aggregated data is also available via a web application,
 
allowing the user to explore all data that the system has
 
collected. For example, Figure 2 shows a human generated
 
map from the Census application.
 
To address security concerns, users are allowed to either
 
disable or keep their existing Android passcode while using
 
Twitch. If users do not wish to answer a question, they may
 
skip Twitch by selecting ‘Exit’ via the options menu. This
 
design decision has been made to encourage the user to give
 
Twitch an answer, which is usually faster than exiting.
 
Future designs could make it easier to skip a task, for
 
example through a swipe-up.
 
 
=== Introducing modules of the system ===
 
Below, we introduce the three main crowdsourcing
 
applications that Twitch supports. The first, Census,
 
attempts to capture local knowledge. The following two,
 
Image Voting and Structuring the Web, draw on creative
 
and topical expertise. These three applications are bundled
 
into one Android package, and each can be accessed
 
interchangeably through Twitch's settings menu.
 
 
=== Module 1: Census ===
 
 
==== Problem/Limitations ====
 
Despite progress in producing effective understanding of
 
static elements of our physical world — routes, businesses
 
and points of interest — we lack an understanding of
 
human activity. How busy is the corner cafe at 2pm on
 
Fridays? What time of day do businesspeople clear out of
 
the downtown district and get replaced by socializers?
 
Which neighborhoods keep high-energy activities going
 
until 11pm, and which ones become sleepy by 6pm? Users
 
could take advantage of this information to plan their
 
commutes, their social lives and their work.
 
 
==== Module preview ====
 
Existing crowdsourced techniques such as Foursquare are
 
too sparse to answer these kinds of questions: the answers
 
require at-the-moment, distributed human knowledge. We
 
envision that twitch crowdsourcing can help create a
 
human-centered equivalent of Google Street View, where a
 
user could browse typical crowd activity in an area. To do
 
so, we ask users to answer one of several questions about the world around them each time they unlock their phone.
 
Users can then browse the map they are helping create.
 
 
==== System details ====
 
Census is the default crowdsourcing task in Twitch. It
 
collects structured information about what people
 
experience around them. Each Census unlock screen
 
consists of four to six tiles (Figures 1 and 3), each task
 
centered around questions such as:
 
• How many people are around you?
 
• What kinds of attire are nearby people wearing?
 
• What are you currently doing?
 
• How much energy do you have right now?
 
While not exhaustive, these questions cover several types of
 
information that a local census might seek to provide. Two
 
of the four questions ask users about the people around
 
them, while the other two ask about users themselves; both
 
of which they are uniquely equipped to answer. Each
 
answer is represented graphically; for example, in case of
 
activities, users have icons for working, at home, eating,
 
travelling, socializing, or exercising.
 
To motivate continued engagement, Census provides two
 
modes of feedback. Instant feedback (Figure 4) is a brief
 
Android popup message that appears immediately after the
 
user makes a selection. It reports the percentage of
 
responses in the current time bin and location that agreed
 
with the user, then fades out within two seconds. It is
 
transparent to user input, so the user can begin interacting
 
with the phone even while it is visible. Aggregated report
 
allows Twitch users to see the cumulative effect of all
 
users’ behavior. The data is bucketed and visualized on a
 
map (Figure 2) on the Twitch homepage. Users can filter
 
the data based on activity type or time of day.
 
 
 
=== Module 2: Photo Ranking ===
 
 
==== Problem/Limitations ====
 
Beyond harnessing local observations via Census, we
 
wanted to demonstrate that twitch crowdsourcing could
 
support traditional crowdsourcing tasks such as image ranking (e.g., Matchin [17]). Needfinding interviews and
 
prototyping sessions with ten product design students at
 
Stanford University indicated that product designers not
 
only need photographs for their design mockups, but they
 
also enjoy looking at the photographs. Twitch harnesses
 
this interest to help rank photos and encourage contribution
 
of new photos.
 
 
==== Module details ====
 
Photo Ranking crowdsources a ranking of stock photos for
 
themes from a Creative Commons-licensed image library.
 
The Twitch task displays two images related to a theme
 
(e.g., Nature Panorama) per unlock and asks the user to
 
slide to select the one they prefer (Figure 1). Pairwise
 
ranking is considered faster and more accurate than rating
 
[17]. The application regularly updates with new photos.
 
Users can optionally contribute new photos to the database
 
by taking a photo instead of rating one. Contributed photos
 
must be relevant to the day’s photo theme, such as Nature
 
Panorama, Soccer, or Beautiful Trash. Contributing a photo
 
takes longer than the average Twitch task, but provides an
 
opportunity for motivated individuals to enter the
 
competition and get their photos rated.
 
Like with Census, users receive instant feedback through a
 
popup message to display how many other users agreed
 
with their selection. We envision a web interface where all
 
uploaded images can be browsed, downloaded and ranked.
 
This data can also connect to computer vision research by
 
providing high-quality images of object categories and
 
scenes to create better classifiers.
 
 
=== Module 3: Structuring the Web ===
 
 
==== Problem/Limitations ====
 
Search engines no longer only return documents — they
 
now aim to return direct answers [6,9]. However, despite
 
massive undertakings such as the Google Knowledge Graph
 
[36], Bing Satori [37] and Freebase [7], much of the
 
knowledge on the web remains unstructured and unavailable for interactive applications. For example,
 
searching for ‘Weird Al Yankovic born’ in a search engine
 
such as Google returns a direct result ‘1959’ drawn from
 
the knowledge base; however, searching for the equally
 
relevant ‘Weird Al Yankovic first song’, ‘Weird Al
 
Yankovic band members’, or ‘Weird Al Yankovic
 
bestselling album’ returns a long string of documents but no
 
direct answer, even though the answers are readily available
 
on the performer’s Wikipedia page.
 
 
==== Module preview ====
 
To enable direct answers, we need structured data that is
 
computer-readable. While crowdsourced undertakings such
 
as Freebase and dbPedia have captured much structured
 
data, they tend to only acquire high-level information and
 
do not have enough contributors to achieve significant
 
depth on any single entity. Likewise, while information
 
extraction systems such as ReVerb [14] automatically draw
 
such information from the text of the Wikipedia page, their
 
error rates are currently too high to trust. Crowdsourcing
 
can help such systems identify errors to improve future
 
accuracy [18]. Therefore, we apply twitch crowdsourcing to
 
produce both structured data for interactive applications and
 
training data for information extraction systems.
 
 
==== Module details ====
 
Contributors to online efforts are drawn to goals that allow
 
them to exhibit their unique expertise [2]. Thus, we allow
 
users to help create structured data for topics of interest.
 
The user can specify any topic on Wikipedia that they are
 
interested in or want to learn about, for example HCI, the
 
Godfather films, or their local city. To do so within a oneto-two
 
second time limit, we draw on mixed-initiative
 
information extraction systems (e.g., [18]) and ask users to
 
help vet automatic extractions.
 
When a user unlocks his or her phone, Structuring the Web
 
displays a high-confidence extraction generated using
 
ReVerb, and its source statement from the selected
 
Wikipedia page (Figure 1). The user indicates with one
 
swipe whether the extraction is correct with respect to the
 
statement. ReVerb produces an extraction in SubjectRelationship-Object
 
format: for example, if the source
 
statement is “Stanford University was founded in 1885 by
 
Leland Stanford as a memorial to their son”, ReVerb
 
returns {Stanford University}, {was founded in}, {1885}
 
and Twitch displays this structure. To minimize cognitive
 
load and time requirements, the application filters only
 
include short source sentences and uses color coding to
 
match extractions with the source text.
 
In Structuring the Web, the instant feedback upon accepting
 
an extraction shows the user their progress growing a
 
knowledge tree of verified facts (Figure 5). Rejecting an
 
extraction instead scrolls the user down the article as far as
 
their most recent extraction source, demonstrating the
 
user’s progress in processing the article. In the future, we
 
envision that search engines can utilize this data to answer a
 
wider range of factual queries.
 

Latest revision as of 23:34, 13 February 2016