Social bookmarks as traces left behind as navigational signposts

Social tagging arose out of the need to organize found content that is worth revisiting. It is natural therefore to think of social tagging and bookmarking as navigational signposts for interesting content. The collective behavior of users who tagged contents seems to offer a good basis for exploratory search interfaces, even for users who are not using social bookmarking sites.

In Boston at the CHI2009 conference, we presented a paper that showed how our tag-based search interface called MrTaggy can be used as learning tools for people to find content relating to a particular topic. We have already announced its availability on this blog, and also touched upon the way in which it is implemented. Here we will briefly blog about an evaluation study we did on this system in order to understand its learning effects.

Short Story:

The tag-based search system allows users to utilize relevance feedback on tags to indicate their interest in various topics, enabling rapid exploration of the topic space. It turns out that the experiment shows that the system seems to provide a kind of scaffold for users to learn new topics.

Long Story:

We recently completed a 30-subject study of MrTaggy [see reference below for full detail]. We compared the full exploratory MrTaggy interface to a baseline version of MrTaggy that only supported traditional query-based search.



We tested participants’ performance in three different topic domains.





The results show:

(1) Subjects using the MrTaggy full exploratory interface took advantage of the additional features provided by relevance feedback, without giving up their usual manual query typing behavior.



(2) For learning outcomes, subjects using the full exploratory system generally wrote summaries of higher quality compared to baseline system users.



(3) To also gauge learning outcomes, we asked subjects to generate keywords and input as many keywords as possible that were relevant to the topic domain in a certain time limit. Subjects using the exploratory system were generally able to generate more reasonable keywords than the baseline system users.

(4) Finally, other convergent measures show that they also spent more time on the learning tasks, and had a higher cognitive load. Taken together with the higher learning measure outcomes, the users appear to be more engaged in exploration than the participants using the baseline system.

Our findings regarding the use of our exploratory tag search system are promising. The empirical results show that subjects can effectively use data generated by social tagging as “navigational advice” in the learning domain.

The experimental results suggest that users’ explorations in unfamiliar topic areas are supported by the domain keyword recommendations presented in the related tags list and the opportunity for relevance feedback.

Since social search engines that depend on social cues rely on data quality and increasing coverage of the explorable web space, we expect that the constantly increasing popularity of social bookmarking services will improve social search browsers like MrTaggy. The results of this project point to the promise of social search to fulfill a need in providing navigational signposts to the best contents.


Reference:

Kammerer, Y., Nairn, R., Pirolli, P., and Chi, E. H. 2009. Signpost from the masses: learning effects in an exploratory social tag search browser. In Proceedings of the 27th international Conference on Human Factors in Computing Systems (Boston, MA, USA, April 04 - 09, 2009). CHI '09. ACM, New York, NY, 625-634.

ACM Link

Talk Slides

Game theory and Cooperation in Social Systems

It's almost 2am, but I have been thinking about a summary of a recent Nature paper I read while I was in Boston visiting MIT. I had picked up the article in MIT Tech Talk on a whim during a visit to the Stata Center where MIT's CSAIL laboratory is located.

This article helped me start thinking about the conundrum of:
- why there are so many people willing to spent so much time shuffling and passing links to other people?
- why people write Wikipedia articles when they can spend time doing other things?
- why do users tag photos and URLs when the majority of the benefit is for others to find these items more easily?
In short, why is it that entities in social systems cooperate, especially when the benefit to oneself is not entirely clear at all?

Turns out researchers of microbes have been thinking about some of these cooperation problems as well. "One of the perplexing questions raised by evolutionary theory is how cooperative behavior, which benefits other members of a species at a cost to the individual, came to exist." They have used yeast as a model for understanding what might be happening. Sucrose is not yeast's preferred food source, but some yeast cells will metabolize it when glucose is not available, but the sugar diffuse away, and other free-rider yeast cells (lazy bums!) then benefits from the sugar for free.

Well, if the sugar diffuse away completely, then there is no reason to be the 'cooperating' cell to spent all that energy to benefit others. It gets really interesting when the cooperating yeast cell have preferential access to, say, 1 percent of the sucrose they metabolize. This slight advantage not only allow for the cooperating cells to compete effectively against the cheaters, but also enable the entire yeast community to benefit from having sucrose as an alternative food source. Moreover, no matter what the starting numbers of yeast cells, they end up into an equilibrium state with just the right amount of cooperating cells and cheaters present after some evolutionary period. The MIT team used game theory to model this entire process, and showed why it works the way it does. Darn cool!

This got me thinking about agents in a social system sometimes behave in similar ways, and can be modeled using game theory. I'm sure some of this has already been done. This sort of study is common in behavioral economics, for example. But how does it apply direct in social web system modeling? How can it help explain, for example, the tagging behavior of users in flickr? Perhaps the little bit of benefit that the user gains from organizing photos that she owns or have found is enough to turn them into 'cooperating' agents, from whom other freeriders obtain benefit. Moreover, the idea could be used to model why there are just the right pareto-balance (and power-law distributed) of cooperating agents and freeriders in a social web system.

Reference:

Jeff Gore, Hyun Youk & Alexander van Oudenaarden.
Snowdrift game dynamics and facultative cheating in yeast.
Nature advance online publication 6 April 2009 | doi:10.1038/nature07921

Mapping the Contents in Wikipedia

Having just returned from CHI2009 conference on Human-Computer Interaction, many of the topics there focused on where and how people obtain their information, and how they make sense of it all. A recent research topic in our group is understanding how people are using Wikipedia for their information needs. One question that had constantly come up in our discussion around Wikipedia is what is exactly in it. We have so far done most of our analyses around edit patterns, but not so much analysis have gone into what do people write about? What topics are the most well-represented? Where topic areas have the most conflict?

In one of our recent CHI2009 papers, we explored this issue. Turns out that Wikipedia have these things called Categories, which people use to organize the content into a pseudo-hierarchy of topics. We devised a simple path-based algorithm for assigning articles to large top-level categories in an attempt to understand what topic areas are the most well-represented. The top level categories are:



Using our algorithm, the page "Albert Einstein" can be assigned to these top-level categories:


This mapping makes some intuitive sense. You can see that the impact Albert Einstein has made in various areas of our society such as science, philosophy, history, and religion. Using the same ideas and algorithm, we can now do this mapping for all of the pages in Wikipedia, and find out what top level categories have received the most representation. In other words, we can figure out the coverage of topic areas in Wikipedia.


(You may have to click on the graphic here to see it in more detail.)

We can see that the highest coverage has gone toward the top-level category of "culture and the arts" at 30%, followed by "people" 15%, "geography" 14%, "society and social science" 12%, and history at 11%. What's perhaps more interesting is understanding which ones of these categories have generated the most conflicts! We used the previously developed concept called Conflict Revision Count (CRC) in our CHI2007 paper, and showed which top level categories have the most conflicts:



In this figure, the categories are listed in order of the total amount of conflicts clockwise from "People". This means that People did receive the most amount of conflict, followed by Society and Social Sciences, etc. However, the percentages in each topic is normalized by the number of article-assignments in that topic. So the metric developed here can be interpreted as the amount of conflict in each topic that has been normalized by the size of the topic, which can be interpreted as the amount of contentious in articles of the topic.

"Religion" and "Philosophy" stand out as highly contentious despite having relatively few articles. Turns out that "philosophy" and "religion" have generated 28% of the conflicts contentious-ness each. This is despite the fact that they were only 1% and 2%, respectively, of the total distribution of topics as shown above.

Digging into religion more closely, we see that "Atheism" have generated the most conflict, followed by "Prem Rawat" -- the controversial Guru and religious leader, "Islam" and "Falun Gong".



Wikipedia is the 8th ranked website in the world, so it is clear that a lot of people get their information from Wikipedia. The surprising thing about Wikipedia is that it succeeded at all. Common sense would suggest that an encyclopedia in which anyone can edit anything they want would result in utter nonsense. What happened is exactly the opposite: Many users and groups have gotten together to make sense of complex topics and debate with each other about what information is the most relevant and interesting to be included. This helps with us keeping sane in this information world, because we now have a cheap and always accessible content on some of the most obscure content you might be interested in. At lunch today, we were all just wondering what countries have the lowest birth rate. Well, surprise!! Of course, there is a page for that, which we found using our iPhones.

The techniques we have developed here enable us to understand what content is available in Wikipedia and how various top level categories are covered, as well as the amount of controversy in each category.

There are of course many risks in using online content. However, we have been researching tools that might alleviate these concerns. For example, WikiDashboard is a tool that visualizes the social dynamics behind how an wiki article came into its current state. It shows the top editors of any Wikipedia page, and how much they have edited. It also can show the top articles that a user is interested in.

We are considering adding this capability to WikiDashboard, and would welcome your comments on the analysis and ideas here.

All web users can guide the content in Wikipedia by participating in it. If we realized that the existence of our society depends on the healthy discourse between different segments of the population, then we will see it not just as a source of conflict, but a source of healthy discussion that needs to occur in our world. By having these discussions in the open (with full social transparency), we can ensure all points of view are represented in this shared resource. Our responsibility is to ensure that the discussion and conflicts are healthy and productive.



Reference:
Kittur, A., Chi, E. H., and Suh, B. 2009. What's in Wikipedia?: Mapping Topics and Conflict using Socially Annotated Category Structure. In Proceedings of the 27th international Conference on Human Factors in Computing Systems (Boston, MA, USA, April 04 - 09, 2009). CHI '09. ACM, New York, NY, 1509-1512.

Two published studies of WikiDashboard show that transparency impacts perceived trustworthiness

First Study: CSCW2008 (Best Note Award!) [1]

At CSCW2008 conference about 4 months ago, we published a user study conducted using Amazon's Mechanical Turk showing how dashboards affects user's perception of trustworthiness in Wikipedia articles.

In that experiment, we designed nearly identical dashboards in which only a few elements are changed. We designed a visualization of the history information of Wikipedia articles that aggregates a number of trust-relevant metrics.



We developed high-trust and low-trust versions of the visualization by manipulating the following metrics:
• Percentage of words contributed by anonymous users. Anonymous users with low edit-counts often spam and commit vandalism [1].
• Whether the last edit was made by an anonymous user or by an established user with a large number of prior edits.
• Stability of the content (measured by changed words) in the last day, month, and year.
• Past editing activity. Displayed in graphical form were the number of article edits (blue), number of edits made to the discussion page of the article (yellow), and the number of reverts made to either page (red). Each graph was a mirror image of the other, and showed either early high stability with more recent low stability, or vice versa.

We also included a baseline condition, in which no visualization is used at all.



The results with Mechanical Turk users show that surfacing trust-relevant information had a dramatic impact on users’ perceived trustworthiness, holding constant the content itself. The effect was robust and unaffected by the quality and degree of controversy of the page. Trust could be impacted both positively and negatively. High-trust condition increased trustworthiness above baseline and low-trust condition decreased it below baseline. This result is obviously very encouraging for folks who are keeping score on the effects of transparency on trust.

These results suggest that the widespread distrust of wikis and other mutable social collaborative systems may be reduced by providing users with transparency into the stability of content and the history of contributors.

Second Study: CHI2009 lab study [2]

In this second lab study, we extended the first study by allowing users to fully interact with the live version of WikiDashboard, which provided visualizations of the actual article and the editor histories. Moreover, we used questions from prior credibility research to assess a larger set of trust metrics for both the WikiDashboard condition and the plain old Wikipedia interface with no visualizations. Another experimental condition was whether the article had been independently identified as being of skeptical credibility by the Wikipedia community (by the WikiProject Rational Skepticism page on Wikipedia).

Interestingly, the results here are consistent with the first study. Users who saw WikiDashboard increased their credibility judgments about articles that were both previously designated as Skeptical or Non-Skeptical.

In summary, it seems that both study suggest the presenting more transparent information increases the credibility of the article, no matter whether it is controversial/skeptical or not. This is logical, since if you're buying a car, you would expect to have all of the vehicle's history along with the price information. If you only had the price information, you'd be less likely to deal with that particular dealer. Transparency breeds trust.

Given the prevalent skepticism around Wikipedia's content, it seems that the studies suggest by presenting transparent visualization of the particular authoring history of an article can boost its credibility. This further suggests that some people don't trust Wikipedia simply because they desire more understanding of how the content came to be.

References:

[1] Kittur, A., Suh, B., and Chi, E. H. 2008. Can you ever trust a Wiki? Impacting perceived trustworthiness in Wikipeda. In Proceedings of the ACM 2008 Conference on Computer Supported Cooperative Work (San Diego, CA, USA, November 08 - 12, 2008). CSCW '08. ACM, New York, NY, 477-480. DOI= http://doi.acm.org/10.1145/1460563.1460639

[2] Pirolli, P., Wollny, E., and Suh, B. 2009. So you know you're getting the best possible information: a tool that increases Wikipedia credibility. In Proceedings of the 27th international Conference on Human Factors in Computing Systems (Boston, MA, USA, April 04 - 09, 2009). CHI '09. ACM, New York, NY, 1505-1508. DOI= http://doi.acm.org/10.1145/1518701.1518929

ASC presents 8 papers related to Web2.0 and Social Web Research

The entire ASC group is in Boston this week to present 8 papers at the ACM SIGCHI annual conference. The CHI conference is a well-known academic conference that is considered to be the most prestigious platform for presenting Human-Computer Interaction research. Attended by around 2000 researchers, the acceptance rate for papers are generally in the 14-20%, and thus highly competitive.

Our group is presenting the following papers during the following sessions:

Information Foraging: Tuesday, 9:00 AM - 10:30 AM

• An Elementary Social Information Foraging Model, Peter Pirolli
  
• Remembrance of Things Tagged: How Tagging Effort Affects Tag Production and Human Memory, Raluca Budiu, Peter Pirolli, Lichan Hong
  
• Signpost from the Masses: Learning Effects in an Exploratory Social Tag Search Browser, Yvonne Kammerer, Rowan Nairn, Peter Pirolli, Ed H. Chi
  

Studying Wikipedia: Wednesday, 11:30 AM - 1:00 PM

• So You Know You’re Getting the Best Possible Information: A Tool that Increases Wikipedia Credibility, Peter Pirolli, Evelin Wollny, Bongwon Suh
  
• What's in Wikipedia? Mapping Topics and Conflict Using Socially Annotated Category Structure, Aniket Kittur, Ed H. Chi, Bongwon Suh
  

Social Search and Sensemaking: Wednesday, 4:30 PM - 6:00 PM

• Annotate Once, Appear Anywhere: Collective Foraging for Snippets of Interest Using Paragraph Fingerprinting, Lichan Hong, Ed H. Chi
  
• With a Little Help from My Friends: Examining the Impact of Social Annotations in Sensemaking Tasks, Les Nelson, Christoph Held, Peter Pirolli, Lichan Hong, Diane Schiano, Ed H. Chi
  

Computer Mediated Communication 2: Thursday, 2.30pm - 4:00 PM

• Gregorio is also presenting this paper on work he did while at Penn State:
  Supporting Content and Process Common Ground in Computer-Supported Teamwork
  

If you're at the conference, please come see us!