A Study on Efficient Diffusion of News in an Organization

[joint work between Les Nelson, Rowan Nairn, Ed H. Chi]

In our knowledge economy, enterprises’ competitiveness often depend on the efficiency in which important news travels to the right people at the right times. Knowledge workers depend now heavily on communication channels both inside and outside the enterprise to be kept up to date on the most important information, such as the latest news on competitors, memos on human resources, status of business proposals, and the progress of workflows. The efficiency of news spread in an organization determines not just how the organization might absorb and make sense of the information, but also how it might decide to respond and react.

For example, one study of how email impacts an organization showed that one piece of email may create an organizational footprint that is 30 times larger [1]. A large body of literature surrounds the issue of news flow in organizations, including information seeking, organizational memory, and expertise location. For example, more specific to organizational information flow, sociological research shows that there is greater homogeneity of information within groups of people than between groups of people [2].

News in general is about the communication of current events, where the timeliness of the information is key. ‘Timeliness’ might not necessarily be limited to just up-to-the-minute, ‘breaking’ news. For example, one interviewee in one of our studies recently said: “It's about the leading edge of something. Staying current in a professional sense, I go through bouts of finding information. And I share it”. In the organization, this may constitute keeping up with information for ‘knowing what’ is happening and ‘knowing how’ to do things.

How can organizations better respond to the complex social and technical situation involved in staying current in their areas of business? With respect to news at work, what roles, tools, and practices might we expect in the brokering of news?

INTERVIEW STUDY

We recently conducted an interview study within our research organization. The company is an established research organization, having approximately 200 staff members in one location. Most employees belong to an approximately 5- to 10-person group (we will call this a ‘team’) organized into 4 larger multi-team groups. Each employee has an office, generally located near the rest of his or her group.

The company uses wikis for project and group knowledge repositories. The project wikis typically receive brief but intense activity (e,g, collecting web links on a topic), and then lapse into occasional use. Group wikis are updated infrequently, usually when there is organizational change (e.g., new projects and people). External blogs on topic areas promoted by the company are encouraged. Internal blogs receive infrequent use for general information sharing on topics of wide interest. Microblogging (e.g., Yammer.com) was tried early, but did not persist.

Participants were chosen from a range of positions and tenure with the company, including staff members involved in the primary business production, service people in support of the staff members (e.g., marketing, administrators, staff services), managers, and executive level managers.

16 interviews were conducted in peoples’ offices, starting with a critical incident style interview on the most recent news events received, and then followed by explicit probing to elicit different ways in which news arrives, frequency of such news, and who was involved.

STUDY FINDINGS

We have found a relatively mature practice of relying on the communication channels most commonly used at work, such as email and face-to-face. News not only travel along social networks in the organization, but also there is a strong effort in passing along news that known to be relevant. People are conservative in their choices. Moreover, people tune their social network to ensure they receive the appropriate news.

We find three major ways the company responds to getting receiving and transmitting news:

(1) Email is indeed the channel and medium of choice for news [3];

The figure below shows the frequency in which various ways of passing news back and forth are mentioned in the interview study. Although we find that news arrives and is diffused by many channels, with different levels of timeliness and audience, the primary means of communication is email (either directly or via company mailing lists) and face-to-face conversations in offices, hallways, and at lunch.



(2) News follows peoples’ social/work networks, and there is a strong effort to pass along only news seen as relevant to others;

People filter news streams for their peers as a part of their ongoing conversations at work. The filtering includes quality assessments, time investment appropriate for relaying the news, uniqueness of the news:

One subject said to us,

"I have to read it [news related email] to find out if it is unique enough. I do try to filter if it is worth forwarding. There is a huge quality assessment thing, because I would hate clogging peoples’ streams. I would probably send it to people who are actually engaged in a conversation of this type."

(3) People structure their news networks to get news conveyed in short paths of only the ‘necessary, but sufficient’ recipients. They do this by structuring the channel so that it produces quality news, finding ways to avoid unnecessary communication, or setting up shortest paths.

For example, one subject said on who to follow in Twitter:

"I went through [lots of] phases. Imagine a spiral. I could overhear conversations and pick up derivative connections. Then it got to be a little overwhelming so I went and winnowed those down... and again. The people you follow dictate the information you get. And there were three factors. One is how informative or interesting they were to my interests. The second one was how frequently they updated. If they updated 50 times a day I couldn’t keep up with that. And the third reason is strategically, who I want to build a relationship with".

DESIGN IMPLICATIONS

We take from our findings above the following requirements for systems aimed at work news propagation:

1. Integrate into the email habitat to maximize chances of adoption;

2. Facilitate also putting news receivers in control. While email has its advantages, it is in some sense a sender-controlled system;

3. Allow targeting to continue but increase the chance of serendipitous but relevant connections in a way that keeps the social paths for news short and efficient;

4. Enhance the ability to target news to others without overloading email further;

5. Allow the emergence of shared interest spaces.


References

[1] IDC white paper "The Diverse and Exploding Digital Universe: An Updated Forecast of Worldwide Information Growth Through 2011", 2008.

[2] Burt, R. S. 2004. Structural holes and good ideas. American Journal of Sociology 110, 2 (September), 349–99.

[3] Ducheneaut, N. ; Bellotti, V. Email as habitat: An exploration of embedded personal information management. ACM Interactions. 2001 September-October; 8 (5): 30-38.

Technology Mediated Social Participation Workshop at PARC next week

Earlier this year PARC and Univ. of Maryland approached NSF with the idea of funding two workshops on Technology-Mediated Social Participation. NSF eagerly provided funding and simultaneously started a new program on Social-Computational Systems (SoCS).

Technology Mediated Social Participation
WEST COAST WORKSHOP & PANEL - Dec 10-11 @ PARC
With the goal of drawing up a strong scientific research agenda and educational recommendations necessary for a new era of social participation technologies, PARC is hosting the first of two workshops designed to bring together a diverse set of researchers from a variety of disciplines.

The West Coast Workshop will focus on three major themes:

• Integration of theory: from individual behavior to collective action
  
• Social intelligence and capital: understanding connections
  
• Research challenges: shareable infrastructure, ethics, and protection
  

In addition, Peter Pirolli and Jenny Preece will be hosting a special PARC Forum on Technology Mediated Social Participation on Thursday December 10, featuring panelists Ben Shneiderman, Amy Bruckman, Bernardo Huberman, and Cameron Marlow. The Forum will be streamed and recorded as well. We'll be live-testing/ soft launching the PARC Forum livestream at: http://www.justin.tv/parcinc

Check out extensive blog post by Peter here.

Twitter hashtag is #TMSP.

Workshop event page: Event - Technology Mediated Social Participation Workshop - PARC (Palo Alto Research Center)

Part 4 on WikiSym paper: A proposed modified model of Wikipedia Growth

As mentioned in the first post on the slowing growth rate of Wikipedia, it appears that article growth reached a peak around 2007. Rather than exponential growth, it appears that Wikipedia display logistic growth. A hypothetical logistic Lotka-Volterra population growth model bounded by a limit K is shown in the following Figure:


A hypothetical logistic Lotka-Volterra population growth model bounded by a limit K.

The above figure was generated by a Lotka-Volterra population model that assumes a resource limitation K. This K variable is known as the carrying capacity, which is the limit of the population growth. Translated into our case and using the articles as the stand-in for a population, this is the maximum number of articles that Wikipedia might reach eventually. This limit might be reached because knowledge below a threshold of notability are not eligible to become an encyclopedia entry, or that there are no one around in the community who knows enough about the subject to write it up.

In either case, according to this model, at the early stages of population growth the growth rate appears exponential, but the rate decelerates as it approaches the limit K. If the total amount of encyclopedic knowledge were some constant K, then the write-up of that knowledge into Wikipedia might be expected to follow a logistic such as this above Figure.

But there is a general sense that the stock of knowledge in the world is also growing. For instance, studies of scientific knowledge (e.g., [13][23]) suggest that it exhibits exponential growth. Also, events in the world (e.g., the election of Barack Obama or Lindsey Lohan’s rehabilitations) create new possibilities for write-up.

A possible modification to the logistic growth model is as follows: We suggest that if the total amount of knowledge exhibited some monotonic growth as a function of time, K(t), one might expect a variant of logistic growth as depicted in the Figure below:


A hypothetical Lotka-Volterra population growth model bound by a limit K(t) that itself grows as a function of time.

As originally recognized by Darwin in relation to the growth of biological systems [7], competition (the “struggle for existence”) increases as populations hit the limits of the ecology, and advantages go to members of the population that have competitive dominance over others. By analogy, we suggest that:

(a) that the population of Wikipedia editors is exhibiting a slowdown in its growth due to limited opportunities to make novel contributions, and

(b) the consequences of these (increasing) limitations in opportunities will manifest itself in increased patterns of conflict and dominance.

The limitations in opportunities might be the result of multiple and diverse constraints. For example, on one hand, we expect that the capacity parameter K is determined by limits that are internal to the Wikipedia community such as the number of available volunteers that can be coordinated together, physical hours that the editors can spend, and the level of their motivation for contributing and/or coordinating.

On the other hand, we expect that the capacity depends also on external factors such as the amount of public knowledge (available and relevant) that editors can easily forage and report on (e.g., content that are searchable on the web) and the properties of the tools that the editors and administrators are using (e.g., usability and functionalities).

In summary, globally, the number of active editors and the number of edits, both measured monthly, has stopped growing since the beginning of 2007. Moreover, the evidence suggests they follow a logistic growth function.

Our paper will finally be presented by Bongwon Suh at the WikiSym 2009 conference. The citation and link to the full paper is:
Bongwon Suh, Gregorio Convertino, Ed H. Chi, Peter Pirolli. The Singularity is Not Near: Slowing Growth of Wikipedia. In Proc. of WikiSym 2009, (to appear). Oct, 2009.

Thanks goes to my co-authors, who should receive equal credit for this research!

PART 3: Population Shifts in Wikipedia

The research done at ASC continues to get more press, including Time magazine, NYTimes, Repubblica [Italian Newspaper]. We have been busy trying to put together a bunch more academic papers on Web2.0 (particularly some Twitter research we have been doing), so we haven't updated this blog in a while. I figure today I'd take some time and blog a bit more about our results.

To investigate which factors affected the slowdown in edit growth, we examine the evolution of the population of active editors. The stalled growth of edit activities that we have described might be partially explained by changes in the editor population. We use the same editor classification as previous posts to count the number of active editors in each month. The figures below show three views of the evolution of the population of the five editor classes.



Monthly active editors by editor class. (This is a breakdown of the total editor population depicted earlier)

The Figure above shows the monthly frequencies of active editors by class. As expected from the power law distribution, the distribution of editors is very skewed: most of the editors contribute very few edits and very few editors contribute most of the edits. In fact, the two most prolific classes of editors (100-999 and 1000+) account for only about 1% of the population, but they contribute about 55% of edits (33% and 23% respectively).


Monthly active editors by user class. The vertical axis uses a logarithmic scale.


The Figure above uses a logarithmic scale to show the consistent slowdown of the growth among all editor classes over time, which is not clear in the first figure for editors in 100-999 and 1000+ classes. The monthly population of active editors stops growing after March 2007: a surprisingly abrupt change in the evolution of the Wikipedia population for all the editor classes. This change is consistent with the slowdown of the editing activity shown in Part 1.

[Interesting enough, even though we see that that the number of 1000+ class of editors plateaued, we know from Part 2 that this class of users have been increasing their contribution rate. Their average monthly edits per editor for the years 2005 to 2008 were 1740, 1859, 1869, and 2095, respectively.]


Percentages of monthly active editors by their class. Note that the graph is truncated to highlight the declining population of 10-99 editor classes [shown in purple]. (Sorry that the coloring of the editor classes is not consistent from the earlier plots.)

The last Figure shows the percentage of monthly active editors among the five classes. Note that the Y-axis is truncated: it omits the bottom 50% which represents the very long tail of once-monthly-editors. Notice how the 10-99 editor class [shown in purple] is being squeezed and becoming a small portion of the overall population. The 10-99 editor class went from 9% in 2005 to 6% in 2008.

A healthy community requires that people can move from novice contributors to occasional contributors to elite contributors. In other words, the upward mobility of the contributors is important for a healthy community. The trend here suggest that there are some resistance in moving beyond the 10-99 edits/month barrier. Could this be evidence of the Wiki-lawyering barriers?

One theory that I might suggest is that we want a well-balanced pyramid structure in the community population. Not too top heavy, and not too bottom heavy, and with a healthy middle class. How can we design the mechanisms [incentives and appropriate barriers] on the site so that we have this structure?

PART 2: More details of changing editor resistance in Wikipedia

In the last week, we have received interesting press coverage in New Scientist (as well as Fast Company, Business Insider, and syndicated elsewhere), on the work done in our team on Wikipedia growth rate, and how it has plateaued, changing from an exponential growth model to one that look more linear. Even though this wasn't necessarily new finding, but it was really a teaser for some other observations we have found in the Wikipedia data that is about to be published in WikiSym2009 conference in October.

In the figure below, we see how the slowdown in growth of Wikipedia activity, specifically around different editor classes is different. For each month, we first partition the editors into different classes based on their monthly editing frequency. We then compare the total edit activities among the different editor classes over time.


Monthly edits by user class (in thousands).


[Consistently with the power law, we classified users using an exponential scale: we defined the classes of editors using powers of 10, e.g. 10^0, 10^1, 10^2. This resulted in five classes of users for each month: editors contributing 1 edit (i.e., 10^0), 2 to 9 edits (2-9 class), 10 to 99 (10-99 class), 100 to 999 (100-999 class), and more that 1000 edits (1000+ class).] Note that the classification of the editors was recalculated for each month.

Since the beginning of 2007, the trends of four classes slightly decrease their monthly edits. In contrast, only the highest-frequency class of editors (1000+ edits, dark blue line) shows an increase in their monthly edits.

Another way to look at this data is to analyze the relative amount of activities for each editor class by transforming the data into percentages of the total edits. The figure below complements the information in the figure above by showing the percentage of the volume of edits that each class contributes in relation to the total.


Monthly percentage of edits by each user class.

The two highest frequency classes of editors account for more than half of the total monthly edits (56% from 01/2005 to 08/2008). Furthermore, since 2005 the proportion of contributions by the highest-frequency editor class has increased slightly. In fact, the editors in 1000+ class have kept producing at an increasing rate over the past four years (their average monthly edits per editor for the years 2005 to 2008 were 1740, 1859, 1869, and 2095, respectively).

We now focus on specific evidence about what might have contributed to such slowdown. Revert is the action of deleting a prior edit. The following figure shows the percentage of edits that were reverted (reverted edits) monthly for each editor class. Note that edits related to vandalism and edits performed by robots are excluded.


Monthly ratio of reverted edits by editor class

This illustrates two indicators of a growing resistance from the Wikipedia community to new content.

First, the figure shows that the total percentage of edits reverted increased steadily over the years. The total percentage of monthly reverted edits (see dashed black line) has steadily increased over the years for the all classes of editors (e.g. 2.9, 4.2, 4.9, and 5.8 percent of all edits for 2005 through 2008 as shown by the dash line).

Second, more interestingly, low-frequency or occasional editors experience a visibly greater resistance compared to high-frequency editors [see the top two reddish lines, as compared to other lines]. The disparity of treatment of new edits from editors of different classes has been widening steadily over the years at the expense of low-frequency editors.

We consider this as evidence of growing resistance from the Wikipedia community to new content, especially when the edits come from occasional editors.

PART 1: The slowing growth of Wikipedia: some data, models, and explanations

In September of 2008, we blogged about a curious change in Wikipedia that we didn't know how to explain that we had known for a while, and the ASC group has been looking into understanding this change in the last 6-9 months or so. The change that we were curious about was that the growth rates of Wikipedia have slowed. We were not the only ones wondering about this change. The Economist (archived here), for example, wrote about it.

We are about to publish a paper in WikiSym 2009 on this topic, and I thought we should start to blog about what we found.


Monthly edits and identified revert activity

The conventional wisdom about many Web-related growth processes is that they're fundamentally exponential in nature. That is, if you want some fixed amount of time, the content size and number of participants will double. Indeed, prior research on Wikipedia has characterized the growth in content and editors as being fundamentally exponential in nature. Some have claimed that Wikipedia article growth is exponential because there is an exponential growth in the number of editors contributing to Wikipedia [1]. Current research show that Wikipedia growth rate has slowed, and has in fact plateaued (See figure at right). Since about March of 2007, the growth pattern is clearly not exponential. What has changed, and how should we modify our thinking about how Wikipedia works? Prior research had assumed Wikipedia works on a "edit begets edit" model (That is, a preferential attachment model where the more an article gets edits, the more likely it would receive more edits, and thus resulting in exponential growth [2].) Such a model does not preclude some ultimate limitation to growth, although at the time it was presented [2] there was an apparent trend of unconstrained article growth.


Monthly active editor - number of users who have edited at least once in that month


The number of active editors show exactly the same pattern. The 2nd figure on the right shows how since its peak in March 2007 (820,532), the number of monthly active editors in Wikipedia has been fluctuating between 650,000 and 810,000. This finding suggests that the conclusion in [1][2] may not be valid anymore. We have a different process going on in Wikipedia now.


Article growth per month in Wikipedia. Smoothed curves are growth rate predicted by logistic growth bounded at a maximum of 3, 3.5, and 4 million articles.

Some Wikipedians have modeled the recent data, and believe that a logistic model is a much better way to think about content growth. Figure here shows that article growth reached a peak in 2007-2008 and has been on the decline since then. This result is consistent with a growth processes that hits a constraint – for instance, due to resource limitations in systems. For example, microbes grown in culture will eventually stop duplicating when nutrients run out. Rather than exponential growth, such systems display logistic growth.

We will continue to blog about what we believe might be happening in the next few weeks, as we find time to summarize the results.

[1] Almeida, R.B.m, Mozafari, B., and Cho, J., On the evolution of Wikipedia. ICWSM 2007, Boulder, Co., 2007.
[2] Spinellis, D., and Panagiotis, L. The collaborative organizations of knowledge. Communications of the ACM, 51(8), 68-73, 2008.

Social attention and interactions are key to learning processes

I just finished reading a long article in the journal Science on how social factors are increasing recognized as extremely important in a new science on learning [1].

Learning is fundamentally a social activity, the article partially argued. "Social cues highlight what and when to learn." Meltzoff et al. summarize a whole slew of recent research that showed how young infants learn by imitation and copying others actions, and they build abstractions and models of others' behaviors. In fact,

"Children do not slavishly duplicate what they see but reenact a person’s goals and intentions. For example, suppose an adult tries to pull apart an object but his hand slips off the ends. Even at 18 months of age, infants can use the pattern of unsuccessful attempts to infer the unseen goal of another. They produce the goal that the adult was striving to achieve, not the unsuccessful attempts."

One point made in the article is how much the greater environment outside of school is becoming an important part of the ecology of learning.

"Elementary and secondary school educators are attempting to harness the intellectual curiosity and avid learning that occurs during natural social interaction. The emerging field of informal learning is based on the idea that informal settings are venues for a significant amount of childhood learning. Children spend nearly 80% of their waking hours outside of school. They learn at home; in community centers; in clubs; through the Internet; at museums, zoos, and aquariums; and through digital media and gaming."

Social learning, of course, is a major part of the social web. Wikipedia was designed to be an easy-to-use and freely available reference, and all of the social interactions offered by various online forums are rapidly becoming a part of the educational experience for secondary school pupils. I would argue, for example, that Wikipedia has done more for continuing education for all adult learners than any educational institution could have done by itself. ASC's research have purposefully been focused on learning and information access, instead of entertainment, because of our recognition of the importance of social factors in various kinds of learning.

As an example, social learning was explicitly part of the design of our SparTag.us prototype, which is now just being offered in limited beta software to Firefox users, was announced at the recent CHI2009 conference. It streams the annotations you make as you browse the web. The stream is collected into your notebook, and by default this stream of annotation is made available to anyone interested in it. This makes it possible to aggregate social attention later.


[1] Foundations for a New Science of Learning. A. N. Meltzoff, P. K. Kuhl, J. Movellan and T. J. Sejnowski. Science, 325 (5938), 284-288. [DOI: 10.1126/science.1175626].

[2] Photo: Alan Decker and the Machine Perception Lab, UC San Diego.

Historical Roots behind TagSearch and MrTaggy

Boing Boing recently covered our work on TagSearch algorithm, and the MrTaggy prototype. We built the prototype to show how ideas from search in the past are relevant in the new world of "social search".

Boing Boing published the following response after asking us about the historical roots of TagSearch algorithm and the MrTaggy UI:

Several pieces of earlier important PARC work inspired the TagSearch algorithm and MrTaggy's user interface and experience.

First, one of the most efficient ways of browsing and navigating toward a desired information space was illustrated by the pioneering research on Scatter/Gather, a collaborative project on large-scale document space navigation between amazing researchers such as Doug Cutting (of Lucene, Hadoop fame) and Jan Pedersen (chief scientist at AltaVista, Yahoo, Microsoft for search). The research done in early to mid 90s, showed how a textual clustering algorithm can be used to quickly divide up an information space (scatter step), ask the user to specify which subspaces they're interested in (gather step). By iterating over this process, one can very quickly narrow down to just the subset of information items they're interested in. Think of it as playing 20 questions with the computer.

Second, also around the mid-90s, an important information access theory was being developed at PARC in our research group called Information Foraging, which showed that you can mathematically model the way people seek information using the same ecological equations used to model how animals forage for food. We noticed that we can use information foraging ideas to model how people used Scatter/Gather to browse for information. It turns out that it was possible to predict how people use the information cues (which we called 'information scent') in each cluster to determine whether they were interested in the contents inside the cluster. It turns out that Scatter/Gather can be shown to be a very efficient way to communicate to the user the topic structure of a very large document collection. In other words, people learned the structure of the information space much more efficiently using Scatter/Gather interfaces.

I hope it is quite clear that the relevance feedback mechanisms are very much inspired by Scatter/Gather. The related tags communicate the topic structure of what's available in the collection. Through this process, we designed MrTaggy, hoping that it would be just as efficient as Scatter/Gather in communicating the topic structure of the space.

Third, our group had developed Information Scent algorithms and concepts to build real search and recommendation systems. These algorithms build upon earlier work on a human memory model called Spreading Activation. TagSearch algorithm uses similar concepts here. It constructs a kind of Bayesian modeling of the topic space using the tag co-occurrence patterns. TagSearch's algorithm owes its heart and soul in concepts in Spreading Activation, which helps us find documents that are related to certain tags, and vice versa.

Visualization used to improve team coordination

This past Thursday I spent some time at IBM Almaden research center to attend the NPUC conference, which focused on the future of software development. In computing, software development is one of the most energy intensive collaborations, and often requiring significant coordination. There are elements of competition thrown-in for good measure, and of course, everyone is working in the same workspace, which is often coordinated by version control software. Sounds quite like Wikipedia, doesn't it?

One of the interesting talks given at NPUC was Gina Venolia's talk on using visualizations to represent the structure of the code. This representation can be used individually to make sense of the system, as well as being used by a team to explain structure to others. As a map to the system, they help anchor conversations between developers by providing for an intermediate representation of the knowledge structure that they must share for effective coordination.

This is a fascinating area to think about how augmented social cognition ideas could provide for better tools for collaborative software development. For example:

* Each developer could get an color on the map. Overlaps between two developer can then be easily visualized to see areas where they need to coordinate in the past and in the future.

* Building up an understanding from the code of who is working with whom, annotations and comments made by one developer could be send over to another developer's map when code is checked into the system.

* Social analytic can be used to discover where developers are clashing with each other (like how we have discovered conflicts in Wikipedia).

Microsoft Research indeed has been thinking about awareness tools toward this direction. A project named FASTDash works to increase awareness between developers in software teams. Lots of exciting possibilities!

Live data again: WikiDashboard visualizes the editing patterns of 'David Rohde' case...

Yesterday, NYTimes finally broke the silence on the kidnapping of David S. Rohde by the Taliban. Turns out, Rohde had escaped, and that the news media finally reported the kidnapping since the publicity on the case would no longer be a bargaining chip for his captors. The NYTimes article showed how keeping this news off of Wikipedia was nearly impossible if it weren't for the coordinated effort of several administrators and Jimbo Wales himself.

WikiDashboard visualized this editing pattern directly. In the figure below, I've highlighted the various edit wars between the anonymous editors (97.106.51.95; 97.106.45.230; and 97.106.52.36, which are believed to be the same person) and some of the administrators such as Rjd0060 and MBisanz and the involvement of a robot XLinkBot. You can also see the huge attention on this article in the last week or so in the visualization.


Check out the editing history and the edit war in detail by reading the edit history.

All of this makes for a great way for us to announce that WikiDashboard now works on the live Wikipedia data again; Thanks to the heroic efforts of Bongwon Suh in my group. He figured out how to execute his SQL query in a quick way on the new DB server.

How social media, twitter, and blogs might change reading bias...

newspapers (Tehrān)
Originally uploaded to Flickr by birdfarm

Since May, ASC has had a lot of activities focused on understanding how Google Wave, Twitter, and other new social media is changing the way we consume news and respond to it. I just finished reading some really interesting articles and watching some videos of how people's behaviors seems to be changing.

First, on June 8th, there was a report that described how, because of the great variety of choices now people have in what they read online, readers now tend to choose news that only fit their view. The research, done by researchers at Ohio State, showed how students tend to seek out and spent time reading media articles that focus on points of views that fit their political ideologies. Students spent 36% more time reading articles that agreed with their points of view.

Perhaps this isn't too surprising, but it has a huge implication for the future of political discourse, since a healthy political debate can only happen when an educated populace is willing to spend time to consider both sides of the issue. This is why Wikipedia has a neutral point of view principle for all articles. The above news article further suggests that the students prefer blogs instead of traditional media outlets for their news. This supports the idea that they read blogs that cater to particular points of views. Moreover, 30% of those surveyed believed blogs are actually more accurate. If this is true, one question to consider is whether having a more balkanized news diet might further polarize the public opinion, and further erode healthy dialog that is necessary for the society to function.

On a more positive note, I also watched Clay Shirky's recent talk on how social media is changing political discourse, because it now enable for not just 1-to-1 (point to point, or telephone/telegram-like technology) or 1-to-many (TV, radio, etc). It now also enables many-to-many communication and coordination. He tells stories of how the Chinese citizens used social media to get out the word about the Sichuan earthquake. They told stories of the heartache as well as the discovery of the corruption of the officials who were responsible for the bad construction jobs on school buildings.

Social media does seem to have changed the speed, cost, and the ability of the public to communicate and coordinate with each other. Citizen journalism does seem like it might have the potential to tip the balance of power back to the people.

Ironically, after these two pieces of information, I'm trying to decide whether I want to feel happy or sad about the state of affairs. I need to spend more time thinking about the changes social media is bringing to the world.

Science2.0 and Collaboratories

I'm in Hong Kong on some personal business and have had some alone time to think about our research direction. One of the things we have been doing lately at PARC is understanding more about the past work on collaboration, and how it might be changed (or not) by Web2.0 design principles. We have been talking to Gary and Judy Olson, who are recognized experts in collaboration systems and models for large science remote laboratories formed by scientists across many institutions. These laboratories (called collaboratories by the Olsons) are a great way to understand what works and what doesn't work in the real world, when CSCW and distant collaboration technologies are put to the test and used in real everyday scientific work. These studies are interesting because they're real 'living laboratories' and scientists engaged in these collaborations because it is necessary to do real work.

Interestingly, one of the best articles that summarizes their work is written by Technology Review (found here). Studying more than 200 collaboratories, the Olsons found that there are a number of pre-requisites for successful collaboration:
- Make sure your research community is ready
- Tackle big questions
- Get each individual participant on board
- Gear up for major technical challenges
- Put enough resources into project management
- Establish a common vocabulary
- Patience, visionary planning and stable management.

What's perhaps most interesting about this list is the amount of common sense it contains, and how it would be impossible to escape these pre-requisites even in Web2.0 collaboration systems. I find it interesting intellectual exercise to apply these requirements to successful Web2.0 systems (such as Wikipedia, delicious, and digg) to see if they meet these requirements.

Social bookmarks as traces left behind as navigational signposts

Social tagging arose out of the need to organize found content that is worth revisiting. It is natural therefore to think of social tagging and bookmarking as navigational signposts for interesting content. The collective behavior of users who tagged contents seems to offer a good basis for exploratory search interfaces, even for users who are not using social bookmarking sites.

In Boston at the CHI2009 conference, we presented a paper that showed how our tag-based search interface called MrTaggy can be used as learning tools for people to find content relating to a particular topic. We have already announced its availability on this blog, and also touched upon the way in which it is implemented. Here we will briefly blog about an evaluation study we did on this system in order to understand its learning effects.

Short Story:

The tag-based search system allows users to utilize relevance feedback on tags to indicate their interest in various topics, enabling rapid exploration of the topic space. It turns out that the experiment shows that the system seems to provide a kind of scaffold for users to learn new topics.

Long Story:

We recently completed a 30-subject study of MrTaggy [see reference below for full detail]. We compared the full exploratory MrTaggy interface to a baseline version of MrTaggy that only supported traditional query-based search.



We tested participants’ performance in three different topic domains.





The results show:

(1) Subjects using the MrTaggy full exploratory interface took advantage of the additional features provided by relevance feedback, without giving up their usual manual query typing behavior.



(2) For learning outcomes, subjects using the full exploratory system generally wrote summaries of higher quality compared to baseline system users.



(3) To also gauge learning outcomes, we asked subjects to generate keywords and input as many keywords as possible that were relevant to the topic domain in a certain time limit. Subjects using the exploratory system were generally able to generate more reasonable keywords than the baseline system users.

(4) Finally, other convergent measures show that they also spent more time on the learning tasks, and had a higher cognitive load. Taken together with the higher learning measure outcomes, the users appear to be more engaged in exploration than the participants using the baseline system.

Our findings regarding the use of our exploratory tag search system are promising. The empirical results show that subjects can effectively use data generated by social tagging as “navigational advice” in the learning domain.

The experimental results suggest that users’ explorations in unfamiliar topic areas are supported by the domain keyword recommendations presented in the related tags list and the opportunity for relevance feedback.

Since social search engines that depend on social cues rely on data quality and increasing coverage of the explorable web space, we expect that the constantly increasing popularity of social bookmarking services will improve social search browsers like MrTaggy. The results of this project point to the promise of social search to fulfill a need in providing navigational signposts to the best contents.


Reference:

Kammerer, Y., Nairn, R., Pirolli, P., and Chi, E. H. 2009. Signpost from the masses: learning effects in an exploratory social tag search browser. In Proceedings of the 27th international Conference on Human Factors in Computing Systems (Boston, MA, USA, April 04 - 09, 2009). CHI '09. ACM, New York, NY, 625-634.

ACM Link

Talk Slides

Game theory and Cooperation in Social Systems

It's almost 2am, but I have been thinking about a summary of a recent Nature paper I read while I was in Boston visiting MIT. I had picked up the article in MIT Tech Talk on a whim during a visit to the Stata Center where MIT's CSAIL laboratory is located.

This article helped me start thinking about the conundrum of:
- why there are so many people willing to spent so much time shuffling and passing links to other people?
- why people write Wikipedia articles when they can spend time doing other things?
- why do users tag photos and URLs when the majority of the benefit is for others to find these items more easily?
In short, why is it that entities in social systems cooperate, especially when the benefit to oneself is not entirely clear at all?

Turns out researchers of microbes have been thinking about some of these cooperation problems as well. "One of the perplexing questions raised by evolutionary theory is how cooperative behavior, which benefits other members of a species at a cost to the individual, came to exist." They have used yeast as a model for understanding what might be happening. Sucrose is not yeast's preferred food source, but some yeast cells will metabolize it when glucose is not available, but the sugar diffuse away, and other free-rider yeast cells (lazy bums!) then benefits from the sugar for free.

Well, if the sugar diffuse away completely, then there is no reason to be the 'cooperating' cell to spent all that energy to benefit others. It gets really interesting when the cooperating yeast cell have preferential access to, say, 1 percent of the sucrose they metabolize. This slight advantage not only allow for the cooperating cells to compete effectively against the cheaters, but also enable the entire yeast community to benefit from having sucrose as an alternative food source. Moreover, no matter what the starting numbers of yeast cells, they end up into an equilibrium state with just the right amount of cooperating cells and cheaters present after some evolutionary period. The MIT team used game theory to model this entire process, and showed why it works the way it does. Darn cool!

This got me thinking about agents in a social system sometimes behave in similar ways, and can be modeled using game theory. I'm sure some of this has already been done. This sort of study is common in behavioral economics, for example. But how does it apply direct in social web system modeling? How can it help explain, for example, the tagging behavior of users in flickr? Perhaps the little bit of benefit that the user gains from organizing photos that she owns or have found is enough to turn them into 'cooperating' agents, from whom other freeriders obtain benefit. Moreover, the idea could be used to model why there are just the right pareto-balance (and power-law distributed) of cooperating agents and freeriders in a social web system.

Reference:

Jeff Gore, Hyun Youk & Alexander van Oudenaarden.
Snowdrift game dynamics and facultative cheating in yeast.
Nature advance online publication 6 April 2009 | doi:10.1038/nature07921

Mapping the Contents in Wikipedia

Having just returned from CHI2009 conference on Human-Computer Interaction, many of the topics there focused on where and how people obtain their information, and how they make sense of it all. A recent research topic in our group is understanding how people are using Wikipedia for their information needs. One question that had constantly come up in our discussion around Wikipedia is what is exactly in it. We have so far done most of our analyses around edit patterns, but not so much analysis have gone into what do people write about? What topics are the most well-represented? Where topic areas have the most conflict?

In one of our recent CHI2009 papers, we explored this issue. Turns out that Wikipedia have these things called Categories, which people use to organize the content into a pseudo-hierarchy of topics. We devised a simple path-based algorithm for assigning articles to large top-level categories in an attempt to understand what topic areas are the most well-represented. The top level categories are:



Using our algorithm, the page "Albert Einstein" can be assigned to these top-level categories:


This mapping makes some intuitive sense. You can see that the impact Albert Einstein has made in various areas of our society such as science, philosophy, history, and religion. Using the same ideas and algorithm, we can now do this mapping for all of the pages in Wikipedia, and find out what top level categories have received the most representation. In other words, we can figure out the coverage of topic areas in Wikipedia.


(You may have to click on the graphic here to see it in more detail.)

We can see that the highest coverage has gone toward the top-level category of "culture and the arts" at 30%, followed by "people" 15%, "geography" 14%, "society and social science" 12%, and history at 11%. What's perhaps more interesting is understanding which ones of these categories have generated the most conflicts! We used the previously developed concept called Conflict Revision Count (CRC) in our CHI2007 paper, and showed which top level categories have the most conflicts:



In this figure, the categories are listed in order of the total amount of conflicts clockwise from "People". This means that People did receive the most amount of conflict, followed by Society and Social Sciences, etc. However, the percentages in each topic is normalized by the number of article-assignments in that topic. So the metric developed here can be interpreted as the amount of conflict in each topic that has been normalized by the size of the topic, which can be interpreted as the amount of contentious in articles of the topic.

"Religion" and "Philosophy" stand out as highly contentious despite having relatively few articles. Turns out that "philosophy" and "religion" have generated 28% of the conflicts contentious-ness each. This is despite the fact that they were only 1% and 2%, respectively, of the total distribution of topics as shown above.

Digging into religion more closely, we see that "Atheism" have generated the most conflict, followed by "Prem Rawat" -- the controversial Guru and religious leader, "Islam" and "Falun Gong".



Wikipedia is the 8th ranked website in the world, so it is clear that a lot of people get their information from Wikipedia. The surprising thing about Wikipedia is that it succeeded at all. Common sense would suggest that an encyclopedia in which anyone can edit anything they want would result in utter nonsense. What happened is exactly the opposite: Many users and groups have gotten together to make sense of complex topics and debate with each other about what information is the most relevant and interesting to be included. This helps with us keeping sane in this information world, because we now have a cheap and always accessible content on some of the most obscure content you might be interested in. At lunch today, we were all just wondering what countries have the lowest birth rate. Well, surprise!! Of course, there is a page for that, which we found using our iPhones.

The techniques we have developed here enable us to understand what content is available in Wikipedia and how various top level categories are covered, as well as the amount of controversy in each category.

There are of course many risks in using online content. However, we have been researching tools that might alleviate these concerns. For example, WikiDashboard is a tool that visualizes the social dynamics behind how an wiki article came into its current state. It shows the top editors of any Wikipedia page, and how much they have edited. It also can show the top articles that a user is interested in.

We are considering adding this capability to WikiDashboard, and would welcome your comments on the analysis and ideas here.

All web users can guide the content in Wikipedia by participating in it. If we realized that the existence of our society depends on the healthy discourse between different segments of the population, then we will see it not just as a source of conflict, but a source of healthy discussion that needs to occur in our world. By having these discussions in the open (with full social transparency), we can ensure all points of view are represented in this shared resource. Our responsibility is to ensure that the discussion and conflicts are healthy and productive.



Reference:
Kittur, A., Chi, E. H., and Suh, B. 2009. What's in Wikipedia?: Mapping Topics and Conflict using Socially Annotated Category Structure. In Proceedings of the 27th international Conference on Human Factors in Computing Systems (Boston, MA, USA, April 04 - 09, 2009). CHI '09. ACM, New York, NY, 1509-1512.

Two published studies of WikiDashboard show that transparency impacts perceived trustworthiness

First Study: CSCW2008 (Best Note Award!) [1]

At CSCW2008 conference about 4 months ago, we published a user study conducted using Amazon's Mechanical Turk showing how dashboards affects user's perception of trustworthiness in Wikipedia articles.

In that experiment, we designed nearly identical dashboards in which only a few elements are changed. We designed a visualization of the history information of Wikipedia articles that aggregates a number of trust-relevant metrics.



We developed high-trust and low-trust versions of the visualization by manipulating the following metrics:
• Percentage of words contributed by anonymous users. Anonymous users with low edit-counts often spam and commit vandalism [1].
• Whether the last edit was made by an anonymous user or by an established user with a large number of prior edits.
• Stability of the content (measured by changed words) in the last day, month, and year.
• Past editing activity. Displayed in graphical form were the number of article edits (blue), number of edits made to the discussion page of the article (yellow), and the number of reverts made to either page (red). Each graph was a mirror image of the other, and showed either early high stability with more recent low stability, or vice versa.

We also included a baseline condition, in which no visualization is used at all.



The results with Mechanical Turk users show that surfacing trust-relevant information had a dramatic impact on users’ perceived trustworthiness, holding constant the content itself. The effect was robust and unaffected by the quality and degree of controversy of the page. Trust could be impacted both positively and negatively. High-trust condition increased trustworthiness above baseline and low-trust condition decreased it below baseline. This result is obviously very encouraging for folks who are keeping score on the effects of transparency on trust.

These results suggest that the widespread distrust of wikis and other mutable social collaborative systems may be reduced by providing users with transparency into the stability of content and the history of contributors.

Second Study: CHI2009 lab study [2]

In this second lab study, we extended the first study by allowing users to fully interact with the live version of WikiDashboard, which provided visualizations of the actual article and the editor histories. Moreover, we used questions from prior credibility research to assess a larger set of trust metrics for both the WikiDashboard condition and the plain old Wikipedia interface with no visualizations. Another experimental condition was whether the article had been independently identified as being of skeptical credibility by the Wikipedia community (by the WikiProject Rational Skepticism page on Wikipedia).

Interestingly, the results here are consistent with the first study. Users who saw WikiDashboard increased their credibility judgments about articles that were both previously designated as Skeptical or Non-Skeptical.

In summary, it seems that both study suggest the presenting more transparent information increases the credibility of the article, no matter whether it is controversial/skeptical or not. This is logical, since if you're buying a car, you would expect to have all of the vehicle's history along with the price information. If you only had the price information, you'd be less likely to deal with that particular dealer. Transparency breeds trust.

Given the prevalent skepticism around Wikipedia's content, it seems that the studies suggest by presenting transparent visualization of the particular authoring history of an article can boost its credibility. This further suggests that some people don't trust Wikipedia simply because they desire more understanding of how the content came to be.

References:

[1] Kittur, A., Suh, B., and Chi, E. H. 2008. Can you ever trust a Wiki? Impacting perceived trustworthiness in Wikipeda. In Proceedings of the ACM 2008 Conference on Computer Supported Cooperative Work (San Diego, CA, USA, November 08 - 12, 2008). CSCW '08. ACM, New York, NY, 477-480. DOI= http://doi.acm.org/10.1145/1460563.1460639

[2] Pirolli, P., Wollny, E., and Suh, B. 2009. So you know you're getting the best possible information: a tool that increases Wikipedia credibility. In Proceedings of the 27th international Conference on Human Factors in Computing Systems (Boston, MA, USA, April 04 - 09, 2009). CHI '09. ACM, New York, NY, 1505-1508. DOI= http://doi.acm.org/10.1145/1518701.1518929

ASC presents 8 papers related to Web2.0 and Social Web Research

The entire ASC group is in Boston this week to present 8 papers at the ACM SIGCHI annual conference. The CHI conference is a well-known academic conference that is considered to be the most prestigious platform for presenting Human-Computer Interaction research. Attended by around 2000 researchers, the acceptance rate for papers are generally in the 14-20%, and thus highly competitive.

Our group is presenting the following papers during the following sessions:

Information Foraging: Tuesday, 9:00 AM - 10:30 AM

• An Elementary Social Information Foraging Model, Peter Pirolli
  
• Remembrance of Things Tagged: How Tagging Effort Affects Tag Production and Human Memory, Raluca Budiu, Peter Pirolli, Lichan Hong
  
• Signpost from the Masses: Learning Effects in an Exploratory Social Tag Search Browser, Yvonne Kammerer, Rowan Nairn, Peter Pirolli, Ed H. Chi
  

Studying Wikipedia: Wednesday, 11:30 AM - 1:00 PM

• So You Know You’re Getting the Best Possible Information: A Tool that Increases Wikipedia Credibility, Peter Pirolli, Evelin Wollny, Bongwon Suh
  
• What's in Wikipedia? Mapping Topics and Conflict Using Socially Annotated Category Structure, Aniket Kittur, Ed H. Chi, Bongwon Suh
  

Social Search and Sensemaking: Wednesday, 4:30 PM - 6:00 PM

• Annotate Once, Appear Anywhere: Collective Foraging for Snippets of Interest Using Paragraph Fingerprinting, Lichan Hong, Ed H. Chi
  
• With a Little Help from My Friends: Examining the Impact of Social Annotations in Sensemaking Tasks, Les Nelson, Christoph Held, Peter Pirolli, Lichan Hong, Diane Schiano, Ed H. Chi
  

Computer Mediated Communication 2: Thursday, 2.30pm - 4:00 PM

• Gregorio is also presenting this paper on work he did while at Penn State:
  Supporting Content and Process Common Ground in Computer-Supported Teamwork
  

If you're at the conference, please come see us!

How MrTaggy is implemented...

A short time ago, we announced the MrTaggy browsing and searching engine for social bookmarks here. One of the neat features of this system is its relevance feedback mechanism which enables users to click on keywords to navigate toward the information that they are interested in.

The overall system uses a sophisticated MapReduce computation in the backend, and the implementation is non-trivial. Here is how it works. The diagram below was recently published in an IEEE Computer Magazine article, and it roughly describes how the data flows thru the whole system. (Click on it to enlarge it.)



First, a crawling module goes out to the web and crawls social tagging sites, looking for tuples of the form . Tuples are stored in a MySQL database. In our current system, we have roughly 150 million tuples.

A MapReduce system based on Bayesian inference and spreading activation then computes the probability of each URL or tag being relevant given a particular combination of other tags and URLs. Here we first construct a bigraph between URLs and tags based on the tuples and then precompute spreading activation patterns across the graph.

To do this backend computation in massively parallel way, we used the MapReduce framework provided by Hadoop (hadoop.apache org). The results of this computation are stored in a Lucene index so that we can make the retrieval of spreading activation patterns as fast as possible.

Finally, a web server serves up the search results through an interactive frontend. The frontend responds to user interaction with relevance feedback arrows by communicating with the web server using AJAX techniques and animating the interface to an updated state.

Reference:
Ed H. Chi, "Information Seeking Can Be Social," IEEE Computer, vol. 42, no. 3, pp. 42-46, March, 2009.

Information Seeking can be Social: the potential for Social Search

As part of information seeking, exploratory search involves ill-structured problems and more open-ended goals, with persistent, opportunistic, iterative, multi-faceted processes aimed more at learning than answering a specific query [Marchionini 2006]. Whereas for the fact-retrieval searches, an optimal path to the document(s) containing the required information is crucial, learning and investigation activities lead to a more continuous and exploratory process with the knowledge acquired during this “journey” being essential as well [White et al. 2007]. Therefore, information seeking systems should focus on providing cues that might make these explorations more efficient.

One possible solution is building social information seeking systems, in which social search systems utilizes social cues provided by a large number of other people. What is social search? How might we build social search systems? Is there a need for such solutions?

Researchers and practitioners now use the term “social search” to describe search systems in which social interactions or information from social sources is engaged in some way [Evans and Chi 2008]. Current social search systems can be categorized into two general classes:

(a) Social answering systems utilize people with expertise or opinions to answer particular questions in a domain. Answerers could come from various levels of social proximity, including close friends and coworkers as well as the greater public. Yahoo! Answers (answers.yahoo.com) is one example of such systems. Early academic research includes Ackerman’s Answer Garden [Ackerman, 1996], and recent startups include Mechanical Zoo’s Aardvark (vark.com) and ChaCha’s mobile search (chacha.com).

Some systems utilize social networks to find friends or friends of friends to provide answers. Web users also use discussion forums, IM chat systems, or their favorite social networking systems like Facebook and Friendfeed to ask their social network for answers that are hard to find using traditional keyword-based systems. These systems differ in terms of their immediacy, size of the network, as well as support for expert finding.
http://www.blogger.com/img/blank.gif
Importantly, the effectiveness of these systems depends on the efficiency in which they utilize search and recommendation algorithms to return the most relevant past answers, allowing for better constructions of the knowledge base.

(b) Social feedback systems utilize social attention data to rank search results or information items. Feedback from users could be obtained either implicitly or explicitly. For example, social attention data could come from usage logs implicitly, or systems could explicitly ask users for votes, tags, and bookmarks. was one early example from early 2001 that used click data on search results to inform search ranking. The click data was gathered implicitly through the usage log. Others like Wikia Search (search.wikia.com), and most recently Google, are allowing users to explicitly vote for search results to directly influence the search rankings.

Vote-based systems are becoming more and more popular recently. Google’s original ranking algorithm PageRank could also be classified as an implicit voting system by essentially treating a hyperlink as a vote for the linked content. Social bookmarking systems such as delicious allow users to search their entire database for websites that match particular popular tags.

One problem with social cues is that the feedback given by people is inherently noisy. Finding patterns within such data becomes more and more difficult as the data size grows [Chi and Mytkowicz, 2008]

In both classes, there remains opportunity to apply more sophisticated statistical and structure-based analytics to improve search experience for social searchers. For example, expertise-finding algorithms could be applied to help find answerers who can provide higher-quality answers in social answering systems. Common patterns between question-and-answer pairs could be exploited to construct semantic relationships that could be used to construct inferences in question answering systems. Data mining algorithms could construct ontologies that are useful for browsing through the tags and bookmarked documents.

References
1. Ackerman, M. S.;McDonald, D. W. 1996. Answer Garden 2: merging organizational memory with collaborative help. In Proceedings of the 1996 ACM Conference on Computer Supported Cooperative Work (Boston, Massachusetts, United States, November 16 - 20, 1996). M. S. Ackerman, Ed. CSCW '96. ACM, New York, NY, 97-105. DOI= http://doi.acm.org/10.1145/240080.240203
2. Chi, E. H.; Mytkowicz, T. Understanding the efficiency of social tagging systems using information theory. Proceedings of the 19th ACM Conference on Hypertext and Hypermedia; 2008 June 19-21; Pittsburgh, PA. NY: ACM; 2008; 81-88.
3. Evans, B.; Chi, E. H. Towards a Model of Understanding Social Search. In Proc. of Computer-Supported Cooperative Work (CSCW). ACM Press, 2008. San Diego, CA.
4. Marchionini, G. Exploratory search: From finding to understanding. Communications of the ACM, 49, 4 (2006), 41-46.
5. White, R.W., Drucker, S.M., Marchionini, M., Hearst, M., schraefel, m.c. Exploratory search and HCI: designing and evaluating interfaces to support exploratory search interaction. Extended Abstracts CHI 2007, ACM Press (2007), 2877-2880.

Announcing MrTaggy.com: a Tag-based Exploration and Search System

I'm pleased to announce MrTaggy.com, a tag-based exploration and search system for bookmarked content on the Web. The tagline for the project is "An interactive guide to what's useful on the Web", since all of the content has been socially vetted (i.e. someone found it useful enough to bookmark it.)

MrTaggy is an experiment in web search and exploration built on top of a PARC algorithm called TagSearch. Think of MrTaggy as a cross between a search engine and a recommendation engine: it’s a web browsing guide constructed from social tagging data. We have collected about 150 million bookmarks from around the Web.

Unlike most search engines, MrTaggy doesn’t index the text on a web page. Instead, it leverages the knowledge contained in the tags that people add to web pages when using social bookmarking services. Tags describe both the content and context of a web page, and we use that information to deliver relevant contents.

The problem with using social tags is that they contain a lot of noise, because people often use different words to mean the same thing or the same words to mean different things. The TagSearch algorithm is part of our ongoing research to reduce the noise while amplifying the information signal from social tags.

We also designed a novel search UI to explore the tag space. The Related Tags sidebar outlines the content landscape to help you understand the space. The relevance feedback capabilities enable you to tell the system both positive and negative cues about directions where you want to go. Try clicking on the Thumbs Up and Down to give feedback to MrTaggy about the tags or results that you liked, and see how your rating changes the result set on-the-fly. At the top of the result set, we have also provided top search results from Yahoo's search engine when we think the results there might help you.

Enterprise Use

In addition to exploring TagSearch in the consumer space, we have also explored the use of TagSearch in the enterprise social tagging and intranet search systems. Surprisingly, the algorithm worked well even with a small amount of data (<50,000 bookmarks). For enterprise licensing of the underlying technology and API, contact Lawrence Lee, Director of Business Development, at lawrence.lee [at] parc [dot] com.

We would appreciate your feedback (comment on the blog here), or send them to mrtaggy [at] parc [dot] com, or submit at mrtaggy.uservoice.com.

Click here to try MrTaggy.com

WikiDashboard and the Living Laboratory

Our work on WikiDashboard was slashdotted last weekend. It caused our server to fail and crash repeatedly, and we tried our best to keep it running. We received thousands of hits, and got many comments. Interestingly, this occurred because of an MIT TechReview article on the system, which was in turn caused by the reporter coming to my talk at MIT last Tuesday (video here).

The whole experience is a very good example of the concept of the Living Laboratory. We were interested in engaging the real world in doing social computing research, and found Wikipedia to be a great way to get into the research, while benefiting the discourse around how knowledge bases should be built.

We had argued that Human-Computer Interaction (HCI) research have long moved beyond the evaluation setting of a single user sitting in front of a single desktop computer, yet many of our fundamentally held viewpoints about evaluation continues to be ruled by outdated biases derived from this legacy. We believe that we need to engage with real users in 'Living Laboratories', in which researchers either adopt or create functioning systems that are used in real settings. These new experimental platforms will greatly enable researchers to conduct evaluations that span many users, places, time, location, and social factors in ways that are unimaginable before.

Outdated Evaluative Assumptions

Indeed, the world has changed. Trends in social computing as well as ubiquitous computing had pushed us to consider research methodologies that are very different from the past. In many cases, we can no longer assume:

Only a single display: Users will pay attention to only one display and one computer. Much of fundamental HCI research methodology assumes the singular occupation of the user is the display in front of them. Of course, this is no longer true. Not only do many users already use multiple displays, they also use tiny displays on cell phones and iPods and peripheral displays. Matthews et al. studied the use of peripheral displays, focusing particularly on glance-ability, for example. Traditional HCI and psychological experiments typically force users to attend to only one display at a time, often neglecting the purpose of peripheral display designs.

Only knowledge work: Users are performing the task as part of some knowledge work. The problem with this assumption is that non-information oriented work, such as entertainment applications, social networking systems, are often done without explicit goals in mind. With the rise of Web2.0 applications and systems, users are often on social systems to kill time, learn the current status of friends, and to serendipitously discover what might capture their interests.

Isolated worker: Users performing some task by themselves. Much of knowledge work turn out to be quite collaborative, perhaps more so than first imagined. Traditional view of HCI assumed the construction of a single report by a single individual that is needed by a hierarchically organized firm. Generally speaking, we have come to view such assumption with contempt. Information work, especially work done by highly paid analysts, is highly collaborative. Only the highly automated tasks that are routine and mundane are done in relative isolation. Information workers excel at exception handling, which often require the collaboration of many departments in different parts of the organizational chart.

Stationary worker: User location placement is stationary, and the computing device is stationary. A mega-trend in information work is the speed and mobility in which work is done. Workers are geographically dispersed, making collaboration across geographical boundaries and time-zone critical. As part of this trend, work is often done on the move, in the air while disconnected. Moreover, situation awareness is often accomplished via email clients such as Blackberries and iPhones. Many estimates now suggest that already more people access the internet on their mobile phone than on desktop computers. This certainly has been the trend in Japan, a bellwether of mobile information needs.

Task duration is short: Users are engaged with applications in time scales measures in seconds and minutes. While information work can be divided and be composed of many slices of smaller chunks of subgoals that can be analyzed separately, we now realize that many user needs and work goals stretch over for long period of time. User interests in topics as diverse as from news on the latest technological gadgets to snow reports for snowboarding need to be supported over periods of days, weeks, months and even years. User engagement with web applications are often measured in much longer periods of time as compared to more traditional psychological experiments that geared toward understanding of hand-eye coordination in single desktop application performance. For example, Rowan and Mynatt studied peripheral family portraits in the digital home over a year-long period and discovered that behavior changed with the seasons (Rowan and Mynatt, 2005).

The above discussion point to how, as a field, HCI researchers have slowly broken out of the mold in which we were constrained. Increasingly, evaluations are often done in situations in which there are just too many uncontrolled conditions and variables. Artificially created environments such as in-lab studies are only capable of telling us behaviors in constrained situations. In order to understand how users behave in varied time and place, contexts and other situations, we need to systematically re-evaluate our research methodologies.

Time has come to do a great more deal of experimentation in the real world, using real and living laboratories.