Short and Tweet: Experiments on Recommending Content from Information Streams (Specifically, Twitter)

Information streams have recently emerged as a popular means for information awareness. By information streams we are referring to the general set of Web 2.0 feeds such as status updates on Twitter and Facebook, and news and entertainment in Google Reader or other RSS readers. More and more web users keep up with newest information through information streams. At the CHI2010 conference, we presented a new system called Zerozero88.com that recommends contents (particularly URLs that have been posted in Twitter) to users based on their profile on Twitter. Through recommender systems, we hope to better direct user attention to the most interesting URLs that are posted on Twitter that the user should pay attention to.

As a domain for recommendation, information streams have three interesting properties that distinguish them from other well-studied domains:

1. Recency of content: Content in the stream is often considered interesting only within a short time of first being published. As a result, the recommender may always be in a “cold start” situation, i.e. there is not enough data to generate a good recommendation.
2. Explicit interaction among users: Unlike other domains where users interact with the system as isolated individuals, with information stream users explicitly interact by subscribing to others’ streams or by sharing items.
3. User-generated content: Users are not passive consumers of content in information streams. People are often content producers as well as consumers.

In a modular approach, we explored three separate dimensions in designing such a recommender: content sources, topic interest models for users, and social voting:

1. Content Sources: Given limited access to tweets and processing capabilities, our first design question is how to select the most promising candidate set of URLs to consider for recommendations. We chose two strategies: First, Sarwar et al. [1] have shown that by considering only a small neighborhood of people around the end user, we can reduce the set of items to consider, and at the same time expect recommendations of similar or higher quality.
   
   Second, we also considered a popularity-based URL selection scheme. URLs that are posted all over Twitter are probably more interesting than those rarely mentioned by anyone.
   
   
2. Topic Modeling: Using topic relevance is an established approach to compute recommendations. The topic interest of a user is modeled from text content the user has interacted with before, and candidate items are ranked by how well they match the topic interest profile of the user. Another way to model the user's interest is by modeling the topics of the tweets made by the people she follows.
   
3. Social Voting: Assuming the user has a stable interest and follows people according to that interest, people in the neighborhood should be similar minded enough so that voting on the neighborhood can function effectively. However, the “one person, one vote” basis in the approach above may not be the best design choice in Twitter, because some people may be more trustworthy than others as information sources. Andersen et al. discussed several key insights in their theory of trust-based recommender systems [2], one of which is trust propagation. Intuitively, trust propagation means my trust in Alice will increase when the people whom I trust also show trust in Alice. Following this argument, a person who is followed by many of a user’s followees is more trustworthy as an information source, and thus should be granted more power in the voting process.
   
   

The figure below describes the overall design of the system. The URL Source selectors from the lower left are content items that feed into the system to be ranked. The left side of the system does the topic modeling, which can come from either the user's own tweets, or the followee's tweets. The social voting model is implemented using modules on the right.



We implemented 12 recommendation engines in the design space we formulated above, and deployed them to a recommender service on the web to gather feedback from real Twitter users. The best performing algorithm improved the percentage of interesting content to 72% from a baseline of 33%.

Overall, we found that:

1. The social voting process seems to contribute the most to the recommender accuracy.
2. The topic models also contribute to the accuracy, but modeling using the user's self tweets is more accurate (with the caveat that the user actually tweets, not merely listen by following people).
3. Selecting URLs based on the neighborhood seems to work better than globally popular URLs, but the results are not yet statistically significant.
4. The best performing algorithm is FoF-Self-Vote (that is, using the neighborhood for URL content sources, self-tweets for topic modeling, and social voting.)
   

You can try out the beta system at http://zerozero88.com, but since it is still in beta, we can probably only enable the accounts of a limited number of people who sign up.

You can also read more about our results in the published paper [3].

Update 2010-08-23: Slides available here.

References

[1] Sarwar, B.M., Karypis, G., Konstan, J.A., Riedl, J. 2002. Recommender systems for large-scale ECommerce: Scalable neighborhood formation using clustering. In Proc of ICCIT 2002.

[2] Andersen, R., Borgs, C., Chayes, J., Feige, U., Flaxman, A., Kalai, A., Mirrokni, V., and Tennenholtz, M. 2008. Trust-based recommendation systems: an axiomatic approach. In Proc of WWW ‘08.

[3] Chen, J., Nairn, R., Nelson, L., Bernstein, M., and Chi, E. 2010. Short and tweet: experiments on recommending content from information streams. In Proceedings of the 28th international Conference on Human Factors in Computing Systems (Atlanta, Georgia, USA, April 10 - 15, 2010). CHI '10. ACM, New York, NY, 1185-1194. DOI= http://doi.acm.org/10.1145/1753326.1753503

ASC presents 8 papers related to Web2.0 and Social Web Research

The entire ASC group is in Boston this week to present 8 papers at the ACM SIGCHI annual conference. The CHI conference is a well-known academic conference that is considered to be the most prestigious platform for presenting Human-Computer Interaction research. Attended by around 2000 researchers, the acceptance rate for papers are generally in the 14-20%, and thus highly competitive.

Our group is presenting the following papers during the following sessions:

Information Foraging: Tuesday, 9:00 AM - 10:30 AM

• An Elementary Social Information Foraging Model, Peter Pirolli
  
• Remembrance of Things Tagged: How Tagging Effort Affects Tag Production and Human Memory, Raluca Budiu, Peter Pirolli, Lichan Hong
  
• Signpost from the Masses: Learning Effects in an Exploratory Social Tag Search Browser, Yvonne Kammerer, Rowan Nairn, Peter Pirolli, Ed H. Chi
  

Studying Wikipedia: Wednesday, 11:30 AM - 1:00 PM

• So You Know You’re Getting the Best Possible Information: A Tool that Increases Wikipedia Credibility, Peter Pirolli, Evelin Wollny, Bongwon Suh
  
• What's in Wikipedia? Mapping Topics and Conflict Using Socially Annotated Category Structure, Aniket Kittur, Ed H. Chi, Bongwon Suh
  

Social Search and Sensemaking: Wednesday, 4:30 PM - 6:00 PM

• Annotate Once, Appear Anywhere: Collective Foraging for Snippets of Interest Using Paragraph Fingerprinting, Lichan Hong, Ed H. Chi
  
• With a Little Help from My Friends: Examining the Impact of Social Annotations in Sensemaking Tasks, Les Nelson, Christoph Held, Peter Pirolli, Lichan Hong, Diane Schiano, Ed H. Chi
  

Computer Mediated Communication 2: Thursday, 2.30pm - 4:00 PM

• Gregorio is also presenting this paper on work he did while at Penn State:
  Supporting Content and Process Common Ground in Computer-Supported Teamwork
  

If you're at the conference, please come see us!

User Needs during Social Search

There has been a lot of buzz around social search in the online tech community, but I am largely disappointed by the new tools and services I've encountered. It's not that these sites are unusable, but that they each seem to take on a different conception of what social search is and when/how it will be useful. Have these sites actually studied users doing social search tasks?

Social search may never have one clear, precise definition---and that's fine. However, my instinct is to look at the users and their behaviors, goals, and needs before designing technology. Actually useful social search facilities may be some ways off still (despite the numerous social search sites that advertise themselves as the future of search). First, we need to address some questions, such as:

1. Where are social interactions useful in the search process?


2. Why are social interactions useful when they occur?

Study Methods
To answer these questions, Ed Chi & I ran a survey on Mechanical Turk asking 150 users to recount their most recent search experience (also briefly described here and here). We didn't provide grand incentives for completing our survey (merely 20-35 cents), but we structured the survey in a narrative format and figured that most people completed it because it was fun or interesting. (This is a major reason for Turker participation.)

For example, instead of asking a single open-ended question about the search process, we first asked people when the episode occurred, what type of information they were seeking, why they needed it, and what they were doing immediately before they began their search. After this, we probed for details of the search act itself along with actions users took after the search. Our 27-question survey was structured in a before-during-after type format, primarily to establish a narrative and to collect as much detailed information about the context and purpose of users' actions.

We collected responses from 150 anonymous, English-speaking users with diverse backgrounds and occupations. In fact, there was so much diversity in our sample that the most highly represented professions were in Education (9%) and Financial Services (9%). The next ranking professions were Healthcare (7%) and Government Agency (6%) positions. We were quite surprised by the range of companies people worked for: from 1-person companies run out of people's homes to LexisNexis, Liberty Mutual, EA Games, and the IRS!

Our data analysis resulted in a model of social search that incorporated our findings of the role of social interactions during search with related work in search, information seeking and foraging. Without presenting the whole model here, I will highlight the summary points and conclusions from our work. (The full paper is available here.)

Search Motivations
There were two classes of "users" in our sample who we named according to their inherent search motivations. The majority of searchers were self-motivated (69%), meaning that their searches were self-initiated, done for their own personal benefit, or because they had a personal interest in finding the answer to a question. The remaining 31% of users were "externally-motivated"---or were performing searches because of a specific request by a boss, customer, or client.

Not surprisingly, a majority (70%) of externally-motivated searchers interacted with others before they executed a search. The fact that these searches were prompted by other people often led to conversations between the searcher and requester so that the searcher could gather enough information to establish the guidelines for the task. This class of behavior is noteworthy because even though these users engaged in social interactions, they were often required to or may not have otherwise had the occasion to interact.

Although only 30% of self-motivated searchers interacted with others before they executed a search, their reasons for interacting were more varied. While some still needed to establish search guidelines, others were seeking advice, brainstorming ideas, or collecting search tips (e.g., keywords, URLs, etc.). In many cases, these social interactions were natural extensions of their natural search process---these users were performing self-initiated searches afterall. Again this is noteworthy, suggesting that self-motivated searchers would be best supported by social search facilities.

Search Acts
Next, we identified three types of search acts: navigational, transactional, and informational. These classifications were based on Broder’s (2002) taxonomy of information needs in web search, and I'm only going to review our users' informational search patterns (searching for information assumed to be present, but otherwise unknown) since it proved to be the most interesting. Informational search is typically an exploratory process, combining foraging and sensemaking. As an example:

An environmental engineer began searching online for a digital schematic of a storm-water pump while simultaneously browsing through printed materials to get "a better idea of what the tool is called." This search was iteratively refined as the engineer encountered new information, first on metacrawler.com and then on Google, that allowed him to update his representation of the search space, or what might be called a "search schema." He finally discovered a keyword combination that provided the desired results.

Over half of search experiences in our sample were informational in nature (59.3%), and their associated search behaviors (foraging and sensemaking) led to interactions with others nearly half the time. Furthermore, 61.1% of information searchers were self-motivated. It appears there is a demand and a desire for social inputs where the search query is undeveloped or poorly specified, and personally relevant.

Post-Search Sharing
Finally, we noticed that, again, nearly half our users (47.3%) shared information with others following their search. This is not wholly unexpected, but points to the need for better online organizational and sharing tools, especially ones that could be built into the web browser or search engine itself. Instead, an interesting finding is why people chose to share information.

Externally-motivated searchers almost always shared information out of obligation---to provide information back to the boss or client who requested the search in the first place. Self-motivated searchers, however, often shared information to get feedback, to make sure the information was accurate and valid, or because they thought others would find it interesting.

Summary and Conclusion
In summary, we classified two types of users in our study: externally-prompted searchers and self-motivated searchers. The self-motivated were the most interesting because of their search habits, propensity to seek help from others, and the reasons behind their social exchanges. For this class of users, a majority performed informational, exploratory searches where the search query was ambiguous, unclear, or poorly specified, leading to a need for guidance from others. Their social interactions, therefore, were primarily used to brainstorm, get more information, and further develop their search schema before embarking on their search. Finally, the search process didn't end after these users identified preliminary search results---they often shared their findings out of interest to others, but also to get feedback, validate their results, and contemplate refining and repeating their search.

It is noteworthy that we did not ask users to report social search experiences in the survey. Instead, we asked for their most recent search act, regardless of what it was, expecting that across all 150 examples we would be able to begin finding generalizable patterns. Indeed, a large majority performed social search acts, but nearly all of the social exchanges were done through real-world interactions---not through online tools. It is no surprise that online tools need to better support social search experiences (our study is only further proof of this); but our study does contribute to a better understanding of user needs during "social" search, which may lead to tools that can best identify and support the class of users and search types best suited for explicit and implicit social support during search.

Finally, in response to the questions I posed at the very beginning:

Where are social interactions useful in the search process?
Before, during, and after a "search act"! Over 2/3 of our sample interacted with others at some point during the course of searching. However, social interactions may not benefit everyone equally---they appear to provide the best support for self-motivated users and users performing informational searches.

Why are social interactions useful when they occur?
It depends! The reasons for engaging with others ranged from a need to establish search guidelines to a need for brainstorming, collecting search tips, seeking advice, getting feedback, and validating search results. Social support during search may be best appreciated and adopted if it directly addresses these types of user needs.

Brynn M. Evans, Ed H. Chi. Towards a Model of Understanding Social Search. In Proc. of Computer-Supported Cooperative Work (CSCW), (to appear). ACM Press, 2008. San Diego, CA.