Using 'collective intelligence' to detect the start of a recession period?

I try to read the Economist regularly, but it's hard to keep up because each issue is so good. I just came across an article in the Economist from Jan 10th 2008 issue talking about using the frequency of the word 'recession' in Washington Post and NYTimes to identify the start of an actual recession period. Interestingly, according the graph below and the Economist, "This simple formula pinpointed the start of recession in 1981 and 1990 and 2001." Seems somewhat believable to me.


However, since news articles are written by the 'elite' journalists at Washington Post and the NYTimes, so this isn't quite what people have in mind when they think of 'wisdom of the crowd'. So I tried Google Trends instead, to see if the way people searched for the keyword 'recession' also correspond to the start of the recession period predicted by the R-index method by the Economist. Sure enough, the results seem to agree:


I then checked the same trend on the access traffic numbers for the 'recession' article on Wikipedia, and found the same peak in January:


I guess we don't need any more evidence that 'recession' started in January, or at least everyone seems to be obsessed about it then.

I then thought to myself: Perhaps, 'wisdom of the searchers' can also be used to predict who will win the presidential race in November? Here I deliberately made sure that blue is the keyword 'Obama', while red is the keyword 'McCain'.

It sure looks like Obama has the upper hand right now.

WikiDashboard: Providing social transparency to Wikipedia

WikiDashboard Tool (alpha-release)

We are pleased to announce the release of our first research prototype of a social dynamic analysis tool for Wikipedia called WikiDashboard. This is a quick guide to our social dynamic analysis tool for Wikipedia

Motivation

The idea is that if we provide social transparency and enable attribution of work to individual workers in Wikipedia, then this will eventually result in increased credibility and trust in the page content, and therefore higher levels of trust in Wikipedia.

You might ask "Why would increasing social transparency result in higher quality articles and increase trust?"

Indeed, the quality of the articles in Wikipedia has been debated heavily in the press [here, here, here, here, and let's not forget the Nature magazine debacle].

Wikipedia itself keeps track of these studies and openly discusses them here, which is a form of social transparency itself. However, even Wales himself have has been quoted as saying that "while Wikipedia is useful for many things, he would like to make it known that he does not recommend it to college students for serious research." Indeed, the standard complaint I often hear about Wikipedia is that because of its editorial policy (anyone can edit anything), it is an unreliable source of information.

The opposite point of view, however, has not been debated or expressed nearly as much: Precisely because anyone can edit anything and that anyone can examine the edit history and see who has made them, it will (or has already) become a reliable source of information. I think Michael Scott, the character on the popular TV show "The Office", puts it succinctly: "Wikipedia is the best thing ever. Anyone in the world, can write anything they want about any subject. So you know you are getting the best possible information."

While tongue-in-cheek, it brings up a valid point. Because the information is out there for anyone to examine and to question, incorrect information can be fixed and two disputed points of view can be examined side-by-side. In fact, this is precisely the academic process for ascertaining the truth. Scholars publish papers so that theories can be put forth and debated, facts can be examined, and ideas challenged. Without publication and without social transparency of attribution of ideas and facts to individual researchers, there would be no scientific progress. Therefore, it seems somewhat ironic that the History Department at the Middlebury College have banned its students from citing Wikipedia sources .

Related Work

Indeed, just very recently WikiScanner has brought the issue and idea of social transparency to the forefront. It helps people find out the organizations where anonymous edits in Wikipedia are coming from. A week or two later, WikiRage helps identify the hottest trends in Wikipedia.

From academic works, we have seen interesting work from IBM called History Flow that visualizes the edits to article pages in Wikipedia, and the UCSC Wiki Trust Coloring Demo that demonstrated how trust could be visualized line-by-line. These are all examples of how being able to better understand editing history and editing patterns at a glance could dramatically help users uncover problems and the trustworthiness of contents on Wikipedia.

These tools and other discussions [NYTimes , blogs, and slashdot discussion] are noticing that accountability and transparency appears to be at the heart of the process that helps generate quality articles.

Guide to our tool

The tool can be used just as if you're on the Wikipedia site itself. All of the functions (such as the article search function, and the edit and history tabs) work just as before. The site provides the dashboard for each page in Wikipedia, while proxying the rest of the content from Wikipedia.

Note that we only currently have edit data up until 2007/07/16, so more recent edits are not included in the charts. We're working to fix this.

See our guide for help on understanding the visualizations in the WikiDashboard.

Some Interesting Examples

We will use the 2008 presidential election as an example. In the figure below, we see that the activities on this page has been heating up lately:
2008 US Presidential election
http://wikidashboard.parc.com/wiki/2008_presidential_election


Here are some notable Democractic Party candidates:
Hillary Clinton
http://wikidashboard.parc.com/wiki/Hillary_Rodham_Clinton


John Edwards
http://wikidashboard.parc.com/wiki/John_Edwards


Barack Obama
http://wikidashboard.parc.com/wiki/Barack_Obama


Here are some notable Republican candidates:

Rudy Giuliani
http://wikidashboard.parc.com/wiki/Rudy_Giuliani


John McCain
http://wikidashboard.parc.com/wiki/John_McCain


Ron Paul
http://wikidashboard.parc.com/wiki/Ron_Paul


Summary

We're curious of how the Web community will use this tool to surface social dynamics and editing patterns that might otherwise be difficult to find and analyze in Wikipedia. We are also interested in applying this tool to Enterprise Wikis. Please let us know by leaving a comment on this blog post on patterns you find or questions for us. Alternatively, (if you wish to contact us in private), email us at:
wikidashboard [at] parc [dot] com

Thanks,

Bongwon Suh
Ed H. Chi

Palo Alto Research Center
(joint work with our ex-colleagues Bryan Pendleton, Niki Kittur, now both at CMU)