Saturday, August 2, 2008

Using 'collective intelligence' to detect the start of a recession period?

I try to read the Economist regularly, but it's hard to keep up because each issue is so good. I just came across an article in the Economist from Jan 10th 2008 issue talking about using the frequency of the word 'recession' in Washington Post and NYTimes to identify the start of an actual recession period. Interestingly, according the graph below and the Economist, "This simple formula pinpointed the start of recession in 1981 and 1990 and 2001." Seems somewhat believable to me.


However, since news articles are written by the 'elite' journalists at Washington Post and the NYTimes, so this isn't quite what people have in mind when they think of 'wisdom of the crowd'. So I tried Google Trends instead, to see if the way people searched for the keyword 'recession' also correspond to the start of the recession period predicted by the R-index method by the Economist. Sure enough, the results seem to agree:


I then checked the same trend on the access traffic numbers for the 'recession' article on Wikipedia, and found the same peak in January:


I guess we don't need any more evidence that 'recession' started in January, or at least everyone seems to be obsessed about it then.

I then thought to myself: Perhaps, 'wisdom of the searchers' can also be used to predict who will win the presidential race in November? Here I deliberately made sure that blue is the keyword 'Obama', while red is the keyword 'McCain'.

It sure looks like Obama has the upper hand right now.

5 comments:

Anonymous said...

The number of revisions to the Wikipedia article on "Recession" is also pretty strong on Jan 2008.

http://sergionunes.com/p/wikichanges/?q=Recession

On the Obama vs. McCain issue the "Wikipedia vote" is also on Obama :)

http://sergionunes.com/p/wikichanges/?q=Barack_Obama&q2=John_McCain

Ed H. Chi said...

Sergio:

I didn't include that data, because from WikiDashboard (our tool), I knew that one user was responsible for 15.8% (or 117) of all of the edits on the 'Recession' article:
http://wikidashboard.parc.com:80/wiki/Recession

For Obama: http://wikidashboard.parc.com:80/wiki/Obama
and McCain:
http://wikidashboard.parc.com:80/wiki/Mccain
I thought the edit frequency results were less convincing, so I didn't include it in my discussion.

In either case, thanks for the pointer to your tool and the plots. They looked really useful. I hope you find our WikiDashboard tool to be similarly useful.

Anonymous said...

I'm aware of the excellent WikiDashboard. It is really a useful addition to Wikipedia. I've mentioned it on a paper to be published at WikiSym'08.

Thanks for your feedback.

Jerry Steele said...

Hmm...it would be interesting to see how the Google Trend prediction compares to something like a prediction market. I would expect the markets to be more accurate over the long haul because in a prediction market, you can lose something if you're wrong, so you have to be more careful about how, where, and when you express interest in your candidate. In the blogosphere you can spout off whatever with no risk.

Also, the Google Trends may simply be showing that there are more Obama supporters using social media than McCain supporters.

Ed H. Chi said...

fitzgeraldsteele:

Google Trends is actually reporting the usage of the word 'obama' as search terms.

http://wikidashboard.parc.com/wiki/Google_Trends

So even if the social media is buzzing about 'obama', presumably this doesn't necessarily affect the number of people who search for the keyword 'McCain'.

I think the comparison with Prediction Market is exactly right. The current prediction in the Iowa Electronic Market (IEM) is for Obama to win as well:
http://iemweb.biz.uiowa.edu/graphs/graph_PRES08_VS.cfm