Today I presented our work on using Amazon's Mechanical Turk service as a user testing method to PARC's Computing Science Lab (CSL). Several of the researchers in the audience asked "what does the demographic of Mechanical Turk users look like, and whether it is a reasonable sample of the real demographic" that one might want for user testing of HCI systems. I thought that was a great question.
Luckily, our very own intern Brynn Evans recently found a great blog post about the demographics of Mechanical Turk. For example, some have surmised that since MT pays so little, perhaps many of the turkers are from third world countries with lower minimum wages. This turned out not to be the case. About 82% of the users are from either the US, Canada, or UK.
What about income distributions? Perhaps people with lower wages or salaries are more willing to participate. Well, the self-reported income distribution looks remarkably like the income distribution of general online users.
As one might have suspected, the answer is that turkers participate not just for money, but for fun and for a sense of game. Bringing mechanical turk really in line with ESP games.
For more details, see: A Computer Scientist in a Business School: Mechanical Turk: The Demographics
7 comments:
In that post showing the demographics of Mechanical Turk users it says, "Turkers are a pretty representative sample" with regard to education. The chart shows that over 50% of MTers have a bachelors degree and roughly another 20% have masters degrees or PhDs.
But in the US, only about 26% of the population has a college degree. (Source U.S. Census.) Less than half of that have higher degrees. So MT users are not representative in this way -- they are actually much more highly educated than the norm.
This is probably the opposite of what you'd expect if MT was appealing to low-wage earners (since education and wages are correlated). On the other hand, it probably is representative of participants in traditional psych experiments, which are overwhelmingly college students. So at least we scientists are being consistent about the portion of the population we keep studying. :-)
Ravella:
That's great that you were able to dig this up. I wonder to what extent the education distribution of MT population is representative of general web users.
In either case, as you mentioned, it's probably representative of participants in psych experiments at universities.
We scientists have a tendency to study whatever population is available to us. No matter what, I think we might be able to draw from a larger population than in the past.
I just wanted to identify myself. I was logged in to my blog account as ravella and didn't realize my real name wouldn't appear. So it's me, Ed.
Thanks for the reply. I wish somehow we could get at a more representative sample. Something to think about.
--Ellen
Glad to read these follow-up comments. How interesting if the Turkers are really more like college students than the assumption that they are low-wage workers. I think the issue of demographics and utility of Turkers is still on the table, though (having run some mturk studies myself) I tend to agree with Ed that we may be able to draw from a larger population than in the past (and more quickly and easily).
Perhaps we should do a validation study of Turker demographics?
Brynn:
Are you suggesting that the original study that was cited was not enough to ascertain the demographics of MT? What else would you like to find out?
It should be trivial to replicate the study, especially with the new MTurk interface.
If I would do it again, I would run it over a longer period of time and I would ask more people to participate.
Now, regarding the demographics, I think that they are representative of the general online users in the US, which is more educated and younger than the general population. Unfortunately, I cannot find any pointers right now that would describe the demographics of online users.
Panos, I also did a brief search for the demographics of online users in the US. The Pew Internet & American Life Project had the following listing: Demographics of Internet Users.
(Not quite what we're looking for, I think.)
You can also download a huge spreadsheet of all the internet-related questions they've asked people about since 2000: Usage Over Time.
Finally, re-reading my earlier comment, I'm not sure why I said the question of demographics was still on the table. With a population on mturk that might be in flux from HIT to HIT, perhaps I meant that we wouldn't necessarily which segment of that population would participate on a given HIT. Perhaps I'm just generally interested in who these turkers are--can you ever have too much information about them?
Post a Comment