Bringing the psych lab online

The Internet has already fundamentally changed the way that people communicate, shop, and even date, but now it is poised to revolutionize psychological studies by enabling researchers to quickly and easily recruit thousands of study volunteers from around the world, and by changing the way the public interacts with researchers.

By conducting experiments online, researchers have been able to enlist as many as 65,000 volunteers to take part in studies of cognition, a number far larger than they could bring into the lab. Such studies, however, have been dogged by questions about whether anonymous, unpaid volunteers tested online can produce data that is as high quality as that gathered through in-person lab testing.

New research conducted by Harvard scientists may put those questions to rest.

A team led by Laura Germine, a postdoctoral research associate in Harvard’s Psychology Department, and made up of Ken Nakayama, Edgar Pierce Professor of Psychology and chair of the Department of Psychology, Jeremy Wilmer of Wellesley College, and Christopher F. Chabris ’88, Ph.D. ’99, assistant professor of psychology at Union College, has shown that data gathered through online volunteers can be just as good as that from in-person experiments. Their research is described in a forthcoming issue of Psychonomic Bulletin & Review.

“What this says is that people shouldn’t be afraid of the Web as a way of conducting their research,” said Germine, the paper’s first author. “Despite the cost advantages, despite the time advantages, researchers continue to worry that data from Web volunteers will not be as good as data from paid lab participants. We’ve shown that data from self-selected Web volunteers can be very good. The thing I like to say about using the Web is that it’s fast, it’s cheap, but it’s not dirty. In experiments like ours, what you’re getting is good, reliable data.”

To test whether Web studies are as valid as those done in the lab, researchers recruited thousands of volunteers through Germine’s site, TestMyBrain.org. The volunteers, who navigated from search engines and social networking sites, took part in a handful of tests to learn more about themselves and contribute to scientific research. These tests were designed to assess everything from facial recognition ability to a person’s capacity for remembering a long string of numbers. Researchers then compared the results of the online tests with those done in the lab.

“We looked at three basic metrics,” Germine explained. “We looked at mean performance, we looked at variation in performance, and we looked at internal consistency. In some ways, I think, the third measure — the consistency, or internal reliability — is the most important, because it allows you to look at how someone’s performance on one part of the test predicts how they will perform on another part of the test. If they stopped paying attention or started watching YouTube videos halfway through a test, that would be reflected in lower reliability.”

On each measure, Germine said, the results from studies conducted with Web volunteers were the same as those done in the lab. The only variation researchers found came when they compared the average, or mean, performance of the general public with that of elite college students from Harvard and Wellesley on IQ-based tests.

“For tests based on visual recognition and perception that are less related to IQ, average scores were the same for Web and lab,” Germine said.

While it has simplified the process of gathering data, the Web has also changed the relationship between researchers and those they study. By forming communities online, Germine said, people who suffer from a particular disorder have been able to raise awareness — and drive research in areas that might otherwise not be studied.

“The other benefit that comes from using the Web is that people can offer insights about themselves we might not think to ask,” Germine said. “The best example of this is with the selective developmental disorders.”

Arguably the most prominent example of such a disorder is prosopagnosia, also known as “face blindness,” in which individuals have difficulty recognizing people, sometimes even in their own families. Although the condition was occasionally reported following a stroke, when people claimed to have had the disorder from birth, they were often dismissed by doctors and researchers. As tests for facial recognition were made available online, researchers began to realize that the disorder was far more common than previously thought.

“As a result, we learned a great deal,” Germine said. “Because we had these tests available online, and because people were able to participate in this exploration online, the knowledge they gained validated their own experience, which encouraged them to get in contact with a researcher. So there was this circle of knowledge that became very beneficial to everyone involved.

“I think psychology, as a field, is unique in that the research questions can be made fairly accessible to the public,” she added. “The Web offers us another avenue to take advantage of that, and to engage people in our science in a new way. I think there’s a great opportunity for collaboration between scientists and ‘citizen scientists’ to advance scientific knowledge.”