Crowdsourcing with Mobile Apps Brings 'Big Data' To Psychological Research

PRESS RELEASE

WASHINGTON — A fast-paced game app where players pretend they are baggage screening officers operating airport x-ray scanners has provided researchers with billions of pieces of data in record time, according to an article published by the American Psychological Association.

“Mobile devices offer researchers an exciting new means to crowdsource an experiment using games that are actually tests of cognition or other brain functions,” said Stephen R. Mitroff, PhD, of Duke University, lead author of the report, published in APA’s Journal of Experimental Psychology: Human Perception and Performance®. “Questions that could have taken decades to answer in a laboratory setting, or that could not be realistically answered in a lab, can be examined using big data gathered in a relatively short time.”

To demonstrate the potential of mobile technology to gather data, the researchers partnered with Kedlin Co., the developer of the popular mobile app game Airport Scanner, which challenges players to identify illegal items in luggage passing through an airport x-ray scanner. Players view one bag at a time and tap their touchscreen to identify banned items. The game includes a logbook of illegal items (e.g., guns, hand grenades, switchblades) and legal items (e.g., ear phones, clothing), and the list expands from a handful of possible target items to hundreds as players progress through the game.

Between January 2013 and November 2014, the game provided anonymous data from more than 2 billion trials from over 7 million mobile devices that Mitroff and his research team at the Duke Center for Cognitive Neuroscience have been analyzing, the report said. Players consent to the data collection when installing the game on their smartphone or other hand-held device.

One research question explored through the Airport Scanner game was examining players’ ability to spot items that occurred on screen very rarely, or in less than 0.1 percent of all bags.

“When a target appears only 0.1 percent of the time, you need 1,000 trials to get just a single case of that target. Too many trials would be needed to realistically assess detection of that particular target in a laboratory setting,” Mitroff said. “With the large dataset, we were able to look at hundreds of cases for each of the nearly 30 rarest targets.”

Another experiment found that when two banned items both appeared in a bag, players were more likely to miss the second to-be-found item when the two were different than if they were identical. For example, players were more likely to miss a dynamite stick after finding a gasoline can than they were to miss a dynamite stick after first finding a dynamite stick.

Sample images from Airport Scanner.

Sample images from Airport Scanner: the left image contains one target (hand grenade), the middle contains two identical targets (two exemplars of the dynamite stick target type), and the right image contains two different target types (derringer, gasoline can). Airport Scanner images appear with permission from Kedlin Co. Copyright 2014 by the Kedlin Company.

“While using a game interface to assess cognitive abilities is not new to psychological research, mobile technology offers a phenomenal opportunity to examine cognitive processes on a large scale,” Mitroff said. “In 1998, 1 million trials were analyzed by Jeremy M. Wolfe, PhD, of Harvard Medical School, to understand visual search. This was a mind-blowing amount of data that took 10 years to collect. Today, we can collect over a million trials a day through Airport Scanner.”

Other advantages of crowdsourcing include that it can be a less-expensive means of collecting data automatically and continuously and that it has the ability to mimic real-world situations that are difficult to address in a laboratory, according to the report.

Crowdsourcing research data does have disadvantages, the authors noted. Researchers must have the skills to create a fun game app or they need to partner with a developer. Collecting large amounts of data may not necessarily result in high-quality data, which requires carefully developing research questions to be addressed through the game. And collecting data through crowdsourcing means researchers have no control over who is playing, they said.

Article: “What Can 1 Billion Trials Tell Us About Visual Search?” Stephen R. Mitroff, PhD, Adam T. Biggs, PhD, Stephen H. Adamo, BS, Emma Wu Dowd, MSc, Jonathan Winkle, BS, and Kait Clark, PhD, Duke University, Journal of Experimental Psychology: Human Perception and Performance, online Dec. 8, 2014.

Stephen R. Mitroff and Ben Sharpe of Kedlin Co. can both be contacted by email.

The American Psychological Association, in Washington, D.C., is the largest scientific and professional organization representing psychology in the United States. APA's membership includes nearly 130,000 researchers, educators, clinicians, consultants and students. Through its divisions in 54 subfields of psychology and affiliations with 60 state, territorial and Canadian provincial associations, APA works to advance the creation, communication and application of psychological knowledge to benefit society and improve people's lives.