But Are They Really Learning? The First Controlled Study of an iPad Learning App

Learning apps for mobile devices have been a hot topic in educational technology for some time — but most developers haven’t been able to back their claims of educational value for students. Until now: Our friends at Motion Math, Jacob Klein and Gabriel Adauto, have just announced the results of a recent study evaluating students’ engagement with their iPad app. Learn more about the study and what the researchers found.

How does one know that an educational experience is actually helping students learn? Our company Motion Math makes educations games for the iPad and iPhone that let kids play with numbers. It’s easy for us to think, as we’re making our apps, and watching students play them, to believe that learning is happening, especially because we spend a lot of time ensuring that our designs follow good pedagogical and usability principles. However, the history of educational technology is littered with many false promises and disappointing results, most recently given an overview by Matt Richtel of The New York Times. For these reasons, and for our own self-understanding, it’s important that we sometimes hold our learning technologies up to scientific scrutiny.

That’s why we decided to put one of our apps, a fractions estimation game for the iPad, to the test. To our knowledge this is the first ever experiment to measure the learning impact of an iPad app. Using a grant from the Noyce Foundation we commissioned Professor Michelle Riconscente, an expert in educational technology and assessment at USC, to study if our game Motion Math HD achieved two main goals: to help children master estimation of fractions on the number line, and to give students a more positive attitude toward learning fractions. Some of the design choices that ensured the study’s rigor:

• All the test items measuring fractions proficiency were taken from the California Standards Test (CST), the National Assessment of Educational Progress (NAEP), and Trends in International Mathematics and Science Study (TIMSS). In other words, the questions were not specially crafted to artificially produce a positive effect.
• We hired an independent third party to conduct the research.
• The report describes its methods, participants, data, outcomes, and interpretations.
• The experiment included a control group of similar students from the same schools.

The game research organization GameDesk has recently published the results of Riconscente’s study of 122 5th graders. The main findings:

• Students who played Motion Math for 20 minutes for five days improved on a fractions test by an average of 15%.
• Students’ attitudes towards fractions improved 10%.
• Virtually all students rated the game as fun and believed that it helped them learn.

This is a very encouraging result that our approach to engaged, interactive learning works. The full report includes an overview of how difficult it is for students to learn fractions, a description of the study’s design, some helpful critiques of our game’s design, and recommendations for learning app producers and researchers.

A few early reviewers have suggested to us that this study is overkill. Well, certainly not every kid’s app can afford the time and money it takes for a formal assessment. When designing an app, there are hundreds of design decisions to make, and the majority can’t be guided by rigorous experimental data. A designer needs to follow his or her intuition, listen to experts, read the learning literature, discern the elements of good design, and do user-testing to watch and see if students seem to be learning.

The problem is that without introducing the scientific method, using hard data of assessment, there’s no way for our design guidelines to evolve. Many of our assumptions about education (and especially about a new educational technology such as the iPad) are almost certainly wrong, and lead to the creation of bad learning products that don’t make children wiser or smarter. They only waste their time. So if we rely only on the unsystematic or dispersed observations of teachers or experts, we’ll have some input, but we won’t really know. Controlled experiments are an important method to move educational product design forward.

We’re hoping that this study kicks off a conversation — please comment below.

• What should the standards of evidence be for learning apps?
• What do you think about the study?
• How can we speed up and bring down the cost of rigorous assessment?
• How can parents and teachers be confident that a learning app actually helps children learn?

Jacob Klein and Gabriel Adauto are co-founders of Motion Math, the makers of award-winning learning games that let kids play with numbers, and a Cooney Center Mobile Innovation Prize finalist. Their most recent game Motion Math: Hungry Fish teaches mental addition and subtraction with a fish who eats numbers. It’s available for free for iPad, iPhone, and iPod Touch.