SRI released their two-year, Gates-funded study of Khan Academy and I've read through their research brief. Regardless of what I summarize here, you should read it, too. It's an easy read and it will help you understand the limitations of the study and prevent unnecessary conclusion-jumping.
Here's what I'm taking away from the research brief:
Methodology: Researchers selected 20 schools and 70+ teachers across nine sites for this study, choosing a range of school types from a pool of volunteers. One public elementary school accounted for 8 of the schools, 50 of the teachers, and about half of the 2000 students that participated each year. The participants had "tight relationships" (p. 2) with Khan Academy, and feedback from the schools led to modifications in the program. Other than one site in one year, the participants in the study did not use Khan Academy as their primary curriculum, and the use of Khan Academy varied across settings and over time. This study isn't designed to establish causal claims, so we shouldn't make any. But that's okay — we can still learn a lot from education research that can't determine cause and effect. I see this as an efficacy study, not an effectiveness study (Sloane, 2008). Researchers do efficacy studies to judge if a program might be doing any good under idealized conditions. If an efficacy study suggests a program has value, then we can move to an effectiveness study with a more rigorous design to see if the program "works" in real-world conditions. (Unfortunately, this kind of organization and the funding to support it is not the norm.)
Favorable Findings: Khan Academy was generally well-liked by both students and teachers. Student engagement seemed high and teachers liked the modular nature of the materials, which made it easy to use as a supplement for the regular curriculum. At two research sites, increased time spent on Khan Academy was associated with better-than-expected test scores, decreased math anxiety, and increased confidence in math ability. Again, don't jump to conclusions — this represents just some of the students and the methods weren't rigorous. Increased time on something leading to improvement isn't all that surprising, and we should want to see further study.
Unfavorable Findings: Although teachers who used them thought they were somewhat to very useful, about 40% of teachers didn't refer to Khan Academy's student progress data more than once a month. Seventy percent of teachers who rarely or never used Khan Academy data preferred to use their own observations and assessments to judge student performance. Teachers also experienced difficulty finding content that was appropriately aligned. The study says Khan Academy responded to these things and made improvements. That's a good thing.
There is a lot more to see in the research brief, and even more in the implementation report. Browsing the latter, I see some interesting things, such as:
- Of the 14 schools participating in both years of the study, 11 appeared to significantly decrease Khan Academy usage in the second year. Researchers found this was likely due to less direct support from Khan Academy in year two and shifting goals and priorities (pp. 23-24).
- Students in the study aren't watching many Khan Academy videos (pp. 26-27).
I'm sure there's a great deal more there, and I encourage you to note anything you find in the comments.
You can still count me among Khan Academy skeptics — while I don't doubt there are some positive outcomes, I'm not sure Khan Academy fits into our best visions for teaching and learning mathematics, something I've written about on several occasions. For me, it's not all that helpful to know a program "works" unless I have a way to judge how well it "works" compared to other options, and in this case there's a lot of research yet to be done to find those answers. Unfortunately, I'm not expecting more big Khan Academy research any time soon. It took longer than expected for this study to be released and I started to wonder if it would be released at all. Given the study's limitations and use of only a subset of the data to make positive associations to outcomes I question if Gates or any other funder is ready to pony up the funding for a more rigorous study. That's too bad, because Khan Academy isn't going away and there are going to be more questions about how schools should use it, if we can be sure it's worth using at all.
References
Murphy, R., Gallagher, L., Krumm, A ., Mislevy, J., & Hafter, A. (2014). Research on the Use of Khan Academy in Schools. Menlo Park, CA: SRI Education.
Sloane, F. C. (2008). Randomized trials in mathematics education: Recalibrating the proposed high watermark. Educational Researcher, 37(9), 624–630. doi:10.3102/0013189X08328879