AERA Philadelphia 2014: Travels and Tribulations

I'm in Philadelphia for this year's Annual Meeting of the American Educational Research Association, or AERA. This conference is a big undertaking — I think of it not just in terms of the 15,000+ people and their research, but the collective time, effort, and money that goes into everyone's travel arrangements, hotel reservations, and conference registrations. It's a big number, whatever it is, and as an attendee of the conference I feel like I have a duty to see that the funds that sent me here are well-spent.

To those ends, I had planned to catch an early flight from Denver and to get to Philadelphia in time to attend some late-afternoon sessions and tonight's AERA presidential address. I didn't make it. I spent most of the day yesterday getting my posters ready and packing for a 10-day AERA-NCTM conference marathon. Instead of getting a good nights' rest, I packed and prepared into the early morning and headed to catch the early bus (3:20 AM!) to the Denver airport. I ended up missing that bus by just seconds but caught the 4:40 AM bus which still gave me ample time to get through DIA for a 7:00 flight.

We got on the plane, backed away from the gate, sat for a moment, and then heard the pilot explain that there was a problem and we'd be pulling back into the gate and waiting for maintenance to give the plane a look. His explanation was intriguing: There were two ignition systems for the engines, and the systems for each engine work independently of the other. Somehow, one of the systems had failed on both engines, which the pilot said was very unusual. To make things even more strange, the plane next to us was having the exact same problem. We sat there in the plane for almost 3 hours before the problem was resolved (how, we were never told), our plane was de-iced, and we were in the air.

I had hoped to be in Philly at 12:30 local time but it ended up being closer to 3. Then I waited for a hotel shuttle to arrive and was the last of 10 people to get dropped off at their hotel. By then it was about 5:30 here, which meant my total trip had taken about 12 hours and I'd done it on no sleep. With 9 days of conferences ahead of me, now I'm just looking for some rest. Really, at times like this I can't help but ask, "Why does anybody bother?" Rested, tomorrow I hope to spend a day at the convention center answering that question.

Results from the SRI Study of Khan Academy

SRI released their two-year, Gates-funded study of Khan Academy and I've read through their research brief. Regardless of what I summarize here, you should read it, too. It's an easy read and it will help you understand the limitations of the study and prevent unnecessary conclusion-jumping.

Here's what I'm taking away from the research brief:

Methodology: Researchers selected 20 schools and 70+ teachers across nine sites for this study, choosing a range of school types from a pool of volunteers. One public elementary school accounted for 8 of the schools, 50 of the teachers, and about half of the 2000 students that participated each year. The participants had "tight relationships" (p. 2) with Khan Academy, and feedback from the schools led to modifications in the program. Other than one site in one year, the participants in the study did not use Khan Academy as their primary curriculum, and the use of Khan Academy varied across settings and over time. This study isn't designed to establish causal claims, so we shouldn't make any. But that's okay — we can still learn a lot from education research that can't determine cause and effect. I see this as an efficacy study, not an effectiveness study (Sloane, 2008). Researchers do efficacy studies to judge if a program might be doing any good under idealized conditions. If an efficacy study suggests a program has value, then we can move to an effectiveness study with a more rigorous design to see if the program "works" in real-world conditions. (Unfortunately, this kind of organization and the funding to support it is not the norm.)

Favorable Findings: Khan Academy was generally well-liked by both students and teachers. Student engagement seemed high and teachers liked the modular nature of the materials, which made it easy to use as a supplement for the regular curriculum. At two research sites, increased time spent on Khan Academy was associated with better-than-expected test scores, decreased math anxiety, and increased confidence in math ability. Again, don't jump to conclusions — this represents just some of the students and the methods weren't rigorous. Increased time on something leading to improvement isn't all that surprising, and we should want to see further study.

Unfavorable Findings: Although teachers who used them thought they were somewhat to very useful, about 40% of teachers didn't refer to Khan Academy's student progress data more than once a month. Seventy percent of teachers who rarely or never used Khan Academy data preferred to use their own observations and assessments to judge student performance. Teachers also experienced difficulty finding content that was appropriately aligned. The study says Khan Academy responded to these things and made improvements. That's a good thing.

There is a lot more to see in the research brief, and even more in the implementation report. Browsing the latter, I see some interesting things, such as:

  • Of the 14 schools participating in both years of the study, 11 appeared to significantly decrease Khan Academy usage in the second year. Researchers found this was likely due to less direct support from Khan Academy in year two and shifting goals and priorities (pp. 23-24).
  • Students in the study aren't watching many Khan Academy videos (pp. 26-27).

I'm sure there's a great deal more there, and I encourage you to note anything you find in the comments.

You can still count me among Khan Academy skeptics — while I don't doubt there are some positive outcomes, I'm not sure Khan Academy fits into our best visions for teaching and learning mathematics, something I've written about on several occasions. For me, it's not all that helpful to know a program "works" unless I have a way to judge how well it "works" compared to other options, and in this case there's a lot of research yet to be done to find those answers. Unfortunately, I'm not expecting more big Khan Academy research any time soon. It took longer than expected for this study to be released and I started to wonder if it would be released at all. Given the study's limitations and use of only a subset of the data to make positive associations to outcomes I question if Gates or any other funder is ready to pony up the funding for a more rigorous study. That's too bad, because Khan Academy isn't going away and there are going to be more questions about how schools should use it, if we can be sure it's worth using at all.

References

Murphy, R., Gallagher, L., Krumm, A ., Mislevy, J., & Hafter, A. (2014). Research on the Use of Khan Academy in Schools. Menlo Park, CA: SRI Education.

Sloane, F. C. (2008). Randomized trials in mathematics education: Recalibrating the proposed high watermark. Educational Researcher, 37(9), 624–630. doi:10.3102/0013189X08328879

Team Reading of Academic Literature

In classes with +David Webb we sometimes read articles with a particular role in mind, a practice he picked up while a student at the University of Wisconsin. This week we're reading Windschitl and Thompson's Transcending Simple Forms of School Science Investigation: The Impact of Preservice Instruction on Teachers' Understandings of Model-Based Inquiry, published in AERJ in 2006. There are only five students in the class so we're using these five roles:

  1. Empirical Critic - Finds faults with the method. Finds limits to generalizability. Finds instances of shallowness in interpretation, obliviousness to context. Finds weakness in internal and external validity.
  2. Theoretical Critic - Identifies theoretical givens. Challenges the chain of reasoning by which the claimed findings are related to broader theoretical issues. Offers explanations that proponents of rival theories would give to the claimed findings.
  3. Literary Truth Squad - Identifies claims in which cited literature is used as evidence. Succinctly restates the claim. Tracks down the cited literature and evaluates whether there was warrant for treating the cited literature as evidence for the claim that was made.
  4. Worth-Finder - Beyond the identifiable faults of a work, what can still be gained from it? What good did it do? What is the most memorable about it? Are there implications that were unreported by the author?
  5. Sociologist of Knowledge - Whose work was cited? Whose wasn't? What disciplinary and theoretical traditions are being followed? What ideological commitments are being made by the author(s)? Are there geographical and institutional biases in the citation pattern? Prospectively, who used this work, and how?

This week I'm the Literary Truth Squad (which always reminds me of Police Squad) so my reading of the article is interrupted by a lot of trips to Google Scholar and readings of abstracts. Warranted or not, I've gained a bit of a reputation for having a good grasp of the literature. But since this is a science education article and not math ed, I'm having to look up everything. It's time-consuming and for a 53-page article I'm going to have to pick-and-choose where to look more closely. Sometimes I can guess it's a relevant reference just from the title of the article, and that speeds up the process. Sometimes the reference isn't a published article, but rather a paper presented at a conference. Those can be difficult to find, if they're online at all. Good conference presentations often end up as articles in journals, but the conference paper gets cited anyway and there's no real way yet to automatically refer the paper reference to a later paper.

This process is nothing like casual, lay-on-the-couch reading, but that's the point. It forces me to read with a purpose and be accountable to my classmates, each of whom will be reading equally carefully with their own roles in mind. When I was teaching I struggled having students work in groups until I helped group members identify roles and responsibilities and how they should work together. This is very similar and adds a nice structure to our in-class discussions, so long as we don't let pride in our work get in the way of brevity. I may look at all 78 cited resources but there's probably little need to inform my classmates of more than just a few.

Now...back to truth squading!

RYSK: Boaler's The Development of Disciplinary Relationships: Knowledge, Practice and Identity in Mathematics Classrooms (2002)

This is the 22nd in a series describing "Research You Should Know" (RYSK)

Today Geoff Krall (@emergentmath) posed this on Twitter:



It's a cycle I've seen before, and you probably have, too. Students struggle on complex tasks (or, far worse, we just assume they'll struggle without giving them the opportunity) so we opt for the "back to basics" approach and change our mathematical practices to include a lot of repetition on lower-level, procedural tasks. We convince ourselves that this is the right thing until either the students tune out, or underperform, or both, and then we bark about "raising the bar" and the cycle begins anew.

Geoff's cycle reflects knowledge and practice, but I wanted to dig a little deeper into the idea of "student resistance." Jo Boaler (@joboaler) found herself in a similar position after she'd done some studies that linked reform-oriented practices to a more flexible and robust form of mathematical knowledge, but felt there were some stronger links to make between knowledge and a student's mathematical identity, or the way they see themselves as and becoming knowers and do-ers of mathematics.

Boaler described this process in an article titled The Development of Disciplinary Relationships: Knowledge, Practice and Identity in Mathematics Classrooms, published in 2002 in the journal For the Learning of Mathematics. You can find a preprint of the article on Boaler's faculty website, but I took the time tonight to add a summary of the article to the MathEd.net Wiki:

http://mathed.net/wiki/Boaler_(2002)_FLM

My takeaway from reading this particular Boaler article is that while both traditional and reform approaches can result in student learning, the attention to student identity and affect in the reform approach shapes student learning in a way that makes the knowledge more useful in more situations, or, in the relative absence of knowledge, gives students both the disposition and a set of practices to make mathematical progress. When students get caught in Geoff's "Cycle of Low Performance," it's not that they aren't learning. Instead, they just don't see their knowledge as particularly valuable, nor do they see themselves as active users of that knowledge. As teachers, we need to design our classrooms and activities in ways that give students opportunities to have some authority over their mathematical ideas if we expect them to use their knowledge productively.

The MathEd.net Wiki

Today I'm announcing a project over a year in the making, the MathEd.net Wiki.

The project has its roots in my own attempts as a graduate student to connect the literature I engage in. Tools like Zotero and Mendeley helped me organize PDFs and references, but I really wanted something that could help connect literature and citations together. Several times I tried setting up a wiki to do this, but each time it felt disorganized and overwhelming. As I prepared for my comprehensive exams in the fall of 2012, I finally found an organizational system for the wiki I thought I could sustain. I've been quietly contributing to it as a private side project ever since, and now feel it's reached the critical mass it needed for me to stay committed to it — and to make it public.

One of the things that drove me to graduate school was a desire to engage in the academic scholarship of my profession. Unfortunately, that's not easy to do as a teacher — either the literature is behind paywalls or inaccessible without a way to get the background knowledge needed to understand it. Usually both. When I started the "Research You Should Know" series on this blog, I hoped I could make some small dent in that problem. But blog posts still didn't let me make all the connections between the literature that I wanted to make. I wanted something that was less linear, and I think this wiki could be that thing.

Having read about similar specialized wikis, I know most fail to meet the expectations of their creators. I'm keeping my expectations modest. It's still a side project. (I have to finish my PhD program sometime, right?) It's a work in progress, and always will be. I don't expect dozens of contributors, although I'd really enjoy having at least a few. I don't expect the simple act of having access to more research to revolutionize anyone's practices. You deserve access to it, but alone it won't solve all our problems. I'll keep contributing to it, and I encourage you to visit the wiki, read the welcome message, and browse around a bit. I would love to get feedback, so either leave a comment or send me an email with your ideas!