Jo Boaler, Standing Tall

Last night, Jo Boaler (whose work I've written about before) took to Twitter (welcome, Jo!) to share details of "harassment and persecution" regarding her research, which she has written about on Stanford's website (PDF). Those in the math community had some understanding that this had been going on, and I applaud Boaler's decision to bring it out in the open.

I'm sure much will be said about this in the coming days, but I hope at least some small part of the conversation addresses the discoverability and sharability of academic work. When I search for "boaler railside" on Google, this is what I see:

Instead of the first result pointing me to Boaler's 2008 article in Teachers College Record, I'm instead pointed directly to the Bishop, Clopton, and Milgram paper at the heart of this controversy. As Boaler has pointed out, it has never been published in a peer-reviewed journal. But it is published, in the modern sense, with perhaps something more important than peer review: a top ranking on Google. The second link points to Boaler's faculty profile, through which a couple of clicks will take you Boaler's self-hosted copy of the Railside article. I'm linking directly to it here not only because it's an article you should keep and read, but because it obviously needs all the Google PageRank help it can get. The third link in my search also refers to the "refutation" of Boaler's work, although the site no longer appears to exist.

Why is Boaler's original work not easier to find? Let's look at the Copyright Agreement of the Teacher's College Record. According to TCR, it is their policy to "acquire copyright for all of the material published on" and that such a policy "is designed to promote the widest distribution of the material appearing on while simultaneously protecting the rights of authors and of as the publisher." For TCR, this "widest distribution" means putting the article behind a $7 paywall -- not an extravagant amount, but enough to keep most people from reading the work, which means not linking to it and not elevating its search rankings. (A search in Google Scholar, however, returns it as the top result.) Given the attacks on Boaler and her scholarship, has this copyright policy been "protecting the rights of authors?" In Boaler's case, it's obvious it hasn't. But then again, by signing over copyright I'm not sure exactly what rights TCR says she has left to protect.

I'm glad Boaler is sharing the article on her website. If she wasn't, I'd attempt to gain the rights to share it here, and that's not cheap:

Yes, republishing the article costs $500. Is it worth it for me to pay out of my own pocket? Probably not. But is it worth $500 to the greater mathematics education community to have it more discoverable, searchable, and sharable? Given what she's went through, is it worth it to Jo Boaler? Yes, it is, and that's why encourage all authors to publish in open access journals or otherwise negotiate their copyright agreement to ensure greater rights over their own work, including the ability to post and share in ways that improve search rankings.


  1. I've read through both papers and I am confused by the Bishop, Clopton, and Milgram paper. Most of their analysis rests heavily on correctly identifying the schools in question; and they never explain how they actually found the schools. How exactly are they certain they've correctly identified the schools? What technique did they use?

    They've also done an item analysis of the tests themselves, which is made somewhat difficult by the lack of a link to the tests Jo actually used. They've also indicated that they "found these tests on Jo Boaler's website" but given that I cannot find the tests myself, it makes it hard to refute their claims in this respect which is part of the reason we use a formal peer review process.

    I think I need more information to make a judgement, but it certainly seems that the research style of Bishop, Clopton, and Milgram is suspect.

    1. It is a bit of a tricky situation that raises a larger issue: how do you balance the need for subject confidentiality with the need for study replicability and independent review?

      A recent comment by Bishop on the math-teach list ( doesn't seem to clear up much about how they think they identified Boaler's schools. He says for two schools it was obvious, but one was "completely wrong." I can't tell if that was an error they fixed prior to "publication" or not, but given the continuing strength of the argument I can only assume it is. (Which is hazardous, as relying on assumptions seems to be near the heart of the problem.)

      I hope we can agree that a much better contribution to this debate from Milgram et al. would have been to replicate the study with different schools while being more open with their methods and data. (That's asking a lot, given that I doubt Milgram et al. have had much need to study and practice qualitative research methods.) There are some strong beliefs that educational research is not scientific due to the lack of replication of studies, but too rarely are these critiques accompanied with attempts to actually replicate someone's results, or at least refer to similarly rigorous studies with differing outcomes.

  2. Jo Boaler does a good job of arguing that these guys are mean (although she leaves out certain details, such as the fact that the Stanford police became involved only because she herself filed a frivolous complaint).

    But she doesn't even try to answer their most devastating arguments:

    1. She designed her own math test, which is inherently suspicious (interventions always look better when the designer administers his or her own test), and did so poorly (there were errors, and the tests didn't rise to the level of the California standards).

    2. She doesn't dispute that they correctly identified the schools at issue or that they correctly point out that math performance did NOT miraculously rise at her intervention school.

  3. Number 2 is a nonstarter: why would a researcher bound by confidentiality agreements confirm or deny that Milgram et al identified the schools? An admission either way would provide information and therefore violate confidentiality. I note for the record that Anonymous commenters and Guests have been busy defending the unpublished Milgram/Bishop study on Inside Higher Ed and other venues and have been making similar claims in similar language. I hope their IP addresses are published.

  4. Well, that's explains what is wrong with much education research. Study a mere two schools and give them your own test (note: this is a completely ridiculous way to establish a reliable finding in the first place). Claim that it's all confidential. If anyone wants to check your data or try to replicate research, claim that there's nothing you can do. If anyone finds out which schools they were and shows that those schools aren't actually doing as well as was claimed, refuse to comment even on that, on the basis that "it's all confidential and you just have to trust me."

    This is a great way of coming up with completely unfalsifiable and unscientific claims that no one else in the world can ever check.

    By the way, of course Milgram and Bishop's paper is unpublished -- it is essentially one long peer review of Boaler's work. Peer reviews are never published. Neither are long articles that systematically explain what's wrong with another published article.

    To expect Milgram and Bishop to be a published peer-reviewed paper is absurd, and indeed the very basis of the criticisms is that it is possible to criticize a paper in, say, a blog comment rather than solely by means of publishing yet another paper. If these people took their own arguments seriously, they would criticize Milgram and Bishop only by trying to publish a peer-reviewed article.

  5. You might be interested in Wayne Bishop's two new responses to Boaler, first on the merits of her work and second on her allegations:

  6. James Milgram mentioned how the schools could be found. Either by mistake or deliberate obfuscation, two of the schools are assigned an incorrect environment (it appears they were flipped) and Railside is described as an urban school, when it is not.

    "Number 2 is a nonstarter: why would a researcher bound by confidentiality agreements confirm or deny that Milgram et al identified the schools?"

    If she signed a confidentiality agreement, I believe she did so in error. It was an NSF funded grant, and Stanford's own requirements mandate that she not withhold the school names.

    I found the schools, identified one and explain how to identify the others here:

    Anyone who wants to can identify the schools and check their CST scores.

    1. If there is no legal or ethical requirement to maintain the schools' anonymity (and I can't see why there would be), then why not just name all three schools? Are you afraid of the political repercussions to your career? Are Ms. Boaler's acolytes that powerful?