- Empirical Critic - Finds faults with the method. Finds limits to generalizability. Finds instances of shallowness in interpretation, obliviousness to context. Finds weakness in internal and external validity.
- Theoretical Critic - Identifies theoretical givens. Challenges the chain of reasoning by which the claimed findings are related to broader theoretical issues. Offers explanations that proponents of rival theories would give to the claimed findings.
- Literary Truth Squad - Identifies claims in which cited literature is used as evidence. Succinctly restates the claim. Tracks down the cited literature and evaluates whether there was warrant for treating the cited literature as evidence for the claim that was made.
- Worth-Finder - Beyond the identifiable faults of a work, what can still be gained from it? What good did it do? What is the most memorable about it? Are there implications that were unreported by the author?
- Sociologist of Knowledge - Whose work was cited? Whose wasn't? What disciplinary and theoretical traditions are being followed? What ideological commitments are being made by the author(s)? Are there geographical and institutional biases in the citation pattern? Prospectively, who used this work, and how?
This week I'm the Literary Truth Squad (which always reminds me of Police Squad) so my reading of the article is interrupted by a lot of trips to Google Scholar and readings of abstracts. Warranted or not, I've gained a bit of a reputation for having a good grasp of the literature. But since this is a science education article and not math ed, I'm having to look up everything. It's time-consuming and for a 53-page article I'm going to have to pick-and-choose where to look more closely. Sometimes I can guess it's a relevant reference just from the title of the article, and that speeds up the process. Sometimes the reference isn't a published article, but rather a paper presented at a conference. Those can be difficult to find, if they're online at all. Good conference presentations often end up as articles in journals, but the conference paper gets cited anyway and there's no real way yet to automatically refer the paper reference to a later paper.
This process is nothing like casual, lay-on-the-couch reading, but that's the point. It forces me to read with a purpose and be accountable to my classmates, each of whom will be reading equally carefully with their own roles in mind. When I was teaching I struggled having students work in groups until I helped group members identify roles and responsibilities and how they should work together. This is very similar and adds a nice structure to our in-class discussions, so long as we don't let pride in our work get in the way of brevity. I may look at all 78 cited resources but there's probably little need to inform my classmates of more than just a few.
Now...back to truth squading!