Wednesday, June 6, 2012

How the Race (to the Top) Was Won (Part 1 of 2)

Race to the Top (RTT), the foremost education policy instrument under the Obama administration, was introduced in 2009 as part of the American Recovery and Reinvestment Act. The first two winning states, Delaware and Tennessee, were announced in the spring of 2010. At that time I wrote a paper about RTT for a policy class but didn't blog about it. Since RTT has continued with more rounds of state competition and a new RTT program for school districts, I hope there's still some relevance in sharing some of what I learned about RTT and what states did to score well on their applications.

Using policy definitions developed by McDonnell and Elmore (1987), RTT is a near-perfect example of an inducement, where money is exchanged for action. Inducements are a simple model:


RTT adds a layer of policymaking to this simple process. In the model below, the state departments of education have all policy instruments at their disposal. The inducement exists between the federal and state policymaking bodies, and not necessarily between the state and the local education authority (LEA). (The new RTT for districts will obviously change this arrangement.) Regardless of the instrument(s) used by the states, the goal, as defined by the U.S. DoE, is for states to "[lead] the way with ambitious yet achievable plans for implementing coherent, compelling, and comprehensive education reform" (U.S. Department of Education, 2010d). Additionally, the DoE clearly states that "Creativity and innovation are rewarded in this competition" (U.S. Department of Education, 2010c. p. 15).


In the ideal RTT scenario, the competition would look like this:
  1. The U.S. DoE rewards the states with the most promising reforms.
  2. Winning states would enact and enforce new education policies.
  3. The effects of the new policies would be measured.
  4. Policies that prove to be successful would be replicated by other states.
There are two significant problems with emphasizing creativity and innovation in RTT. First, creativity and innovation necessitates deviation from proven reforms. You can't be creative by saying, "We're going to do what we know works." Delaware and Tennessee's "innovative" reforms (a label worth questioning) may have helped them win Phase 1 of RTT, but it may some time before we know if the reforms perform as intended. Implicit in the RTT competition is an assumption that established, effective reforms are too few, too expensive, or too difficult to scale, so RTT challenges states to create new reforms that might be cheaper or more easily implemented. This creates a condition where RTT money gets awarded for the potential of a policy, and not its past, proven effectiveness.

The second problem with emphasizing creativity and innovation is that RTT was not structured as an brainstorming, anything goes kind of policymaking process. RTT comes with a detailed scoring rubric, and any state wishing to score well obviously wrote policies to satisfy the rubric. How does that encourage creativity or innovation? In addition, states or districts applying beyond the first round will likely replicate the highest-scoring applications from Phase 1. This means that policies developed as part of RTT Phase 2 and later are likely to be less diverse, less creative, and less innovative, but no more proven.

So how did states manage this balance of creativity versus scoring high on the rubric? Thankfully, the rubric, the applications, and the judges' scorecards are all publicly available, so we can see exactly what each state proposed in their application. The analysis in this post will answer the following questions:
  1. Where in the RTT rubric could states score the most points?
  2. For the portions of the RTT rubric identified in #1, which states scored highest?
  3. For the states identified in #2, what did their application propose and what were the judges' comments?

Race to the Top Scoring

Here is a summary of RTT Phase 1 scoring:

Selection CriteriaPoints PossiblePercent of TotalAverage Score (Points)Average Score (Percent)Standard Deviation (Points)
A. State Success Factors12525907218.03
B. Standards and Assessments701461889.63
C. Data Systems to Support Instruction47933707.04
C. Data Systems to Support Instruction47933707.04
D. Great Teachers and Leaders13828916621.5
E. Turning Around the Lowest-Achieving Schools5010367211.19
F. General5511376812.96
Competitive Preference Priority 2: Emphasis on STEM15311736.73
TOTAL5001003597267.22
(Source: U.S. Department of Education, 2010a, 2010b)

Looking at how the points on the rubric are allocated, it's clear that for any state to do well they'd need to score well on Section D, "Great Teachers and Leaders." It was worth 28 percent of the 500 total points, and we now can see that of all the sections, the fewest points (as a percent) were awarded in this section. That means there was a lot of potential upside here, and we can dig into the details of "Great Teachers and Leaders" to see which states scored best. In the table below you'll find all the subsections of Section D, along with the scores of the eight states (DE, TN, GA, SC, RI, KY, LA, and KS) who had the top score (in bold) in at least one of those subsections.

Selection CriteriaDETNGASCRIKYLAKS
Overall RttT Phase 1 Rank1236891129
(D) "Great Teachers and Leaders" Score86%83%81%82%88%80%89%63%
(1) Providing high-quality pathways for aspiring teachers and principals82%71%70%74%84%94%89%32%
(2) Improving teacher and principal effectiveness based on performance87%91%86%91%94%76%90%56%
  (2)(i) Measuring student growth88%100%48%88%80%84%96%56%
  (2)(ii) Developing evaluation systems85%91%81%89%96%83%91%68%
  (2)(iii) Conducting annual evaluations92%100%96%90%100%58%92%68%
  (2)(iv) Using evaluations to inform key decisions86%87%91%93%93%76%89%45%
(3) Ensuring equitable distribution of effective teachers and principals85%74%87%73%79%73%90%89%
  (3)(i) Ensuring equitable distribution in high-poverty or high-minority schools83%68%93%68%92%72%88%85%
  (3)(ii) Ensuring equitable distribution in hard-to-staff subjects and specialty areas88%82%78%80%60%74%92%94%
(4) Improving the effectiveness of teacher and principal preparation programs81%90%73%81%90%77%86%66%
(5) Providing effective suport to teachers and principals95%75%75%79%84%92%84%80%
(Source: U.S. Department of Education, 2010a)

Just by reading the titles of the subsections above, you can recognize some of the most contentious areas of education policy we've seen the past several years. So not only is this part of the RTT application about high points, it's high-stakes. If the greatest opportunity for improvement in education truly lies in this area, it will be critical to get these policies right. In Part 2 of this post, we'll look at each subsection and the application from the state with the highest score in that area. Some of the policies are sound, but some aren't, and sometimes the judges' comments indicate divergent interpretations of both the application and the rubric.

References

McDonnell, L. M., & Elmore, R. F. (1987). Getting the job done: Alternative policy instruments. Educational Evaluation and Policy Analysis, 9(2), 133-152. Retrieved from http://www.jstor.org/stable/1163726

U.S. Department of Education. (2010a). Detail chart of the Phase 1 scores for each State. Retrieved from http://www2.ed.gov/programs/racetothetop/phase1-applications/phase1-scores-detail.xls

U.S. Department of Education. (2010b). Race to the Top Scoring Rubric Corrected. Washington, D.C.: U.S. Department of Education. Retrieved from http://www2.ed.gov/programs/racetothetop/scoringrubric.pdf

U.S. Department of Education. (2010c, May 27). Race to the Top Program: Guidance and Frequently Asked Questions. Retrieved from http://www2.ed.gov/programs/racetothetop/faq.pdf

U.S. Department of Education. (2010d, April 16). Race to the Top Fund. ED.gov. Retrieved June 6, 2012, from http://www2.ed.gov/programs/racetothetop/index.html

Sunday, June 3, 2012

Open Access Publishing in Mathematics Education

As I write this, the White House petition to require free access over the internet to scientific journal articles arising from taxpayer-funded research is within 150 signatures of the 25,000 needed to guarantee a response from the White House. If you're unfamiliar with the petition, this video concisely explains the issue:



Most of the advances in open access publishing seem to be in the natural and medical sciences -- mathematics, biology, medicine, etc. Much is this is due to a policy by the National Institutes of Health (NIH) that requires publications from NIH-funded research be made available to the public, and the hope of the petition is that a similar policy would spread to other government funding agencies. Given that a significant amount of education research is funded by the National Science Foundation (NSF), education researchers are going to have to think about their open access publishing options should policies require open access to publicly-funded research.

I'm a teacher, not a researcher. Should I care about open access to research?

Yes! Few things annoy me more than the assumption that teachers should not read or take any interest in published education research. I strongly believe that the more researchers think about teachers as part of the audience for their research, the more relevant that research is likely to be and the more quickly we can implement the results. So if you're a teacher and you come across a research article, pay attention. If it makes no sense to you or seems totally irrelevant to you as a teacher, there's probably something wrong. You would be doing a great service to bring that article and your problems with it to the attention of the research community, and researchers should welcome your input. Right now a lot of high-quality research hides from you in for-profit, closed journals, which I believe has allowed the goals of teachers and researchers to drift apart. With open access journals and greater communication via social media, I hope the divide between teachers and researchers can come together with greater frequency.

Current Top-Tier Mathematics Education Research Journals

In mathematics education, the following four journals are often seen as the most prestigious. Let's look at their current publishing policies:

Journal of Research in Mathematics Education - JRME is NCTM's research journal and  probably the top journal in the field. Unfortunately, they have a very author-unfriendly publishing policy:
Assignment of copyright for the article to the National Council of Teachers of Mathematics is required as a condition of publication. After acceptance by JRME, a manuscript may not be published elsewhere, including on the internet, without written permission from NCTM. Each author of a paper published in JRME will receive five complimentary copies of the issue in which the paper appears.
Wow, five complimentary copies? With those I can freely distribute my work to 5 people, or approximately 0.0000002% of worldwide internet users, all of whom can read this measly blog page.

Educational Studies in Mathematics - This journal was founded by Hans Freudenthal in 1968 and is currently published by Springer. Although Springer is an enormous publishing company with a vested interest in a traditional publishing model, they are making efforts to find ways to increase access while still making a profit. Authors have a choice: (a) Transfer their copyright to Springer or (b) opt into Springer's "Open Choice" program, which makes articles freely available on SpringerLink and allows the author to retain copyright and publish under a Creative Commons Attribution License. The catch? Springer charges the author a $3000 fee.

International Journal of Science and Mathematics Education - This journal is also published by Springer and has the same "Open Choice" option as ESM.

Mathematical Thinking and Learning - This journal is published by Routledge, part of the Taylor & Francis Group. Their copyright agreement (PDF) includes the classic language about why authors should transfer copyright (and when I say "classic," I mean old, as in pre-internet): "The transfer of copyright from author to publisher must be clearly stated in writing to enable the publisher to assure maximum dissemination of the author's work." The thought that putting written work in an expensive journal distributed to a relatively small number of people and institutions "assures" a wider distribution than the open internet is plainly laughable. The copyright agreement does throw a few bones the author's way with these three exceptions:
  1. Authors can copy their own article for their use in classrooms.
  2. Authors can reuse the work in a textbook they might author.
  3. Authors can copy their work for internal distribution within their institution.
Exception #2 is not to be overlooked -- some publishers will not grant that exception. I know one researcher who wanted to re-use an article he'd written as a dissertation chapter and was denied, forcing him to start the research and writing anew.

Current Open Access Journals

Assuming the journals above don't convert themselves to an open access publishing model (one without $3000 fees), the most immediate option for publishing under an open access mandate would be in an journal that's already open access. No, these journals don't have the history or prestige that the above journals have, but I do get the sense that tools like Google Scholar are making the journal name less relevant than in the past. Many of these journals have emerged in just the past 5-10 years, and I'll limit the list below to those that publish primarily in English and appear to receive submissions from U.S.-based researchers. All the journals found below were indexed in the Directory of Open Access Journals (DOAJ).

I divide open access journals into two main camps -- those where the publisher takes copyright, and those where the author retains copyright. Given the open nature of these journals, I imagine some negotiation about copyright would be very acceptable, particularly if using something like the SPARC Author Addendum.

Author Copyright

International Journal for Mathematics Teaching and Learning - IJMTL is a joint publishing effort between Plymouth University, UK, and the College of Nyiregyháza, Hungary. Their author guidelines say nothing about copyright, but authors are expected to do their own copy editing and formatting of their final article. The articles I looked at made no mention of copyright or licensing, so I assume authors have retained copyright and have the right to assign a Creative Commons license if they wish.

Journal of Statistics Education - JSE has been published by the American Statistical Association since 1993 and clearly indicates the author's copyright on each article.

Journal of Urban Mathematics Education - JUME is edited by David Stinson at Georgia State and appears to be one of the higher-quality open access efforts, publishing articles by William Tate, Rico Gutstein, Megan Staples, Jere Confrey, Michael Battista, Jo Boaler, and others. Authors retain copyright with first publication rights granted to JUME.

Numeracy - This journal specializes in quantitative literacy and is hosted by the University of South Florida. Authors retain copyright under a Creative Commons Attribution 3.0 license. There are no publication charges.

Philosophy of Mathematics Education Journal - Edited by Paul Ernest at the University of Exeter, UK, this journal has existed since 1990 and the copyright notice reads: "All materials published herein remain copyright of the named author(s), or of the editor if unattributed. Permission is given to freely copy the journal contents on a not-for-profit basis, provided full credit is given to the author and the journal." This sounds much like a "legal lite" interpretation of a Creative Commons Attribution - Non Commercial license, although there's no explicit mention of derivative works.

Pythagoras - This is the journal of the Association for Mathematics Education of South Africa, in existence since 1980. The copyright notice on a recent article indicates that the author retains the copyright and the work is licensed under a Creative Commons Attribution license.

Technology Innovations in Statistics Education - TISE is edited by Robert Gould at UCLA and authors retain copyright with publication under a Creative Commons Attribution - Non Commercial - Share Alike license.

The Teaching of Mathematics - This is published by the Mathematical Society of Serbia and makes no mention of copyright on their site or on published articles, so I assume copyright would stay with the author.

Publisher Copyright

Contemporary Issues in Technology and Teacher Education - Part of this journal includes articles about technology and mathematics education, with editorial review organized by AMTE. The publisher retains copyright to published articles.

International Electronic Journal of Mathematics Education - IEJME is published by Gökkuşağı, a Turkish publisher, and dates back to 2006. Their author guidelines don't say anything about who retains copyright of the published articles, but the journal itself indicates the copyright is held by Gökkuşağı.

Journal of Mathematics Education at Teachers College - This is a good looking journal with submissions from some well-known authors. The submission guideline page says nothing about copyright, but the journal itself and each article claims copyright for the publisher.

Journal of STEM Education - This journal requires the author pay a fee ($395 for the first 8 pages and author's bio, then $35 for each additional page) and also requires the transfer of copyright.

Statistics Education Research Journal - This international journal appears to include a wide variety of content but requires authors to transfer copyright.

The Mathematics Educator - TME is a student-produced journal from the University of Georgia and was first published in 1990. Despite the maturity of this journal, copyright is very unclear -- the site says nothing about transferring copyright, and the journal itself claims copyright for the publisher in the front matter, but nothing on articles themselves, and articles are available individually.

The Mathematics Enthusiast - Formerly known as the Montana Mathematics Enthusiast, this journal says nothing on the site about transferring copyright, but the articles themselves indicate a copyright held by the publisher.

Conclusion

It's difficult for me to predict exactly how an open access mandate would affect current journals. Those top four journals, because of their prestige, might not change a thing and hope to get submissions from authors who aren't funded by major federal agencies. Some of the open access journals are obscure now and will likely stay that way, at least to U.S. researchers. What I'd like to see is some of the currently closed "second-tier" journals open themselves. Some, like The Journal of Mathematical Behavior (an Elsevier publication), isn't a likely candidate. For a journal like For the Learning of Mathematics, opening access might be easier. (FLM already has a FAQ including the question, "Can I reprint an FLM article on my web site / anthology / lunch box?" with the simple answer, "If you are interested in reprinting articles that appear in FLM, please contact the managing editor.") Some journals already have policies that would seem to pull them in the direction of open access. Teaching Statistics, for example, is closed but allows authors to retain copyright so long as they give an exclusive license to publish to the journal. I don't know what good it is to have a copyright but no right to publish, but some tweaking of those policies might turn such a journal into something open.

Then again, maybe math education researchers will gravitate towards current large open access repositories. The article Number Concepts without Number Lines in an Indigenous Group of Papua New Guinea caught my eye not just for its content, but the fact it is published in PLoS ONE, a journal that's flourished publishing open access science and medicine content, not necessarily education-related articles. But there's no reason PLoS ONE can't expand its scope, something it's likely to do if new governement open access policies demand more open publications in more content areas.