New Research Uncovers Fresh Trouble for VAM Evaluations

As more and more schools implement various forms of Value-Added method (VAM) evaluation systems, we are learning some disturbing things about how reliable these methods are.

Education Week's Stephan Sawchuk, in "'Value-Added' Measures at Secondary Level Questioned," explains that value-added statistical modeling was once limited to analyzing large sets of data. These statistical models projected students' test score growth, based on their past performance, and thus estimated a growth target. But, now 30 states require teacher evaluations to use student performance, and that has expanded use of algorithms for high-stakes purposes. Value-added estimates are now being applied to secondary schools, even though the vast majority of research on their use has been limited to elementary schools.

Sawchuk reports on two major studies that should slow this rush to evaluate all teachers with experimental models. This month, Douglas Harris will be presenting "Bias of Public Sector Worker Performance Monitoring." It is based on a six years of Florida middle school data on 1.3 million math students.

Harris divides classes into three types, remedial, midlevel, and advanced. After controlling for tracking, he finds that between 30 to 70% of teachers would be placed in the wrong category by normative value-added models. Moreover, Harris discovers that teachers who taught more remedial classes tended to have lower value-added scores than teachers who taught mainly higher-level classes. "That phenomenon was not due to the best teachers' disproportionately teaching the more-rigorous classes, as is often asserted. Instead, the paper shows, even those teachers who taught courses at more than one level of rigor did better when their performance teaching the upper-level classes was compared against that from the lower-level classes."

[readon2 url="http://blogs.edweek.org/teachers/living-in-dialogue/2012/11/john_thompson_new_research_unc.html"]Continue reading...[/readon2]

Gross miscalculation

On his Sociological Eye on Education blog for the Hechinger Report, Aaron Pallas writes that in April 2011, Carolyn Abbott, who teaches mathematics to seventh- and eighth-graders at the Anderson School, a citywide gifted-and-talented school in Manhattan, received startling news. Her score on the NYC Department of Education's value-added measurement indicated only 32 percent of seventh-grade math teachers and 0 percent of eighth-grade math teachers scored worse than she. According to calculations, she was the worst eighth-grade math teacher in the city, where she has taught since 2007.

Here's the math: After a year in her classroom, her seventh-grade students scored at the 98th percentile of city students on the 2009 state test. As eighth-graders, they were predicted to score at the 97th percentile. Yet their actual performance was at the 89th percentile of students across the city, a shortfall -- 97th percentile to 89th percentile -- that placed Abbott near the rock bottom of 1,300 eighth-grade mathematics teachers. Anderson is an unusual school; the material on the state eighth-grade math exam is taught in the fifth or sixth grade. "I don't teach the curriculum they're being tested on," Abbott explained. "It feels like I'm being graded on somebody else's work." The math she teaches is more advanced, culminating in high-school level work and the New York State's Regents exam in Integrated Algebra. Of her students taking the Regents in January, all passed with flying colors, more than a third achieving a perfect score of 100.

This summer, the state will release a new iteration of the Teacher Data Reports. For Abbott, these will be a mere curiosity. She has decided to leave the classroom, and is entering the Ph.D. program in mathematics at the University of Wisconsin-Madison this fall.

[readon2 url="http://eyeoned.org/content/the-worst-eighth-grade-math-teacher-in-new-york-city_326/"]Read more...[/readon2]

To Understand The Impact Of Teacher-Focused Reforms, Pay Attention To Teachers

You don’ t need to be a policy analyst to know that huge changes in education are happening at the state- and local-levels right now – teacher performance pay, the restriction of teachers’ collective bargaining rights, the incorporation of heavily-weighted growth model estimates in teacher evaluations, the elimination of tenure, etc. Like many, I am concerned about the possible consequences of some of these new policies (particularly about their details), as well as about the apparent lack of serious efforts to monitor them.

Our “traditional” gauge of “what works” – cross-sectional test score gains – is totally inadequate, even under ideal circumstances. Even assuming high quality tests that are closely aligned to what has been taught, raw test scores alone cannot account for changes in the student population over time and are subject to measurement error. There is also no way to know whether fluctuations in test scores (even fluctuations that are real) are the result of any particular policy (or lack thereof).

[readon2 url="http://shankerblog.org/?p=2434"]Continue Reading[/readon2]