The Arbitrary Albatross: Standardized Testing and Teacher Evaluation

On Chicago's streets and Hollywood's silver screens, education reform has been cast as a false dilemma between students and teachers. Reputable actresses and liberal mayors have both fallen prey. At the center of this drama lie teacher evaluations. A linchpin of the debate, they weigh especially heavily around the necks of educators like me.

Think: Shaky Foundation

With the arrival of spring, testing season is now upon us: America's new national pastime. I believe student results from standardized tests should not be used to evaluate teachers because the data are imprecise and the effects are pernicious. Including such inaccurate measures is both unfair to teachers and detrimental to student learning.

As a large body of research suggests, standardized test data are imprecise for two main reasons. First, they do not account for individual and environmental factors affecting student performance, factors over which teachers have no control. (Think: commitment, social class, family.) Second, high-stakes, one-time tests increase the likelihood of random variation so that scores fluctuate in arbitrary ways not linked to teacher efficacy. (Think: sleep, allergies, the heartache of a recent breakup.)

High-stakes assessments are also ruinous to student learning. They encourage, at least, teaching to the test and, at most, outright cheating. This phenomenon is supported by Campbell's law, which states statistics are more likely to be corrupted when used in making decisions, which in turn corrupts the decision making process itself. (Think: presidential campaigns.)

As a teacher, if my livelihood is based on test results, then I will do everything possible to ensure high marks, including narrowing the curriculum and prepping fiercely for the test. The choice between an interesting project and a paycheck is no choice at all. These are amazing disincentives to student learning. Tying teachers' careers to standardized tests does not foster creative, passionate, skillful young adults. It does exactly the opposite.

[readon2 url="http://www.edutopia.org/blog/standardized-testing-and-teacher-evaluation-aaron-pribble"]Continue reading...[/readon2]

Research-Based Options for Education Policy Making

The first in a new series of two-page briefs summarizing the state of play in education policy research offers suggestions for policymakers designing teacher evaluation systems.

The paper is written by Dr. William Mathis, managing director of the National Education Policy Center, housed at the University of Colorado Boulder School of Education.

Teachers are important, and policies mandating high-stakes evaluations of teachers are at the forefront of popular school reforms. Today’s dominant approach labels teachers as effective or ineffective based in large part on a statistical analysis of students’ test-score performance. Teachers judged effective are rewarded, and those found ineffective are sanctioned.

While such summative evaluations can be useful, lawmakers should be wary of approaches based in large part on test scores: the error in the measurements is large—which results in many teachers being incorrectly labeled as effective or ineffective;1 relevant test scores are not available for the students taught by most teachers, given that only certain grade levels and subject areas are tested; and the incentives created by high-stakes use of test scores drive undesirable teaching practices such as curriculum narrowing and teaching to the test.

Summative initiatives should also be balanced with formative approaches, which identify strengths and weaknesses of teachers and directly focus on developing and improving their teaching. Measures that de-emphasize test scores are more labor intensive but have far greater potential to enrich instruction and improve education.

The paper goes on to give some key research points and advice for policy makers

• If the objective is improving educational practice, formative evaluations that guide a teacher’s improvement provide greater benefits than summative evaluations.
• If the objective is to improve educational performance, outside-school factors must also be addressed. Teacher evaluation cannot replace or compensate for these much stronger determinants of student learning. The importance of these outside-school factors should also caution against policies that simplistically attribute student test scores to teachers.
• The results produced by value-added (test-score growth) models alone are highly unstable. They vary from year to year, from classroom to classroom, and from one test to another. Substantial reliance on these models can lead to practical, ethical and legal problems.
• High-stakes evaluations based in substantial part on students’ test scores narrow the curriculum by diminishing or pushing out non-tested subjects, knowledge, and skills.
• Teacher evaluation systems necessarily involve trade-offs, and specific design choices are controversial, so it is important to involve all key stakeholders in system design or selection.
• To be successful, schools must invest in their teacher evaluation systems. An adequate number of highly trained evaluators must be available.
• Given the wide variety of teacher roles and the many factors that influence learning that are outside the control of the teacher, a wide variety of measures of teacher effectiveness is also indicated. By diversifying, the weakness of any single measure is offset by the strengths of another.
• High-quality research on existing evaluative programs and tools should inform the design of teacher evaluation systems. States and districts should investigate balanced models such as PAR and the Danielson Framework, closely examine the evidence concerning strengths and weaknesses of each model, and never attach high-stakes consequences to teachers which the evidence cannot validly support.

The paper can be read in full below

Research-Based Options for Education Policy Making

Charters spend more on admin, less on class

We observed that SB316 delayed, by 12 months, the requirement passed in the budget to rank schools based upon their classroom spending. As was noted at the time, this was an ill conceived, ill considered plan. As that plan now hits the slow track, research emerges that charter schools spend less in the classroom than traditional public schools.

One of the most frequent criticisms put to traditional public schools is that they waste money on administrative bloat, instead of channeling more funding where it belongs—the classroom. A much leaner and classroom-centered model, some say, can be found in charter schools, because of their relative freedom from stifling bureaucracy.

A new study, however, concludes that this hypothesis has it exactly wrong.

The study, released by the National Center for the Study of Privatization in Education, at Teachers College, Columbia University, examines school spending in Michigan and concludes that charter schools spend more per-pupil on administration and less on instruction than traditional public schools, even when controlling for enrollment, student populations served, and other factors.

Researchers David Arsen, of Michigan State University, and Yongmei Ni, of the University of Utah, found that charters spend $774 more per pupil on administration, and $1,140 less on instruction, than do traditional publics. To come up with their estimates, the authors analyze the level and source of funding for charters and traditional publics, and how they spend money, breaking it out by function. They then use a statistical method known as regression analysis to control for factors that could skew their comparisons of spending on administration and instruction in various schools.

Of the extra $774 that charters devote to administration, $506 went to general administrative services, such as the costs of charter school boards, or the fees of the organizations managing the school.

While Arsen and Ni don't examine in depth the causes of charters' relatively low instructional spending, they speculate that a couple factors could be at work. An obvious one is that more than 80 percent of traditional public schools' spending goes to personnel costs, mostly salaries and benefits—which would presumably drive instructional costs up. Charters, on average, pay lower salaries for teachers with similiar credentials to those hired by traditional publics, and also employ a less experienced and less costly teaching force, the authors say, which would keep instructional costs down.

HEre's the paper in question for your review.

Is Administration Leaner in Charter Schools? Resource Allocation in Charter and Traditional Public Schools

Research doesn’t back up key ed reforms

Via the Washington Post

There is no solid evidence supporting many of the positions on teachers and teacher evaluation taken by some school reformers today, according to a new assessment of research on the subject.

The Education Writers Association released a new brief that draws on more than 40 research studies or research syntheses, as well as interviews with scholars who work in this field.

You can read the entire brief (written by Education Week assistant editor Stephen Sawchuk), but here are the bottom-line conclusions of each section:

Q) Are teachers the most important factor affecting student achievement?

A) Research has shown that the variation in student achievement is predominantly a product of individual and family background characteristics. Of the school factors that have been isolated for study, teachers are probably the most important determinants of how students will perform on standardized tests.

Q) Are value-added estimations reliable or stable?

A) Value-added models appear to pick up some differences in teacher quality, but they can be influenced by a number of factors, such as the statistical controls selected. They may also be affected by the characteristics of schools and peers. The impact of unmeasured factors in schools, such as principals and choice of curriculum, is less clear.

Q) What are the differences in achievement between students who have effective or ineffective teachers for several years in a row?

A) Some teachers produce stronger achievement gains among their students than others do. However, estimates of an individual teacher’s effectiveness can vary from year to year, and the impact of an effective teacher seems to decrease with time. The cumulative effect on students’ learning from having a succession of strong teachers is not clear.

Q) Do teacher characteristics such as academic achievement, years of experience, and certification affect student test scores?

A) Teachers improve in effectiveness at least over their first few years on the job. Characteristics such as board certification, and content knowledge in math sometimes are linked with student achievement. Still, these factors don’t explain much of the differences in teacher effectiveness overall.

Q) Does merit pay for teachers produce better student achievement or retain more-effective teachers?

A) In the United States, merit pay exclusively focused on rewarding teachers whose students produce gains has not been shown to improve student achievement, though some international studies show positive effects. Research has been mixed on comprehensive pay models that incorporate other elements, such as professional development. Scholars are still examining whether such programs might work over time by attracting more effective teachers.

Q) Do students in unionized states do better than students in states without unions?

A) Students tend to do well in some heavily unionized states, but it isn’t possible to conclude that it is the presence or absence of unions that cause that achievement.

What Studies Say About Teacher Effectiveness

Are Ohio's Republicans threatening to pass SB5 again if it is defeated?

After the SB5 debate last night, State Sen Niehaus was asked "If you look at the polls they seem to indicate that voters, even though they are polling against SB5 support portions of it, are you prepared in the senate to move forward worth separate legislation if this fails next month?"

State Sen. Neihaus responded, "Well I think Senator Faber spoke to it, I mean the issues that drove the necessities of having SB5 or issue 2 haven't changed. were on an unsustainable path we cannot continue paying the cost of local government level or state level so those fundamental factors have not changed so we have not prepared anything at this point, given the fact that the situation will remain exactly the same on Nov. 10, then we certainly have an obligation to the voters and residents of Ohio to find a way to make sure, to ensure the continuation of local government services in a reasonable way"

You can fast forward to the 2minute 37 mark for the exchange

I cannot imagine anything that would great more chaos and anger than thwarting the will of Ohio's voters.

What Studies Say About Teacher Effectiveness

The Education Writers Association has produced a brief on teacher effectiveness, which you can read in full here.

it’s important for journalists and others to understand what is known about the topic so far, and what remains unsettled or unknown. This research brief does not synthesize all the studies in this highly technical field. But it does aim to improve the accuracy and clarity of reporting by exploring what the research says about timely questions surrounding the complex topic of teacher effectiveness.

The brief is organized around several prevailing questions about teacher effectiveness in K-12 education. For each question, we’ve reviewed some of the most-important research, identifying key findings and tension points. (Citations in the text refer to a list of sources in the bibliography.) At the end of each section, we present a bottom-line summary of the research.

Nearly all of the studies cited here rely on the use of student test scores as a proxy for learning, a research practice that remains hotly debated. A full discussion of the value of standardized testing lies outside the scope of this paper, but we begin from the same assumption as many scholars: that standardized tests measure important aspects of student learning, but not the full breadth and depth of what students should know and be able to do.

The brief draws on a review of over 40 specific research studies or research syntheses, as well as interviews with scholars who have used primarily quantitative research methods to analyze the relationships between teachers, their attributes, and student achievement.

Here's a summary of the brief

Are teachers the most important factor affecting student achievement?

It can be said:
Research has shown that the variation in student achievement is predominantly a product of individual and family background characteristics. Of the school factors that have been isolated for study, teachers are probably the most important determinants of how students will perform on standardized tests.

Are value-added estimations reliable or stable?

It can be said:
Value-added models appear to pick up some differences in teacher quality, but they can be influenced by a number of factors, such as the statistical controls selected. They may also be affected by the characteristics of schools and peers. The impact of unmeasured factors in schools, such as principals and choice of curriculum, is less clear.

Does merit pay for teachers produce better student achievement or retain more-effective teachers?

It can be said:
In the United States, merit pay exclusively focused on rewarding teachers whose students produce gains has not been shown to improve student achievement, though some international studies show positive effects. Research has been mixed on comprehensive pay models that incorporate other elements, such as professional development. Scholars are still examining whether such programs might work over time by attracting more effective teachers.

Do students in unionized states do better than students in states without unions?

It can be said:
Students tend to do well in some heavily unionized states, but it isn’t possible to conclude that it is the presence or absence of unions that cause that achievement.