On Teacher Evaluation: Slow Down And Get It Right

One of the primary policy levers now being employed in states and districts nationwide is teacher evaluation reform. Well-designed evaluations, which should include measures that capture both teacher practice and student learning, have great potential to inform and improve the performance of teachers and, thus, students. Furthermore, most everyone agrees that the previous systems were largely pro forma, failed to provide useful feedback, and needed replacement.

The attitude among many policymakers and advocates is that we must implement these systems and begin using them rapidly for decisions about teachers, while design flaws can be fixed later. Such urgency is undoubtedly influenced by the history of slow, incremental progress in education policy. However, we believe this attitude to be imprudent.

The risks to excessive haste are likely higher than whatever opportunity costs would be incurred by proceeding more cautiously. Moving too quickly gives policymakers and educators less time to devise and test the new systems, and to become familiar with how they work and the results they provide.

Moreover, careless rushing may result in avoidable erroneous high stakes decisions about individual teachers. Such decisions are harmful to the profession, they threaten the credibility of the evaluations, and they may well promote widespread backlash (such as the recent Florida lawsuits and the growing “opt-out” movement). Making things worse, the opposition will likely “spill over” into other promising policies, such as the already-fragile effort to enact the Common Core standards and aligned assessments.

[readon2 url="http://shankerblog.org/?p=8358"]Continue reading...[/readon2]

How Do High-Performing Nations Evaluate Teachers?

Who decides if a teacher is effective and how is that determination made? School systems across the United States are struggling to answer that question as they try to design and implement teacher evaluation systems that are fair and accurate. It’s no easy task and is not limited to public schools in this country. School systems around the world are tackling the same issue and are finding consensus among education stakeholders to be elusive.

Teacher evaluations were the main topic of discussion at the 2013 International Summit on the Teacher Profession (ISTP) Summit held last week in Amsterdam. Now in its third year, the ISTP brought together leaders from teacher unions and education ministries to discuss issues around teacher quality, specifically the criteria used to determine teacher effectiveness and its purpose.

In most nations, teacher evaluation systems are essentially a “work in progress,” says Andreas Schleicher of the Organization for Economic Cooperation and Development (OECD). Schleicher, who attended the ISTP, is the principal author of the study that was presented at the summitt. The report, Teachers for the 21st Century: Using Evaluations to Improve Teaching, takes a look at how different nations are tackling this thorny issue (or not tackling it) and identifying specific models that appear to work – that is, have buy-in from key stakeholders and can point to demonstrable results in student achievement. Because consensus is so frustratingly elusive, most nations are treading carefully, although there is widespread acknowledgement that improved evaluation systems have to be on the menu of education policy reforms.

Of the 28 countries surveyed in the OECD report, 22 have formal policy frameworks in place at the national level to regulate teacher evaluations. The six education systems that do not have such frameworks include Denmark, Finland, Iceland, Norway and Sweden, but teachers in the countries still received professional feedback. In Denmark, for example, teachers receive feedback from their school administrators once a year. In Norway, teacher-appraisal policies are designed and implemented at the local or school level. In Iceland, evaluation is left to the discretion of individual schools and school boards.

[readon2 url="http://neatoday.org/2013/03/25/how-do-high-performing-nations-evaluate-teachers/"]Continue reading...[/readon2]

Do Different Value-Added Models Tell Us the Same Things?

Via

Highlights

• Statistical models that evaluate teachers based on growth in student achievement differ in how they account for student backgrounds, school, and classroom resources. They also differ by whether they compare teachers across a district (or state) or just within schools.
• Statistical models that do not account for student background factors produce estimates of teacher quality that are highly correlated with estimates from value-added models that do control for student backgrounds, as long as each includes a measure of prior student achievement.
• Even when correlations between models are high, different models will categorize many teachers differently.
• Teachers of advantaged students benefit from models that do not control for student background factors, while teachers of disadvantaged students benefit from models that do.
• The type of teacher comparisons, whether within or between schools, generally has a larger effect on teacher rankings than statistical adjustments for differences in student backgrounds across classrooms.

Introduction

There are good reasons for re-thinking teacher evaluation. As we know, evaluation systems in most school districts appear to be far from rigorous. A recent study showed that more than 99 percent of teachers in a number of districts were rated “satisfactory,” which does not comport with empirical evidence that teachers differ substantially from each other in terms of their effectiveness. Likewise, the ratings do not reflect the assessment of the teacher workforce by administrators, other teachers, or students.

Evaluation systems that fail to recognize the true differences that we know exist among teachers greatly hamper the ability of school leaders and policymakers to make informed decisions about such matters as which teachers to hire, what teachers to help, which teachers to promote, and which teachers to dismiss. Thus it is encouraging that policymakers are developing more rigorous evaluation systems, many of which are partly based on student test scores.

Yet while the idea of using student test scores for teacher evaluations may be conceptually appealing, there is no universally accepted methodology for translating student growth into a measure of teacher performance. In this brief, we review what is known about how measures that use student growth align with one another, and what that agreement or disagreement might mean for policy.

[readon2 url="http://www.carnegieknowledgenetwork.org/briefs/value-added/different-growth-models/"]Continue reading...[/readon2]

After the evaluations binge, the hangover

You don't have to search far, or wide, to find articles, papers, and studies critical of corporate education reformers push for rigid test based teacher evaluations of the kind currently being deployed in Ohio. Our document archive is full of them. But it is unusual to read a paper published by a right wing think tank with a reputation for being anti-teacher, that raises many of the same points teachers themselves have been raising about the headlong rush to implement corporate education reform principles in the area of teacher evaluations.

But that's exactly what the American Enterprise Institute (AEI) have just done wit ha paper titled "The Hangover: Thinking about the Unintended Consequences of the Nation’s Teacher Evaluation Binge". The paper opens with a warning that the recent pushes might have been too much, too soon, and gone too far

Yet the recent evaluation binge is not without risks.

By nature, education policymaking tends to lurch from inattention to overreach. When a political moment appears, policymakers and advocates rush to take advantage as quickly as they can, knowing that opportunities for real change are fleeting. This is understandable, and arguably necessary, given the nature of America’s political system. But headlong rushes inevitably produce unintended consequences—something akin to a policy hangover as ideas move from conception to implementation.

Welcome to teacher evaluation’s morning after.

the Paper discusses a number of problematic area that will be familiar to JTF readers

Flexibility versus control: There is a temptation to prescribe and legislate details of evaluations to ensure rigor and prevent evaluations from being watered down in implementation. But overly prescriptive policies may also limit school autonomy and stifle innovation that could lead to the development of better evaluations.

Evaluation in an evolving system: Poorly designed evaluation requirements could pose an obstacle to blended learning and other innovative models in which it is difficult or impossible to attribute student learning gains in a particular subject to a particular teacher.

Purposes of evaluations: New evaluation systems have been sold as a way both to identify and dismiss underperforming teachers and to provide all teachers with useful feedback to help them improve their performance. But there are strong tensions between these purposes that create trade-offs in evaluation system design.

Evaluating teachers as professionals: Advocates argue that holding teachers responsible for their performance will bring teaching more in line with norms in other fields, but most professional fields rely on a combination of data and managerial judgment when making evaluation and personnel decisions, and subsequently hold managers accountable for those decisions, rather than trying to eliminate subjective judgments as some new teacher evaluation systems seek to do.

Take one look at this evaluation framework that has been inspired by the Ohio legislature and one can see how prescriptive Ohio's teacher evaluation has become.

Ohio has also fallen into many of the traps this paper highlights. The failure to consider team worked teaching, a lack of focus and funding for professional development, and a lack of resources for administrators to provide adequate feedback, to name just a handful.

AEI offer some useful recommendations, some of which might be too late to implement in Ohio

Recognizing these tensions and trade-offs, this paper offers several policy recommendations:

• Be clear about the problems new evaluation systems are intended to solve.
• Do not mistake processes and systems as substitutes for cultural change.
• Look at the entire education ecosystem, including broader labor-market impacts, pre- and in-service preparation, standards and assessments, charter schools, and growth of early childhood education and innovative school models.
• Focus on improvement, not just deselection.
• Encourage and respect innovation.
• Think carefully about waivers versus umbrellas.
• Do not expect legislation to do regulation’s job.
• Create innovation zones for pilots—and fund them.

One might find it gratifying to read reasoned words of caution regarding corporate education reforms from some of the very people responsible for pushing them in the first place, and we can only hope we see more of it. But, it is hard not to suspect that this is the slow dawning of realization that is being drawn from the very real evidence of on-going struggles and failures in corporate education reform policies now being seen across the state and the country.

The Hangover: Thinking about the Unintended Consequences of the Nation’s Teacher Evaluation Binge

Research-Based Options for Education Policy Making

The first in a new series of two-page briefs summarizing the state of play in education policy research offers suggestions for policymakers designing teacher evaluation systems.

The paper is written by Dr. William Mathis, managing director of the National Education Policy Center, housed at the University of Colorado Boulder School of Education.

Teachers are important, and policies mandating high-stakes evaluations of teachers are at the forefront of popular school reforms. Today’s dominant approach labels teachers as effective or ineffective based in large part on a statistical analysis of students’ test-score performance. Teachers judged effective are rewarded, and those found ineffective are sanctioned.

While such summative evaluations can be useful, lawmakers should be wary of approaches based in large part on test scores: the error in the measurements is large—which results in many teachers being incorrectly labeled as effective or ineffective;1 relevant test scores are not available for the students taught by most teachers, given that only certain grade levels and subject areas are tested; and the incentives created by high-stakes use of test scores drive undesirable teaching practices such as curriculum narrowing and teaching to the test.

Summative initiatives should also be balanced with formative approaches, which identify strengths and weaknesses of teachers and directly focus on developing and improving their teaching. Measures that de-emphasize test scores are more labor intensive but have far greater potential to enrich instruction and improve education.

The paper goes on to give some key research points and advice for policy makers

• If the objective is improving educational practice, formative evaluations that guide a teacher’s improvement provide greater benefits than summative evaluations.
• If the objective is to improve educational performance, outside-school factors must also be addressed. Teacher evaluation cannot replace or compensate for these much stronger determinants of student learning. The importance of these outside-school factors should also caution against policies that simplistically attribute student test scores to teachers.
• The results produced by value-added (test-score growth) models alone are highly unstable. They vary from year to year, from classroom to classroom, and from one test to another. Substantial reliance on these models can lead to practical, ethical and legal problems.
• High-stakes evaluations based in substantial part on students’ test scores narrow the curriculum by diminishing or pushing out non-tested subjects, knowledge, and skills.
• Teacher evaluation systems necessarily involve trade-offs, and specific design choices are controversial, so it is important to involve all key stakeholders in system design or selection.
• To be successful, schools must invest in their teacher evaluation systems. An adequate number of highly trained evaluators must be available.
• Given the wide variety of teacher roles and the many factors that influence learning that are outside the control of the teacher, a wide variety of measures of teacher effectiveness is also indicated. By diversifying, the weakness of any single measure is offset by the strengths of another.
• High-quality research on existing evaluative programs and tools should inform the design of teacher evaluation systems. States and districts should investigate balanced models such as PAR and the Danielson Framework, closely examine the evidence concerning strengths and weaknesses of each model, and never attach high-stakes consequences to teachers which the evidence cannot validly support.

The paper can be read in full below

Research-Based Options for Education Policy Making

Kasich pushes public ed to precipice

Already faced with having to deal with massive budget cuts created by the Governor's raid on school funding to balance his own budget, he has now asked all departments, including the Department of Education to provide budget scenarios that include a further 10% cut

State agency heads will be lucky if they get to keep current funding levels when Gov. John Kasich rolls out the next state budget in the spring.

That's the tone being set by a budget guidance document released Monday by Kasich's administration that asks state agencies to plan a pair of scenarios for the 2014-15 budget -- one where they see no growth in funding and a second in which agencies are hit with a 10 percent cut in general revenue funds.

The Governor's own Office of Budget and Management ran the number to discern the impact. It would be ironic if the Governor killed his own education policies via his own fiscal policy.

In areas of academic improvement, the results are dire

• A 10% cut could jeopardize Ohio's ESEA flexibility waiver by reducing services by up to 4,000 hours and not providing the additional support to priority and focus schools required. The 16 fiscal agents (ESCs) that deliver the Ohio Improvement Process for districts in school improvement already do not have the capacity to support all districts in need of support.
• A 10% cut to Ohio's Alternative Education Grant programs could result in as many as 29 programs closing resulting in increased drop-out rates and decreased graduation rates. Students who drop out often result in long term costs to the state through various social service and remediation programs. This funding has experienced a reduction of funding of a little over 40% to what was originally budgeted for in FY09

On Curriculum and assessment, the cuts would end the efforts to increase academic rigor and hinder the newly legislated (and already unfunded) 3rd grade reading guarantee.

• A 10% cut to funding for Content Standards and Model Curricula would jeopardize ODE's ability to manage all work required in statute in this area, such as embedding career connections into Ohio's model curricula. Additional cuts could also jeopardize Ohio's ESEA flexibility waiver by significantly hindering ODE's ability to provide technical assistance to districts in implementing the new academic content standards that begin in the 2013-14 school year. Reductions will jeopardize planned professional development supporting best practices instructional strategies, infusing technology into instruction, and supporting instruction for diverse learners in implementing the new State Learning Standards. This set-aside has experienced a cut of over 28% to amounts originally budgeted for in FY09.
• A 10% cut to Assessment Contracts and & Administration would result in statutory noncompliance for Ohio's student assessment system and would jeopardize Ohio's ESEA flexibility waiver because the funding would not allow the new generation of assessments being developed through the Partnership for Assessment of Readiness for College and Careers (PARCC) and state-developed assessments for science and social studies to begin in the 2014-15 school year.
• A 10% cut to funding for the Kindergarten Readiness Assessment could hinder the release and administration of this literacy assessment which is critical to successful early childhood learning and the implementation of the Third Grade Reading Guarantee included in SB 316.

Cuts would also jeopardize district data collection and analysis, something ODE is already struggling to grapple with as the attendance scrubbing controversy drags on. While we can't currently publish school report cards, these cuts would also risk further future delays putting federal monies at risk.

Under educator quality efforts:

• A 10% cut in funds for Educator Standards and Prep would mean new work required under statute, particularly for teachers and principals, could not be properly managed. Such work includes the implementation of the new teacher and principal evaluation systems required by statute and using evaluation systems to inform decisions including professional development, promotion, retention, placement, tenure and removal of ineffective educators.
• 10% cut to Teacher and Principal Evaluation/Student Growth, could put in jeopardy the continued implementation of the teacher and principal evaluation systems which are included as a part of Ohio's ESEA flexibility waiver. Also, the incorporation of student growth as a metric in these systems, mandated by state law, could be put in jeopardy. LEA requests for support in implementation have been overwhelming and the need for support will continue to grow.

It would be ironic if the Governor killed his own education policies via his own fiscal policy.

Here's the full analysis

Impact of a further 10% cut to Ohio's education budget