Battelle Blasts Papers decision

From our mailbag, Battelle for Kids condemns the Plain Dealer and NPR's decision to publish teacher's value-added scores, calling it "the poster child for name, blame, and shame and the antithesis of our approach to using value-added data"

To: All SOAR districts
From: Jim Mahoney and Bobby Moore
Date: June 17, 2013

Yesterday, a three-part series on value-added was launched by The Cleveland Plain Dealer and State Impact Ohio. It includes both articles and radio segments specific to value-added analysis as a measure of teacher effectiveness. Highlighted in the articles is a link to a database of teacher ratings, hosted by The Plain Dealer and the State Impact Ohio partnership.

Currently, Ohio laws governing the release of teacher records would apply to teacher value-added results. Thus, teacher level value-added information is subject to public records requests through ODE. Through The Plain Dealer and State Impact Ohio database, the general public can now access a teacher's overall composite rating derived from two years of his/her results in grades 4-8 math and reading. These data reflect information for less than 1/3 of the math and reading, grades 4-8 teachers in Ohio.

Battelle for Kids was not aware these ratings would be published in this way, at this time.

While Battelle for Kids does support the use of value-added information for school improvement and as one of several components of a multi-measures evaluation system, value-added should NOT be used in isolation to draw conclusions about a teacher's effectiveness.

Multiple data points over time from multiple perspectives are crucial because teaching and learning and the evaluation of teaching and learning are complex.

Therefore, we are NOT supportive of these ratings being publically available and discourage promoting the use of this public database.

Talking points and articles, to support your local conversations, are available on the Ohio Student Progress Portal.

http://cts.vresp.com/c/?BattelleForKids/f43a0e1b46/fb8aa9ca4e/313346eb88/sflang=en

Obviously, this is the poster child for name, blame, and shame and the antithesis of our approach to using value-added data.

Please call if you have any questions.

Thank you for all you do for Ohio's students!

-Jim and Bobby

Hard to measure love

Ripped from the comments of this Gates Foundation booster article in the NYT, discussing the measurement of teacher effectiveness

It's almost the end of an exhausting school year, and all I can do is laugh when I read articles like this. I'm supposed to be a "teacher," which I guess means I'm supposed to "instruct" students, and the "effectiveness" of my instruction seems to be what the Gates Foundation claims it's trying to assess. But since I've spent a large amount of my time over the last several months serving as the de facto counselor for teenagers who are depressed, anxious, suicidal, self-injurious, suffering from eating disorders, living in chaotic and destructive family situations, lonely, isolated, scared, and confused, teenagers for whom I am for whatever reason the go-to "trusted adult," I've come to the conclusion that the most important thing I have to offer my students is love. Try to measure that.

Few in the corporate education reform movement grasp this kind of sentiment and reality, which is one reason there is such a large disconnect between those in the classroom delivering education policy and those in the boardroom's making education policy.

How does this manifest itself in the real world? From the Gates article

All along, Gates says, he had been asking questions about teacher effectiveness. How do you measure it? What are the skills that make a teacher great? “It was mind-blowing how little it had been studied,” he told me. So, with the help of Thomas Kane, an education professor at Harvard, the Gates Foundation began videotaping some 3,000 teachers across the country. It also collected lots of other data to measure whether a teacher was effective. All of this work, Kane says, was aimed at “identifying the practices that are associated with student achievement.”

With a wealth of data now in hand, the Gates Foundation was ready for the next step: trying to create a personnel system that not only measured teacher effectiveness but helped teachers improve. Although pilot projects have been announced in four school districts, the one that is furthest along is in Hillsborough County, Fla. That district, which is dominated by Tampa, is in the second year of a seven-year, $100 million grant.

Only 2 years into the pilot program, tension is mouinting in Hillsbrough

Don't count school board member Stacy White as a fan of the teacher evaluation system in Hillsborough County public schools.

"I am not saying that we should not hold teachers accountable," White said today at a workshop on the topic. "But you can put me down as a critic of EET as it stands in its current form."

Empowering Effective Teachers, the evaluation system put in place after the school district accepted a seven-year, $100 million grant from the Bill & Melinda Gates Foundation, is nearing the end of its second year.

But the controversy around it is not by any means nearing its end.

"Our teachers feel often times that what they have is Big Brother coming in the classroom to watch over them," White said. "Folks view the peer position as the man or the woman in the black hat."

In fact, in some cases the situation is becoming so tense, one teacher has been suspending for protesting

A veteran teacher was suspended Thursday for rejecting the evaluator chosen for him under a Gates-funded initiative that is revolutionizing the way the Hillsborough County School District assesses its teachers.

School and union officials believe this is the first such act of defiance under Empowering Effective Teachers, a complex system of mentoring and evaluation funded by the Bill and Melinda Gates Foundation.

The district's action comes just one day after the couple themselves, Bill and Melinda Gates, toured Jefferson High School, where the computer mogul hailed the program as a national model and called its success "phenomenal."

Joseph Thomas, 43, a social studies teacher at Newsome High School, said he refused to schedule a peer observation because he feels the evaluator, Justin Youmans, is not qualified to judge him.

Youmans, 29, has his experience teaching elementary school and sixth grade, according to his school district biography. "He thinks like an elementary school teacher," said Thomas, a teacher for 18 years.

These concerns have also been exressed in Ohio. Who will perform the hundreds of tohusands of observations, and will they be suitably qualified in the subject and grade areas they are observing? This is a big question, and relates directly to scaling the concept of multiple classroom observations. What sounds simple in theory, in practice is complex, expensive, and judging by the experiences in Florida, controversial.

You can't do reforms like these on the cheap, let alone in a revenue declining environemt, yet that is what is being attempted.

Choosing blindly

As we continue to explore areas of education reform currently under discussed, we wanted to bring this recently released study from the Brookings Institute's Brown Center on Education Policy, titled "Choosing Blindly: Instructional Materials, Teacher Effectiveness, and the Common Core", to your attention.

Students learn principally through interactions with people (teachers and peers) and instructional materials (textbooks, workbooks, instructional software, web-based content, homework, projects, quizzes, and tests). But education policymakers focus primarily on factors removed from those interactions, such as academic standards, teacher evaluation systems, and school accountability policies. It’s as if the medical profession worried about the administration of hospitals and patient insurance but paid no attention to the treatments that doctors give their patients.

There is strong evidence that the choice of instructional materials has large effects on student learning—effects that rival in size those that are associated with differences in teacher effectiveness. But whereas improving teacher quality through changes in the preparation and professional development of teachers and the human resources policies surrounding their employment is challenging, expensive, and time-consuming, making better choices among available instructional materials should be relatively easy, inexpensive, and quick.

Administrators are prevented from making better choices of instructional materials by the lack of evidence on the effectiveness of the materials currently in use. For example, the vast majority of elementary school mathematics curricula examined by the Institute of Education Sciences What Works Clearinghouse either have no studies of their effectiveness or have no studies that meet reasonable standards of evidence.

Not only is little information available on the effectiveness of most instructional materials, there is also very little systematic information on which materials are being used in which schools. In every state except one, it is impossible to find out what materials districts are currently using without contacting the districts one at a time to ask them. And the districts may not even know what materials they use if adoption decisions are made by individual schools. The National Center for Education Statistics (NCES), which has the mission of collecting and disseminating information related to education in the U.S., collects no information on the usage of particular instructional materials.

This scandalous lack of information will only become more troubling as two major policy initiatives—the Common Core standards and efforts to improve teacher effectiveness—are implemented. Publishers of instructional materials are lining up to declare the alignment of their materials with the Common Core standards using the most superficial of definitions. The Common Core standards will only have a chance of raising student achievement if they are implemented with high-quality materials, but there is currently no basis to measure the quality of materials. Efforts to improve teacher effectiveness will also fall short if they focus solely on the selection and retention of teachers and ignore the instructional tools that teachers are given to practice their craft.

The full report can be read here.

Studies Give Nuanced Look at Teacher Effectiveness

The massive Measures of Effective Teaching Project is finding that teacher effectiveness assessments similar to those used in some district value-added systems aren't good at showing which differences are important between the most and least effective educators, and often totally misunderstand the "messy middle" that most teachers occupy. Yet the project's latest findings suggest more nuanced teacher tests, multiple classroom observations and even student feedback can all create a better picture of what effective teaching looks like.

Researchers dug into the latest wave of findings from the study of more than 3,000 classes for a standing-room-only ballroom at the American Educational Research Association's annual conference here on Saturday.

"The beauty of multiple measures isn't that there are more of them—more can be more confusing—these need to be alligned to the outcomes we care about," said Steve Cantrell, who oversees the MET project for the Bill & Melinda Gates Foundation. Existing teacher evaluation systems often use indicators that are not effective at guaging student achievement, and moreover that lump teachers into too-simplistic categories.

"The middle is a lot messier than a lot of state policies would lead us to believe," Cantrell said. "Teachers don't fall neatly into quartiles. Based on the practice data, if I look at the quartiles, all that separates the 25th and 75th on a class (observation) instrument is .68—less than 10 percent of the scale distribution. In a lot of systems, the 75th percentile teacher is considered a leader and the 25th percentile considered a laggard. ...This would suggest they're a lot closer than being off by two categories."

[readon2 url="http://blogs.edweek.org/edweek/inside-school-research/2012/04/the_most_and_least_effective.html"]Continue reading...[/readon2]

Research doesn’t back up key ed reforms

Via the Washington Post

There is no solid evidence supporting many of the positions on teachers and teacher evaluation taken by some school reformers today, according to a new assessment of research on the subject.

The Education Writers Association released a new brief that draws on more than 40 research studies or research syntheses, as well as interviews with scholars who work in this field.

You can read the entire brief (written by Education Week assistant editor Stephen Sawchuk), but here are the bottom-line conclusions of each section:

Q) Are teachers the most important factor affecting student achievement?

A) Research has shown that the variation in student achievement is predominantly a product of individual and family background characteristics. Of the school factors that have been isolated for study, teachers are probably the most important determinants of how students will perform on standardized tests.

Q) Are value-added estimations reliable or stable?

A) Value-added models appear to pick up some differences in teacher quality, but they can be influenced by a number of factors, such as the statistical controls selected. They may also be affected by the characteristics of schools and peers. The impact of unmeasured factors in schools, such as principals and choice of curriculum, is less clear.

Q) What are the differences in achievement between students who have effective or ineffective teachers for several years in a row?

A) Some teachers produce stronger achievement gains among their students than others do. However, estimates of an individual teacher’s effectiveness can vary from year to year, and the impact of an effective teacher seems to decrease with time. The cumulative effect on students’ learning from having a succession of strong teachers is not clear.

Q) Do teacher characteristics such as academic achievement, years of experience, and certification affect student test scores?

A) Teachers improve in effectiveness at least over their first few years on the job. Characteristics such as board certification, and content knowledge in math sometimes are linked with student achievement. Still, these factors don’t explain much of the differences in teacher effectiveness overall.

Q) Does merit pay for teachers produce better student achievement or retain more-effective teachers?

A) In the United States, merit pay exclusively focused on rewarding teachers whose students produce gains has not been shown to improve student achievement, though some international studies show positive effects. Research has been mixed on comprehensive pay models that incorporate other elements, such as professional development. Scholars are still examining whether such programs might work over time by attracting more effective teachers.

Q) Do students in unionized states do better than students in states without unions?

A) Students tend to do well in some heavily unionized states, but it isn’t possible to conclude that it is the presence or absence of unions that cause that achievement.

What Studies Say About Teacher Effectiveness

Ohio can't wait to start misusing value add

The Columbus Dispatch ran an article "Ratings start to ID effective teachers", which discusses the recent use of teacher level value add scores, primarily as part of RttT, but which also will feature heavily in teacher evaluations going forward.

The article covers a lot of common ground, but not until the 17th of 27 paragraphs does it even mention how inappropriate value add is for this use

Officials involved in producing the new effectiveness ratings say they should not be used to label a teacher as good or bad. This year’s rating is a statement of a teacher’s effectiveness with his or her students from last school year, and nothing more, said Mary Peters, senior director of research and innovation at Battelle for Kids. The Columbus-based nonprofit organization is helping the Education Department develop the effectiveness system.

“We need to be careful about making judgments about one year of data,” Peters said. “These measures were intended for diagnostic purposes, to provide information to help teachers reflect on their practice and determine with whom they are being successful.”

Despite these constant warnings by academics and researchers alike, policy makers, and some government bureaucrats continue to see teacher level value add as a primary tool for teacher evaluation, and it looks for all the world that Ohio can't wait any longer to begin misusing this tool