The Foolish Endeavor of Rating Ed Schools by Graduates’ Value-Added

Via School Finance 101.

Knowing that I’ve been writing a fair amount about various methods for attributing student achievement to their teachers, several colleagues forwarded to me the recently released standards of the Council For the Accreditation of Educator Preparation, or CAEP. Specifically, several colleagues pointed me toward Standard 4.1 Impact on Student Learning:

4.1.The provider documents, using value-added measures where available, other state-supported P-12 impact measures, and any other measures constructed by the provider, that program completers contribute to an expected level of P-12 student growth.

http://caepnet.org/commission/standards/standard4/

Now, it’s one thing when relatively under-informed pundits, think tankers, politicians and their policy advisors pitch a misguided use of statistical information for immediate policy adoption. It’s yet another when professional organizations are complicit in this misguided use. There’s just no excuse for that! (political pressure, public polling data, or otherwise)

The problems associated with attempting to derive any reasonable conclusions about teacher preparation program quality based on value-added or student growth data (of the students they teach in their first assignments) are insurmountable from a research perspective.

Worse, the perverse incentives likely induced by such a policy are far more likely to do real harm than any good, when it comes to the distribution of teacher and teaching quality across school settings within states.

First and foremost, the idea that we can draw this simple line below between preparation and practice contradicts nearly every reality of modern day teacher credentialing and progress into and through the profession:

one teacher prep institution –> one teacher –> one job in one school –> one representative group of students

The modern day teacher collects multiple credentials from multiple institutions, may switch jobs a handful of times early in his/her career and may serve a very specific type of student, unlike those taught by either peers from the same credentialing program or those from other credentialing programs. This model also relies heavily on minimal to no migration of teachers across state borders (well, either little or none, or a ton of it, so that a state would have a large enough share of teachers from specific out of state institutions to compare). I discuss these issues in earlier posts.

Setting aside that none of the oversimplified assumptions of the linear diagram above hold (a lot to ignore!), let’s probe the more geeky technical issues of trying to use VAM to evaluate ed school effectiveness.

There exist a handful of recent studies which attempt to tease out certification program effects on graduate’s student’s outcomes, most of which encounter the same problems. Here’s a look at one of the better studies on this topic.

• Mihaly, K., McCaffrey, D. F., Sass, T. R., & Lockwood, J. R. (2012). Where You Come From or Where You Go?

Specifically, this study tries to tease out the problem that arises when graduates of credentialing programs don’t sort evenly across a state. In other words, a problem that ALWAYS occurs in reality!

Researchy language tends to downplay these problems by phrasing them only in technical terms and always assuming there is some way to overcome them with statistical tweak or two. Sometimes there just isn’t and this is one of those times!

[readon2 url="http://schoolfinance101.wordpress.com/2013/02/25/revisiting-the-foolish-endeavor-of-rating-ed-schools-by-graduates-value-added/"]Continue reading...[/readon2]

10 Education Reform Tactics That Hurt Students and Don’t Improve Education

We write almost exclusively about education reform here at JTF, and there's been an awful lot to write about in recent years. At the very core of our support and objections to various reforms has always been whats best for students. This post from LAProgressive captures a lot of the problems with the current direction corporate education reform is taking us, and the negative effect it has on students

1. Deluging schools with tests in every grade and every subject beginning with pre-kindergarten, to the point where little else goes on in school but preparing for tests.

2. Pushing the arts out of a central role in the life and culture of public schools.

3. Demoralizing teachers, especially the most talented and experienced teachers, by subjecting them to evaluations based on junk science

4. Discriminating against special needs and English Language Learner (ELL) students by giving favorable treatment to charter schools which exclude or drive out such students, and forcing such students to take tests that are developmentally inappropriate for them

5. Destabilizing communities by closing schools that have been important community institutions for generations.

6. Undermining the mentoring and relationship building that are at the core of great teaching, especially in poor and working class communities, by raising class size and substituting online learning for direct instruction without thinking through the consequences of such policies on young people who need personal attention and guidance.

7. Creating such unrealistic pressure on schools, and on administrators and teaching staffs, that cheating on tests becomes endemic.

8. Giving billionaire philanthropists, and wealthy companies which provide services to schools such power over education policy smothering the voices of teachers, parents and students.

9. Replacing veteran teachers, often teachers of color, with poorly trained Teach for America Corps members,most of them white, who go through a 5 week training period before being given their own class, and often leave for other professions after their two year teaching commitment is completed.

10. Adding to mental health problems of students by spending so much on testing that school districts have to fire school counselors, and to the physical problems of students by transforming gym and recess and after school programs into test prep removing opportunities for exercise and play.

There's a lot to recognize in that list, and be worried about.

Stop blaming teachers

The Merriam-Webster dictionary defines scapegoat as one that bears the blame for others, or one that is the object of irrational hostility. Those of us in the education profession would define scapegoat this way: teacher.

Scapegoating teachers has become so popular with policymakers and politicians, the media, and even members of the public that it has blurred the reality of what’s really happening in education. What’s more, it’s eroding a noble profession and wreaking havoc on student learning, says Kevin Kumashiro, author of Bad Teacher!: How Blaming Teachers Distorts the Bigger Picture.

In his book, Kumashiro, president of the National Association for Multicultural Education and professor of Asian American Studies and Education at the University of Illinois at Chicago, explains how scapegoating public-school teachers, teacher unions, and teacher education masks the real, systemic problems in education. He also demonstrates how trends like market-based reforms and fast-track teacher certification programs create obstacles to an equitable education for all children.

[readon2 url="http://neatoday.org/2012/11/26/stop-blaming-teachers/"]Continue reading...[/readon2]

A narrative path forward for teachers

One of the best responses to the corporate education reformers we've read in a long time.

In every country in the world, poverty impedes educational success. Our biggest education problem is that more of our kids are in poverty than any other developed nation. When America's public school teachers get kids who are well-fed and healthy and live in stable homes with parents who have good jobs, those kids do better in school than any other children in the world.

But a group of people who do not teach (or taught for a short while and not very well) have decided to blame teachers - teachers! - for all the problems in our country. They say that "choice" will save our schools, but the "choice" they offer is between underfunded, crumbling public schools and corporatized, autocratic charter schools that they admit they will never serve all children. These schools cherry-pick their students and then falsely claim they have the secret for success. Their inability to educate all students proves that public schools are not the problem - poverty is. 

Why do these people sell this snake oil? Three reasons:

1) Many of them are looking to make money - a lot of money - off of education. They want to do to our schools what they did to our military, turning them into a bunch of Haliburton Highs.

2) They want to finally and completely break the unions. Once the teachers fall, it's all over for the middle class.

3) They need a scapegoat. Teachers didn't create these problems: the corporate titans of Wall Street did. These plutocrats are now paying a gang of carnival barkers a big bunch of money to blame teachers - teachers! - for the problems they themselves made.

The Teacher Evaluation Juggernaut

Ed Week has a piece on the problems new teacher evaluation systems are going to have on resources, and issue we have discussed before.

Teacher evaluation--with all its multiple facets, blind alleys, disputed data models, technocratic hype and roll-out problems-- is on every principal's mind these days. It would be great to think that principals in states with new evaluation plans are eager to begin this work, now having permission to sink more deeply into their roles as instructional guides, to have productive two-way professional conversations with their teachers, thinking together about improving instruction to reach specific goals.

But no. They're worried about another time suck and avalanche of paperwork on top of an already-ridiculous workload. And--you can't blame them. Being a good principal, like being a good teacher, is impossible. There is no way one single human being can cover all the bases, from keeping the buses running on time to staying abreast of the new math curriculum in grades K through 6. Besides, the new evaluation plans have huge problems embedded, beyond the make-work element.

It was the closing comment of this article that caught our attention

In the end, this will be another issue where outcomes are determined by cost-effectiveness. If it's too expensive for principals to fairly evaluate teachers' instructional efficacy, a cheaper strategy--relying more heavily on test data and technology--will be found. In fact, I'm guessing that any number of education publishers and non-profits are working on it right now.

that seems about right, and likely. However, we wouldn't underestimate the significant costs that test and technology based solutions are going to bring either. However you try to dice it, you arrive at the "unfunded mandate" problem. There's simply too much work, and not enough people or money to do it properly.

Corporate education reformers need to step up to the plate and fully fund their projects.

Would you blame the architect if you accidentally burned your house down

The Plain Dealer has a comprehesive article on teacher level value add, that's worth reading in full. We pulled out the following excerpt to highlight the problem with having a secret forumla no one understands, and which might not prove stable over time

Former Cleveland State University professor Douglas Clay said the complicated formula and a calculation process that SAS keeps secret are a concern, especially if pay is eventually tied to ratings.

"It will go to court the first time someone is denied a raise," he predicted.

SAS' White said a simpler calculation of subtracting students' scores on the state achievement tests from the previous year's scores would be easy to understand, but would be so simplistic it would ignore many factors and misclassify many teachers.

Kenston Superintendent Bob Lee is among the educators who are hesitant about applying value-added to individual teachers because of constant shifting or "stabilization" of test scores by the state and changes to the range of scores that meet, exceed or fall short of the standard. He said most legislators and even fellow superintendents do not understand all those details.

"I'm worried it's being rolled out with very little understanding at the state level of how it behaves," he said.

What is interesting about this article is the response in the comments, overwhelmingly opposed to the idea of teacher level value add as a means of evaluation.

cognitiveguy November 13, 2011 at 6:47PM

Value added seems like a fundamentally good idea but this story points to a significant problem with it, which is that it's far too complicated for most educators and parents to understand and, therefore, few educators and parents are likely to buy into it or trust it. ODE needs to explain how progress is measured in a way that everyone can understand. For example, I read in a previous story and on the ODE Web site that multiple regression analysis is used to calculate student progress. Really? Multiple regression analysis? Say that to a parent, teacher, or superintendent and watch their heads spin. By the way, good story.

petre November 13, 2011 at 7:07PM

I pity the teacher who gets students whose parents don't give a ____(fill in the blank). What will they be forced to do to get chronic truants to come to school? To get those from families who don't value education to do anything at all? Teach to the tests only? What about classes like science and history if the measurement is English and math?

Pirate November 13, 2011 at 7:10PM

What do we want to teach our students? Do we want it to be measured with the same bubble tests which have little to do with learning and much to do with measurement so that teachers and students can be defined as smart, or not, with learning disabilities, or not, creative or not, and all these determinations are made by who? Bubble tests which have given the measurement people what they want so that they can define our entire education process from the governors mansion. Fire this one because of poor tests, fail this one and label them a failure for the next five years. It is fact that operating an educational system based upon bubble tests places an identity on children which follows them through school. 'The looking glass self' has harmed many and still does. Only a joyful learner can learn or teach (which is one in the same thing to you who are not educated in education). It is a process which goes on all of the time and the roles can and should switch. Learning is a joy when done right. We are not doing it right and if we get politicians involved in hire and fire of teachers we are dead. The conservative is always bad for freedom in education and seeks to punish and control all in their path. There is already enough of them in education making education a bore and a numbers game like profit and business and selling and marketting worthless junk to the masses. Teachers need to help kids think, and reason and make choices based of helping others and not competing with them and envying them...Our system needs much change, but more bubble tests is not the answer.

heyfrank25 November 13, 2011 at 7:11PM

Hopefully with the regression they will be controlling for socioeconomic factors, student's attendance, parent's negligence, etc. Otherwise these teachers are just screwed. And 40 percent of teachers are failing in the "excellent" rated Perry SD? Obviously these tests have no criterion validity.

susiecat November 13, 2011 at 7:15PM

This is a very bad idea for two reasons. First of all, who will want to work with students who have low ability? If teachers are rated based on student performance, all teachers will want to work with the more capable students. Also, it will create an atmosphere where teachers will no longer share their ideas and methods freely. Right now, teachers share ideas for the betterment of the entire school community. With value added, who will want to share? Who will want to work together for the common good of ALL students, not just those in their own personal class? Teachers will just shut their doors and keep to themselves the wealth of information that they could share with new teachers. There will be a DECLINE in student performance.

CityontheLake November 13, 2011 at 8:12PM

This won't work. If a kid doesn't want to learn something, they're not going to learn it. A teacher's skill in teaching doesn't matter. Now, our education system does have problems, but I think teachers are causing very few of them. This is coming from someone who was a mediocre student all through high school.

Would you blame the architect if you accidentally burned your house down?

djkorn1 November 13, 2011 at 9:11PM

I feel bad for teachers that have high populations of Special Education students. How are students with learning disabilities supposed to get 'more than a years' worth of growth?? According to No Child Left Behind and the Ohio Acheivement Assesment, they are supposed to... (and they count double), if they are poor, they count triple.

This is like trying to win the Superbowl with the Chess club.

kmark92s November 13, 2011 at 10:13PM

Will the state publish the names of the parents who don't show any interest in their children's education? Will the state grade them on how many unanswered phone calls, unattended conferences or how many days their children have missed school? I would love to see that in the same report.

Shu71 November 13, 2011 at 10:24PM

I would like to know if this means that teachers will be able to draft the students in their class? Otherwise all you would need is 1 principal with an ax to grind and poof, you get all of the behavior problems, home problems and family problems in a classroom.

I would also like to know if the legislation includes identical mandatory testing and student service requirements for all Charter and Private Schools that receive a single cent of tax money. After all, those schools (with their controlled student populations) will crow loudest if permitted to cherry pick the best of the best kids. If this is going in to effect in Ohio, then ALL educators/school that receive state dollars need to play by the same rules.

american November 13, 2011 at 10:40PM

I feel bad and unlucky that we have ignorant political leaders who consider teachers enemies of their political party. This cold war is not only against teachers but it is 100% against students. If our leaders take their time to develop our state standards to meet standards in England or Germany or Japan, or France, or China, or India, our children and parents will appreciate their great work. For Politicians to spend their time just to attack teachers instead of thinking to improve the economic situations in Ohio is an evidence that our leaders in Columbus must go.

howardbeale November 13, 2011 at 11:51PM

So, if the State fails to maintain facilities or equipment, doesn’t buy new books or teaching materials, will not hire the required number of educators and lays-off new teachers, and not only ignores the Ohio Supreme Court when it rules that the financing of education in this state is unconstitutional it actually puts up the money to replace the judges, then we should grade the teachers who survive all this junk? Why don’t we begin to grade these administrators and legislators who have been failing us for thirty years? I’m not asking the Plain Dealer, I’m asking you the posters. It is time we rebelled, withheld campaign contributions, and voted against these idiots.

There's an awful lot of confusion, complexity and very good points in just these comments.