Value-added: How Ohio is destroying a profession

We ended the week last week with a post titled "The 'fun' begins soon", which took a look at the imminent changes to education policy in Ohio. We planned on detailing each of these issues over the next few weeks.

Little did we know that the 'fun' would begin that weekend. It came in the manner of the Cleveland Plain Dealer and NPR publishing a story on the changing landscape of teacher evaluations titled "Grading the Teachers: How Ohio is Measuring Teacher Quality by the Numbers".

It's a solid, long piece, worth the time taken to read it. It covers some, though not all, of the problems of using value-added measurements to evaluate teachers

Those ratings are still something of an experiment. Only reading and math teachers in grades four to eight get value-added ratings now. But the state is exploring how to expand value-added to other grades and subjects.

Among some teachers, there’s confusion about how these measures are calculated and what they mean.

“We just know they have to do better than they did last year,” Beachwood fourth-grade teacher Alesha Trudell said.

Some of the confusion may be due to a lack of transparency around the value-added model.

The details of how the scores are calculated aren’t public. The Ohio Education Department will pay a North Carolina-based company, SAS Institute Inc., $2.3 million this year to do value-added calculations for teachers and schools. The company has released some information on its value-added model but declined to release key details about how Ohio teachers’ value-added scores are calculated.

The Education Department doesn’t have a copy of the full model and data rules either.

The department’s top research official, Matt Cohen, acknowledged that he can’t explain the details of exactly how Ohio’s value-added model works. He said that’s not a problem.

Evaluating a teacher on a secret formula isn't a practice that can be sustained, supported or defended. The article further details a common theme we hear over and over again

But many teachers believe Ohio’s value-added model is essentially unfair. They say it doesn’t account for forces that are out of their control. They also echo a common complaint about standardized tests: that too much is riding on these exams.

“It’s hard for me to think that my evaluation and possibly some day my pay could be in a 13-year-old’s hands who might be falling asleep during the test or might have other things on their mind,” said Zielke, the Columbus middle school teacher.

The article also performs analysis on several thousands value add scores, and that analysis demonstrates what we have long reported, that value-add is a poor indicator of teacher quality, with too many external factors affecting the score

A StateImpact/Plain Dealer analysis of initial state data suggests that teachers with high value-added ratings are more likely to work in schools with fewer poor students: A top-rated teacher is almost twice as likely to work at a school where most students are not from low-income families as in a school where most students are from low-income families.
[…]
Teachers say they’ve seen their value-added scores drop when they’ve had larger classes. Or classes with more students who have special needs. Or more students who are struggling to read.

Teachers who switch from one grade to another are more likely to see their value-added ratings change than teachers who teach the same grade year after year, the StateImpact/Plain Dealer analysis shows. But their ratings went down at about the same rate as teachers who taught the same grade level from one year to the next and saw their ratings change.

What are we measuring here? Surely not teacher quality, but rather socioeconomic factors and budget conditions of the schools and their students.

Teachers are intelligent people, and they are going to adapt to this knowledge in lots of unfortunate ways. It will become progressively harder to districts with poor students to recruit and retain the best teachers. But perhaps the most pernicious effect is captured at the end of the article

Stephon says the idea of Plecnik being an ineffective teacher is “outrageous.”

But Plecnik is through. She’s quitting her job at the end of this school year to go back to school and train to be a counselor — in the community, not in schools.

Plecnik was already frustrated by the focus on testing, mandatory meetings and piles of paperwork. She developed medical problems from the stress of her job, she said. But receiving the news that despite her hard work and the praise of her students and peers the state thought she was Least Effective pushed her out the door.

“That’s when I said I can’t do it anymore,” she said. “For my own sanity, I had to leave.”

The Cleveland Plain Dealer and NPR then decided to add to this stress by publishing individual teachers value-added scores - a matter we will address in our next post.

How "top charters" screen students

It's no secret that the vast majority of Ohio charter schools are rated F, but what of some of the "high performing" schools? It is with those in mind, we read with interest the article "The Dirty Dozen: How Charter Schools Influence Student Enrollment" .

This commentary offers a classification of twelve different approaches that charter schools use to structure their student enrollment. These practices impact the likelihood of students enrolling with a given set of characteristics, be it higher (or lower) test scores, students with ‘expensive’ disabilities, English learners, students of color, or students in poverty.
[...]
Yet little attention has been paid to the mechanisms that generate these differences. One exception is an article in February of 2013, written by reporter Stephanie Simon of Reuters, which described a variety of ways that charter schools “get the students they want” (Simon, 2013):

• Applications that are made available just a few hours a year.
• Lengthy application forms, often printed only in English, that require student and parent essays, report cards, test scores, disciplinary records, teacher recommendations and medical records.
• Demands that students present Social Security cards and birth certificates for their applications to be considered, even though such documents cannot be required under federal law.
• Mandatory family interviews.
• Assessment exams.
• Academic prerequisites.
• Requirements that applicants document any disabilities or special needs. The U.S. Department of Education considers this practice illegal on the college level but has not addressed the issue for K-12 schools.

We thought we would pick one charter school and test this hypothesis. We picked DAYTON EARLY COLLEGE ACADEMY, INC. (DECA), as they were elevated by they Fordham Foundation and recently testified on the budget as part of a "coalition of high performing charters".

Following introductions from Fordham’s Terry Ryan, Dayton Early College Academy’s Superintendent Judy Hennessey began to speak in front of the Subcommittee only to be interrupted by Committee Chair Senator Randy Gardner, “Senator [Peggy] Lehner has just commented you lead one of the best schools in the country.”

Jokingly Judy Hennessey nodded and said, “Now we are striving for world class.”

The application process.

Here's DECA's application, which can also be downloaded here.

High School Application 2013-14

The first thing you will note is the application form is 23 pages long, requiring hundreds of pieces of information to be entered including report cards, test scores, disciplinary records, teacher recommendations and medical records. In fact, all mechanisms mentioned in the reuters article commonly used to screen prospective students. This is a significant barrier that only the most determined parent is likely to scale.

The page where the applications can be downloaded clearly states, in bold, "Incomplete applications will not be considered."

A parent who is likely to complete such a detailed, lengthy application is likely a parent who is going to be engaged in their child's education to a greater degree than a parent who is unlikely to apply.

Furthermore, as is pointed out in the 12 approaches charters use to screen for students, this application is in English only. No second language form is available on the application webpage- making English as a second language applications far less likely.

You will also see that on page 5 of the application

Documents needed for a complete application
 Student birth certificate
 Student social security card

"Demands that students present Social Security cards and birth certificates for their applications to be considered, even though such documents cannot be required under federal law." is one of the tell-take screening mechanisms charters use.

The DECA application form also requests that applicants document any disabilities or special needs, another potential barrier spelled out in the article.

So we can plainly see then, that while DECA may produce above average results for a charter school, it can do so because it has a highly selective application process that is likely to screen out lower performing students.

The performance results

We were expecting a charter school whose leader professed to be aiming for "world class standards" to be rated Excellent with Distinction. DECA is not, indeed it is not even rated Excellent, instead it rates as "Effective" according to the latest data available from ODE.

Building IRN	009283
Building Name	Dayton Early College Academy, Inc
District IRN	043844
District Name	Dayton City
County	Montgomery
Principal	Judy Hennessey
Grade Span	7-12
Open/Closed Status (as of 9/18/2012)	Open
Designation	Effective
Number of standards met	14
Number of standards possible	17
Enrollment	411
Performance Index Score 2011-12	99.1
Performance Index Score 2010-11	100.5
Performance Index Score 2009-10	96.2
2012 Attendance AYP	N/A
2012 Graduation AYP	Not Met
2012 Reading AYP	Met
2012 Math AYP	Met
2012 Overall AYP	Not Met
Four-Year "On-Time" Graduation Rate Numerator 2010-11	35

These aren't bad results, indeed compared to the majority of F rated charter schools they are positively giddy. But, given the arduous application screening process, and the "effective" rating, it's a far cry from being world beating, and a very far cry from the world of traditional public schools which have to accept every student from the district that walks through the door.

A decade-long crisis of democracy

We highlighted that despite Ohio voters in the aggregate preferring Democrats over Republicans in the 2012 election, the Republicans will hold a probable super majority 60-39 as a consequence of extreme partisan gerrymandering. The Dispatch was prompted by this result to produce an article about redistricting

Issue 2 is dead, buried deep by Ohio voters last week.

But over and over again, opponents of the redistricting plan, be they Republicans or editorial-page writers, noted that their opposition was not based on the belief that the current system of drawing legislative and congressional districts is good.

In fact, most acknowledged that it remains badly in need of an overhaul.

But if was this paragraph in the article that prompted us to take an even deeper look

Republicans now control 75 percent of the U.S. House seats and nearly two-thirds of the legislative seats in a state that has leaned Republican but is a key battleground state

We analyzed Ohio House of Representative results for each of the past 6 election cycles. By aggregating the votes for Democrats and Republicans in contested races we found a systematic, and extreme disenfranchising of Democratic representation in Ohio

Year	Democratic	Republican	D Seats	R Seats
2012	2,418,815	2,362,310	39	60
2010	1,447,949	1,696,064	40	59
2008	2,296,678	1,982,281	53	46
2006	1,832,548	1,605,801	46	53
2004	1,869,051	2,036,398	38	60
2002	1,243,671	1,364,656	36	63
Total	11,108,712	11,047,510

Based upon the preferences of voters, Democrats should have controlledthe General assemblies after the 2012, and 2006 elections - but were denied by partisan gerrymandering. Furthermore, the majorities that Republicans did earn in all of their successful years should have been much, much smaller - and never reacher super majority status.

Indeed when one looks at the sum total of votes in contest races over the past decade, rather than being center right, the results indicate a center to center left leaning electorate.

It is simply not possible to conclude that Ohioans have been legitimately represented in the 21st century by their preferred choices, either in actuality or in scope. We have a crisis of democracy in Ohio.

Lessons from the teachers strike

When an article concludes with this line, we're going to recommend you read the entire piece.

As both policy and politics, the demonization of teachers unions is a dead end for improving American education. Working with, not against, teachers is the more sensible way to better our schools.

Voters First - Issue 2 - ballot language

The Ohio ballot board, in a partisan split decision chose the following as the language that will appear as Issue 2 (Voters First) on the November 6th ballot.

Proposed Constitutional Amendment
Proposed by Initiative Petition
To add and repeal language in Section l,3,4,6,7,9 and 13 of Article XI, repeal Sections 8 and 14 of Article XI, and add a new Section 16 to Article XI of the Constitution of the State of Ohio

A majority yes vote is necessary for the amendment to pass.

The proposed amendment would:

1. Remove the authority of elected representatives and grant new authority to appointed officials to establish congressional and state legislative district lines.

2. Create a state funded commission of appointed individuals from a limited pool of applicants to replace the aforementioned. The Commission will consist of 12 members as follows: four affiliated with the largest political party, four affiliated with the second largest political party and four not affiliated with either of the two largest political parties. Affirmative votes of 7 of 12 members are needed to select a plan.

3. Require new legislative and congressional districts be immediately established by the Commission to replace the most recent districts adopted by elected representatives, which districts shall not be challenged except by court order until the next federal decennial census and apportionment. In the event the Commission is not able to determine a plan by October 1, the Ohio Supreme Court would need to adopt a plan from all the plans submitted to the Commission.

4. Change the standards and requirements in the Constitution for drawing legislative and congressional districts.

5. Mandate the General Assembly to appropriate all funds as determined by the Commission including, but not be limited to, compensating:
1. Staff
2. Consultants
3. Legal counsel
4. Commision members

If approved, the amendment will be effective thirty days after the election.

SHALL THE AMENDMENT BE APPROVED
YES
NO

Voters first has filed a lawsuit, claiming the ballot language is incomplete

The wording, for example, omits any references to requirements that the commission draw fair districts that reflect the political preferences of Ohio voters.

The ballot language "does not properly identify the substance of the proposal to be voted upon" and was written "to mislead, deceive or defraud the voters," the lawsuit says.

A summary of the initiative can be read, here.

Erasures demonstrate huge sensitivity in ratings

The Dispatch had another speculative piece of reporting on the attendance erasure issue. We'll leave educator Greg Mild at Plunderbund to go over the substance, or lack thereof, of the article itself. We want to concentrate on something else mentioned in the article which stood out.

At the heart of the controversy is this

Though 7 percent [number of deleted records] may not sound like a lot, it could have a big effect: There are students behind those numbers, and some of their standardized test scores were likely discounted when their attendance records were deleted. That means Columbus’ school grades could have been artificially inflated because of records-tampering.

From this, the Department of Education had a remarkable comment

“The math indicates that removing one student could affect the overall rating,” said John Charlton, spokesman for the Ohio Department of Education. “In some districts, it could be one kid. It’s about dropping the right kid.”

Can this be true? Can the ratings of an entire district truly be affected by just a handful of students, or even one? We have an actual true life example, that yes, small numbers of students can indeed cause an entire districts rating to be changed

In Lockland near Cincinnati, removing 36 kids from the rolls — about 5 percent of their student population — lifted the district from a C to a B on the state report card.

Can a school rating system that is so sensitive to just a handful of students truly be measuring the district as a whole? This growing controversy over attendance data is revealing a lot more than people realize.