Fordham hides from facts

UPDATE: Fordham has now published our comment on their site, for which we are grateful. See the comments to this article for their explanation.

The Fordham Foundation, one of Ohio's more vocal charter school boosters, has a post on their website defending the high number of failing charter schools. The piece is written by Aaron Churchill, someone we have observed stretching facts and the truth before (Fordham loses its bearings). Like his previous piece's error addled analysis, his latest defense of failing charter schools goes to great lengths to obfuscate hard truths using indefensible "statistical analysis".

Rather than write a post here on JTF, we tried to leave a long comment pointing out just some of the errors in the post. Fordham has decided they would rather that comment be hidden and not be published, so we are publishing it below, in response to a Fordham reader asking us to

Aaron states "The chart shows that a nearly equal number of charters reside in the state’s bottom 111 schools"

Let's just assume that is correct. What if utterly fails to recognize is that there are orders of magnitude more traditional public school buildings than charter schools - so the fact that so many charter school buildings appear in the bottom 111 should be disturbing to everyone, not glommed onto as a point of false equivalence. As an overall percentage of school buildings charter schools dominate the bottom rankings.

Let's look at another claim made by Aaron...

"The fact of the matter is that taxpayers spend less on each child in a charter school then is spent on their district peers."

That claim is contradicted by the Ohio Department of Education (link here:http://www.scribd.com/doc/117636411/ODE-Analysis-of-Per-Pupil-Cost-of-Charters-and-Publics).

"The average of total expenditure per pupil for public districts is $10,110.72.

The average expenditure per pupil for community schools is $9,064. When broken out: For e-schools it’s $7,027. For non-eschool community schools it’s $10,165.

So only when one combines the cost of the laughably cheap (and ludicrously underperforming) e-schools do Ohio's charters look inexpensive - and that's using ODE as a source.

Aaron did a good job, as all charter school boosters do, of obfuscating the facts - which is that the vast majority of Ohio's charter schools deliver a poor quality education at an inflated cost.

Let's close them down and concentrate our energy on the schools that 95% of Ohio's students go to, and maybe learn some things along the way from the few charter schools that are getting it right, instead of this constant non-debate and excuse making about the terrible charter schools we all know exist in very high numbers.

Fordham likes to hide behind their advocacy for charter school accountability and quality, but whenever they are pressed on this, they obfuscate the difficult facts and revert to defending the rotten and the failing. They may talk a good game, but in the end they are no less a charter school booster as White Hat owner, David Brennan. Mr. Churchill's post and decision to avoid a discussion on it are further proof of that.

Why are Ohio's charter schools so poor?

In a previous post we took a look at the difference in performance between Ohio's traditional public schools and their charter school counterparts, and discovered a wide and growing gap. Why does this gap exist?

One of the clear reasons is in the quality of the workforce. Ohio's traditional public schools have invested a lot of time and money developing an experienced, highly qualified teacher workforce, an investment that Ohio's charter schools have resisted or failed to developing.

Steve Dyer at Innovation Ohio took a look at the latest teacher data made available by ODE and discovered that the typical (i.e. the mode) charter school teacher has 0 years of experience, while their traditional school counterpart had 14 years of experience. We took a look at this too, and came to the same conclusion.

Our analysis also showed that the average level of experience of a charter school teacher is only 4.9 years, while in traditional schools that figure is over 14 years.

Dyer also discovered, what he called "two equally stunning statistics"

1) The average Traditional Public School building has about 2/3 of its teachers with a masters degree. The average Charter School building has about 1/3 of its teachers with a masters degree.
[...]
2) About 1 out of 3 Charter Schools in Ohio have at least some core courses taught by someone with a temporary teaching certificate. Of the more than 3,200 Traditional Public School buildings in the report card data, not a single one has core courses taught by teachers with temporary certificates.

Furthermore, over 10% of Ohio's charter schools have class sizes greater than 25 students per teacher, 20 of them more than 30 students per teacher!

The major innovation attempted by Ohio's charter school community has not been to find ways to deliver higher quality education, but instead to find ways to minimize teacher costs in order to maximize profits. This is made very clear when once looks at this experience and qualification differences, education outcome quality differences and the subsequent difference in average salaries where the charter school average is $33,993 and traditional schools $57,303.

Gates Foundation Wastes More Money Pushing VAM

Makes it hard to trust the corporate ed reformers when they goose their stats as badly as this.

Any attempt to evaluate teachers that is spoken of repeatedly as being "scientific" is naturally going to provoke rebuttals that verge on technical geek-speak. The MET Project's "Ensuring Fair and Reliable Measures of Effective Teaching" brief does just that. MET was funded by the Bill & Melinda Gates Foundation.

At the center of the brief's claims are a couple of figures (“scatter diagrams” in statistical lingo) that show remarkable agreement in VAM scores for teachers in Language Arts and Math for two consecutive years. The dots form virtual straight lines. A teacher with a high VAM score one year can be relied on to have an equally high VAM score the next, so Figure 2 seems to say.

Not so. The scatter diagrams are not dots of teachers' VAM scores but of averages of groups of VAM scores. For some unexplained reason, the statisticians who analyzed the data for the MET Project report divided the 3,000 teachers into 20 groups of about 150 teachers each and plotted the average VAM scores for each group. Why?

And whatever the reason might be, why would one do such a thing when it has been known for more than 60 years now that correlating averages of groups grossly overstates the strength of the relationship between two variables? W.S. Robinson in 1950 named this the "ecological correlation fallacy." Please look it up in Wikipedia. The fallacy was used decades ago to argue that African-Americans were illiterate because the correlation of %-African-American and %-illiterate was extremely high when measured at the level of the 50 states. In truth, at the level of persons, the correlation is very much lower; we’re talking about differences as great as .90 for aggregates vs .20 for persons.

Just because the average of VAM scores for 150 teachers will agree with next year's VAM score average for the same 150 teachers gives us no confidence that an individual teacher's VAM score is reliable across years. In fact, such scores are not — a fact shown repeatedly in several studies.

[readon2 url="http://ed2worlds.blogspot.com/2013/01/gates-foundation-wastes-more-money.html"]Continue reading...[/readon2]

A different kind of investment

Education is often referred to as an investment in our future, and undoubtedly it is, as we prepare our young people to enter the workforce with creativity, energy and entrepreneurship. However, modest investment increases in public education also have an immediate, and direct, beneficial effect upon those making the investments.

A recent study by the Brooking's Institute revealed

An analysis of national and metropolitan data on public school populations and state standardized test scores for 84,077 schools in 2010 and 2011 reveals that:
[...]
Across the 100 largest metropolitan areas, housing costs an average of 2.4 times as much, or nearly $11,000 more per year, near a high-scoring public school than near a low scoring public school. This housing cost gap reflects that home values are $205,000 higher on average in the neighborhoods of high-scoring versus low-scoring schools. Near high-scoring schools, typical homes have 1.5 additional rooms and the share of housing units that are rented is roughly 30 percentage points lower than in neighborhoods near low-scoring schools.

This is clear evidence that investing a few hundred dollars per year to increase your local schools performance will have a dramatic effect on your home values - typically $11,000 per year, according to this study. The reason is no secret of course, people with school age children want to live in areas that have excellent schools, and are prepared to pay a premium for it. These results have been found to be true over and over. The St Louis Fed found

Traditional empirical models of the capitalization of education quality on house prices have established that the quality of primary school education is positively correlated with house prices. Recent capitalization studies have used various approaches to address concerns about omitted variable bias induced by failing to account for the correlation between school quality and unobserved neighborhood characteristics. Most of these variations on the traditional hedonic approach (including the boundary discontinuity regression) have assumed that the house price premium is constant because in all these models the contribution from school quality on house prices is constrained to be linear.

In this paper, we propose an alternative formulation that allows for nonlinear effects of school quality. We show that this formulation is preferred by the data over a baseline linear boundary fixed effects model and that the rate at which the house price premium rises increases over the range of school quality. In other words, the standard linear specification for test scores overestimates the premium at low levels of school quality and underestimates the premium at high levels of school quality.

In the St. Louis metropolitan area, houses associated with a school ranked at 1 SD below the mean are essentially priced on physical characteristics only. In contrast, houses associated with higher-quality schools command a much higher price premium.

Interestingly, and in contrast to many studies in the literature, the price premium remains substantially large, especially for houses associated with above-average schools. This is true even in our most conservative estimates, which complement the boundary discontinuity approach by explicitly controlling for neighborhood demographics. These estimates also reveal that the racial composition of neighborhoods is capitalized directly into house prices.

This then makes the move to downgrade Ohio's schools based on some new, arbitrary standard all the more baffling. Not only will this move potentially produce lower school ratings, it may also destroy tens of millions of dollars worth of housing value at a time when house prices are already under extreme stress and the economy struggling to improve.

Consider this then, when they vote one levies. A few hundred bucks could add thousands of dollars to the value of you home. One can only imagine the added wealth that could be created if the state lived up to its constitutional responsibilities and invested properly in public education too.

Shame, errors and demoralizing

Shame, errors and demoralizing, just some of the emerging rhetoric being used since the NYT and other publications went ahead and published teacher level value add scores. A great number of articles have been written decrying the move.

Perhaps most surprising of all was Bill Gates, in a piece titled "Shame Is Not the Solution". In it, Gates argues

Value-added ratings are one important piece of a complete personnel system. But student test scores alone aren’t a sensitive enough measure to gauge effective teaching, nor are they diagnostic enough to identify areas of improvement. Teaching is multifaceted, complex work. A reliable evaluation system must incorporate other measures of effectiveness, like students’ feedback about their teachers and classroom observations by highly trained peer evaluators and principals.

Putting sophisticated personnel systems in place is going to take a serious commitment. Those who believe we can do it on the cheap — by doing things like making individual teachers’ performance reports public — are underestimating the level of resources needed to spur real improvement.
[...]
Developing a systematic way to help teachers get better is the most powerful idea in education today. The surest way to weaken it is to twist it into a capricious exercise in public shaming. Let’s focus on creating a personnel system that truly helps teachers improve.

Following that, Matthew Di Carlo at the Shanker institute took a deeper look at the data and the error margins inherent in using it

First, let’s quickly summarize the imprecision associated with the NYC value-added scores, using the raw datasets from the city. It has been heavily reported that the average confidence interval for these estimates – the range within which we can be confident the “true estimate” falls – is 35 percentile points in math and 53 in English Language Arts (ELA). But this oversimplifies the situation somewhat, as the overall average masks quite a bit of variation by data availability.
[...]
This can be illustrated by taking a look at the categories that the city (and the Journal) uses to label teachers (or, in the case of the Times, schools).

Here’s how teachers are rated: low (0-4th percentile); below average (5-24); average (25-74); above average (75-94); and high (95-99).

To understand the rocky relationship between value-added margins of error and these categories, first take a look at the Times’ “sample graph” below.

That level of error in each measurement renders the teacher grades virtually useless. But that was just the start of the problems, as David Cohen notes in a piece titled "Big Apple’s Rotten Ratings".

So far, I think the best image from the whole fiasco comes from math teacher Gary Rubinstein, who ran the numbers himself, a bunch of different ways. The first analysis works on the premise that a teacher should not become dramatically better or worse in one year. He compared the data for 13,000 teachers over two consecutive years and found this – a virtually random distribution:

First of all, as I’ve repeated every chance I get, the three leading professional organizations for educational research and measurement (AERA, NCME, APA) agree that you cannot draw valid inferences about teaching from a test that was designed and validated to measure learning; they are not the same thing. No one using value-added measurement EVER has an answer for that.

Then, I thought of a set of objections that had already been articulated on DiCarlo’s blog by a commenter. Harris Zwerling called for answers to the following questions if we’re to believe in value-added ratings:

1. Does the VAM used to calculate the results plausibly meet its required assumptions? Did the contractor test this? (See Harris, Sass, and Semykina, “Value-Added Models and the Measurement of Teacher Productivity” Calder Working Paper No. 54.)
2. Was the VAM properly specified? (e.g., Did the VAM control for summer learning, tutoring, test for various interactions, e.g., between class size and behavioral disabilities?)
3. What specification tests were performed? How did they affect the categorization of teachers as effective or ineffective?
4. How was missing data handled?
5. How did the contractors handle team teaching or other forms of joint teaching for the purposes of attributing the test score results?
6. Did they use appropriate statistical methods to analyze the test scores? (For example, did the VAM provider use regression techniques if the math and reading tests were not plausibly scored at an interval level?)
7. When referring back to the original tests, particularly ELA, does the range of teacher effects detected cover an educationally meaningful range of test performance?
8. To what degree would the test results differ if different outcome tests were used?
9. Did the VAM provider test for sorting bias?

Today, education historian Diane Ravitch published a piece titled "How to Demoralize Teachers", which draws all these problems together to highlight how counter productive the effort is becoming

Gates raises an important question: What is the point of evaluations? Shaming employees or helping them improve? In New York City, as in Los Angeles in 2010, it's hard to imagine that the publication of the ratings—with all their inaccuracies and errors—will result in anything other than embarrassing and humiliating teachers. No one will be a better teacher because of these actions. Some will leave this disrespected profession—which is daily losing the trappings of professionalism, the autonomy requisite to be considered a profession. Some will think twice about becoming a teacher. And children will lose the good teachers, the confident teachers, the energetic and creative teachers, they need.
[...]
Interesting that teaching is the only profession where job ratings, no matter how inaccurate, are published in the news media. Will we soon see similar evaluations of police officers and firefighters, legislators and reporters? Interesting, too, that no other nation does this to its teachers. Of course, when teachers are graded on a curve, 50 percent will be in the bottom half, and 25 percent in the bottom quartile.

Is this just another ploy to undermine public confidence in public education?

It's hard to conclude that for some, that might very well be the goal.

Bloomberg's brain dead brainwave

Billionaire Mayor of New York, and wannabe corporate education reformer Mike Bloomberg has suggested a radically absurd idea

he said, “you would cut the number of teachers in half but you would double the compensation of them, and you would weed out all the bad ones and just have good teachers.

“Double the class size with a better teacher is a good deal for students.”

Bloomberg's opinion is based upon a misguided and factually wrong premise, one he continues to hold to

Karen Matthews, a reporter with The Associated Press, asked, via Twitter, whether the mayor saw one teacher and 62 children as a good model. The mayor’s press secretary, Stu Loeser, shot back: “Are you asking as a journalist, advocate, or mom?”

No doubts haunt the mayor. In 2008 he insisted that class-size research was “unambiguous.”

“I don’t even understand why the subject comes up anymore,” he said, adding that all that mattered was teacher quality.

Let's examine class sizes and see if they matter. Michael C. Morrison, Ph.D. has analyzed 9,000 school districts to determine the impact on class sizes and graduation. His findings are unambiguous.

District probabilities for above average graduation performance are inversely related to district pupil-teacher ratios. As class size increases district probability for above average graduation performance decrease, controlling for district per capita income (a proxy for district socio-economic status) and district total revenue per student (a district proxy for programs and services).

Here's the graph of results

This isn't the only study of course, it's a subject that has been well and extensively researched. The National Bureau of Economic Research (NBER) found

“We find that assignment to a small class increases the probability of attending college by 2.7 percentage points, with effects more than twice as large among blacks. Among those with the lowest ex ante probability of attending college, the effect is 11 percentage points. Smaller classes increase the likelihood of earning a college degree by 1.6 percentage points and shift students towards high-earning fields such as STEM (science, technology, engineering and medicine), business and economics.”

Michael Morrison has detailed further studies on the subject, here.

As for Mayor Bloomberg, he doesn't practice what he is preaching

There’s a final oddity. Among the so-called meritocratic elite, low teacher-to-child ratios are beloved. The mayor’s daughters went to Spence, where classes hover from 10 to 15. Trinity, Dalton, Riverdale, Horace Mann: All charge $35,000 or more per year, and classes rarely exceed 12 in the lower grades.

Imagine if they packed those billionaire's kids into classrooms of 63!