Why Test Scores CAN'T Evaluate Teachers

From the National Education Policy Center. the entire post is well worth a read, here's the synopsis

The key element here that distinguishes Student Growth Percentiles from some of the other things that people have used in research is the use of percentiles. It's there in the title, so you'd expect it to have something to do with percentiles. What does that mean? It means that these measures are scale-free. They get away from psychometric scaling in a way that many researchers - not all, but many - say is important.

Now these researchers are not psychometricians, who aren't arguing against the scale. The psychometricians as who create our tests, they create a scale, and they use scientific formulae and theories and models to come up with a scale. It's like on the SAT, you can get between 200 and 800. And the idea there is that the difference in the learning or achievement between a 200 and a 300 is the same as between a 700 and an 800.

There is no proof that that is true. There is no proof that that is true. There can't be any proof that is true. But, if you believe their model, then you would agree that that's a good estimate to make. There are a lot of people who argue... they don't trust those scales. And they'd rather use percentiles because it gets them away from the scale.

Let's state this another way so we're absolutely clear: there is, according to Jonah Rockoff, no proof that a gain on a state test like the NJASK from 150 to 160 represents the same amount of "growth" in learning as a gain from 250 to 260. If two students have the same numeric growth but start at different places, there is no proof that their "growth" is equivalent.

Now there's a corollary to this, and it's important: you also can't say that two students who have different numeric levels of "growth" are actually equivalent. I mean, if we don't know whether the same numerical gain at different points on the scale are really equivalent, how can we know whether one is actually "better" or "worse"? And if that's true, how can we possibly compare different numerical gains?

[readon2 url="http://nepc.colorado.edu/blog/why-test-scores-cant-evaluate-teachers"]Continue reading...[/readon2]

Where the polls stand - 2 weeks to go

With just one Presidential debate to go, and a little more than two weeks remaining, campaign 2012 continues to be a close affair. Real Clear Politics has the race essentially tied, with Obama favored to win 201 electoral college votes and Mitt Romney 206, with 131 up for grabs.

Despite the narrowing of the national polling, President Obama continues to enjoy a small but persistent lead in swing states, including the all important Ohio

The NYT calculates that this persistent lead is generating a 70.3% chance of President winning Ohio on November 6th

These persistent swing state leads have 538 projecting the President to win 288 electoral votes to Mitt Romney's 255.

Swing state education survey

Some poll results were released recently that delved into the minds of voters in swing states, and their attitudes towards various education topics. There was a lot of positive views expressed, that perhaps run counter to many of the news stories one reads in the local papers.

Here's some select findings

• Education is a top tier issues. 67% say education will be extremely important to them personally in this year’s elections for president and Congress.
• Education is a major economic issue. 34% of voters select “improving education at the elementary, secondary, and post-secondary levels” as one of the top two priorities for getting America’s economy back on track, which ranks in the top tier with “reducing our dependence on foreign oil” (39%) and “reducing the federal budget deficit” (32%). These goals rank as higher priorities than reducing taxes and regulations on business, addressing trade issues, or modernizing transportation infrastructure.
• In light of the previous topic, law makers should note that 78% of voters say that increased funding for education is necessary, including 44% who say it is definitely necessary. Just 21% say it is not necessary.
• 55% say they would be willing to pay $200 more per year in taxes to provide increased education funding
• A note to Cleveland Mayor Frank Jackson, at least half of voters feel it is extremely important to make sure schools continue to provide arts, music, and physical education classes for all students (59% extremely important)
• And a note to the union bashers. When asked which one or two of seven groups have the greatest responsibility for improving education, slightly more than half (52%) of voters hold parents of students most accountable. This notably surpasses the proportion who place responsibility on the shoulders of teachers (31%), elected officials (26%), and society in general (22%). School administrators (15%), students (13%), and teachers unions (5%) are cited least often.
• 90% of voters feel it is extremely (69%) or fairly (21%) important for their governor and state legislature to address the issue of education as a matter of state policy.
• 44% of voters say that the Democratic Party reflects their priorities on the issue of education very or fairly well, while 31% feel the same way about the Republican Party. Among the crucial bloc of independent voters, 40% feel that the Democratic Party reflects their priorities, while 26% feel that way about the Republican Party.

You can read the entire survey below.

College Board Education Survey Key Findings

Poor schools can’t win

Without question, designing school and district rating systems is a difficult task, and Ohio was somewhat ahead of the curve in attempting to do so (and they’re also great about releasing a ton of data every year). As part of its application for ESEA waivers, the state recently announced a newly-designed version of its long-standing system, with the changes slated to go into effect in 2014-15. State officials told reporters that the new scheme is a “more accurate reflection of … true [school and district] quality.”

In reality, however, despite its best intentions, what Ohio has done is perpetuate a troubled system by making less-than-substantive changes that seem to serve the primary purpose of giving lower grades to more schools in order for the results to square with preconceptions about the distribution of “true quality.” It’s not a better system in terms of measurement – both the new and old schemes consist of mostly the same inappropriate components, and the ratings differentiate schools based largely on student characteristics rather than school performance.

So, whether or not the aggregate results seem more plausible is not particularly important, since the manner in which they’re calculated is still deeply flawed. And demonstrating this is very easy.

Rather than get bogged down in details about the schemes, the short and dirty version of the story is that the old system assigned six possible ratings based mostly on four measures: AYP; the state’s performance index; the percent of state standards met; and a value-added growth model (see our post for more details on the old system). The new system essentially retains most of the components of the old, but the formula is a bit different and it incorporates a new “achievement and graduation gap” measure that is supposed to gauge whether student subgroups are making acceptable progress. The “gap” measure is really the only major substantive change to the system’s components, but it basically just replaces one primitive measure (AYP) with another.*

Although the two systems yield different results overall, the major components of both – all but the value-added scores – are, directly or indirectly, “absolute performance” measures. They reflect how highly students score, not how quickly they improve. As a result, the measures are telling you more about the students that schools serve than the quality of instruction that they provide. Making high-stakes decisions based on this information is bad policy. For example, closing a school in a low-income neighborhood based on biased ratings not only means that one might very well be shutting down an effective school, but also that it’s unlikely it will be replaced by a more effective alternative.

Put differently, the most important step in measuring schools’ effectiveness is controlling for confounding observable factors, most notably student characteristics. Ohio’s ratings are driven by them. And they’re not the only state.

(Important side note: With the exception of the state’s value-added model, which, despite the usual issues, such as instability, is pretty good, virtually every indicator used by the state is a cutpoint-based measure. These are severely limited and potentially very misleading in ways that are unrelated to the bias. I will not be discussing these issues in this post, but see the second footnote below this post, and here and here for some related work.)**

The components of the new system

The severe bias in the new system’s constituent measures is unmistakable and easy to spot. To illustrate it in an accessible manner, I’ve identified the schools with free/reduced lunch rates that are among the highest 20 percent (highest quintile) of all non-charter schools in the state. This is an imperfect proxy for student background, but it’s sufficient for our purposes. (Note: charter schools are excluded from all these figures.)

The graph below breaks down schools in terms of how they scored (A-F) on each of the four components in the new system; these four grades are averaged to create the final grade. The bars represent the percent of schools (over 3,000 in total) receiving each grade that are in the highest poverty quintile. For example, looking at the last set of bars on the right (value-added), 17 percent of the schools that received the equivalent of an F (red bar) on the value-added component were high-poverty schools.

[readon2 url="http://shankerblog.org/?p=5511"]Continue reading[/readon2]

Parents Agree – Better Assessments, Less High-Stakes Testing

Educators aren’t alone in being fed up with narrow, punitive student accountability measures. Parents also want well-designed, timely assessments that monitor individual student performance and progress across a range of subjects and skills. That’s one of the key findings in a new study by the Northwest Evaluation Association (NWEA).

NWEA, a non-profit educational services organization headquartered in Portland, set out to find how the views of parents – often ignored in the debate over the direction of public education – stacked up against those of teachers and administrators.

After conducting online surveys of more than 1,000 respondents, NWEA found that these stakeholders essentially want the same thing. Large majorities say that, although year-end tests might provide some sort of useful snapshot, they strongly prefer more timely formative assessments to track student progress and provide educators with the flexibility to adjust their instruction during the school year.

“The research reinforces the notion that no one assessment can provide the breadth and depth of information needed to help students succeed,” explained Matt Chapman, president and CEO of NWEA. “For every child we need multiple measures of performance.”

As the reauthorization of the Elementary and Secondary Education Act (ESEA) slowly moves on Capitol Hill, redefining how student progress is measured will be a key debate. The National Education Association believes it is time to move beyond the No Child Left Behind Law (the 2001 revision of ESEA), scrap the obsession with high-stakes testing and enter into a new phase of education accountability.

“Well-designed assessment systems do have a critical role in student success,” said NEA President Dennis Van Roekel. “We should use assessments to help students evaluate their own strengths and needs, and help teachers improve their practice and provide extra help to the students who need it.”

“I use different types of assessments because all students are different,” explained Krista Vega, a middle school teacher in Maryland and NEA member who participated in the NWEA survey. “I use quizzes, games, teacher-made tests, computerized tests, portfolios, and alternative assignments.”

“What I’m looking for is, first, are they mastering the skill I’m trying to teach, or did they not master the skill? I’m looking to see if there is an area of weakness. I’m looking to see if they have background knowledge sometimes. There’s just a whole range of things that I’m looking for,” Vega said.

Source: Northwest Evaluation Association and Grunwald Associates

According to the survey, it is the types of formative assessments Vega identifies, such as quizzes, portfolios, homework and end-of-unit tests that provide timely data about individual student growth and achievement. Respondents cited these types of assessments as providing educators with the necessary information to pace the instruction and ensure students learn fundamental skills.

Parents are also worried about the narrowing of the curriculum. Large majorities believe it is important to measure students in math and English/language arts but also say it is important to measure performance in science, history, government and civics, and environmental literacy.

The students who are often hurt the most by a restricted curriculum are those who don’t have the opportunities, because of their socioeconomic background, to diversify their learning outside the classroom.

Beyond subject matter, parents and educators believe so-called “higher order” thinking skills such as creativity, communication, problem-solving, and collaboration – so critical in the modern economy and workplace – aren’t being properly measured by current assessment systems.

“It is really, really important,” Vega says “that we prepare students for when they enter the workforce to compete in the 21^st century.”

Read the NWEA Report ”For Every Child, Multiple Measures”

More on NEA’s Position on Student Assessments (Word Document)

Cleveland School Plan Needs Work

This is a very level headed and reasonable approach to modify Frank Jackson's corporate education reform plan into something that might work and would bring more people willingly into the process.

Columbus: Innovation Ohio, a progressive think tank headquartered in Columbus, today released an analysis of the education reform plan recently put forward by Cleveland Mayor Frank Jackson. Governor Kasich has indicated the plan might serve as a model for his own education reform effort, which presumably will include the new school funding formula he promised but so far has failed to deliver. The analysis is available at www.innovationohio.org.

IO said an analysis of the “Cleveland Plan” is important given Ohio’s history of expanding Cleveland education experiments, such as private school vouchers, state-wide. “If Governor Kasich is intent on using the Cleveland Plan as a model for other Ohio school districts, then it’s critical that we get it right,” said IO President Janetta King.

The analysis found a number of “things to like” about the Cleveland Plan, including:

• Innovations such as a Global Language Academy, an Environmental Science School, Early Childhood Education Academies in every neighborhood, and an English Immersion School for all children for whom English is a second language;
• A focus on high-quality preschool education, as well as on college and workforce readiness; and
• A series of proposed changes to state law that would, for example, give the Cleveland Metropolitan School District flexibility to manage its fiscal assets and close loopholes in existing law that allow poorly-performing Charter Schools to continue operating.

IO said other ideas, like adoption of a year-round school calendar, support for high-quality Charter Schools, and the aggressive pursuit of talented teachers, “have potential, but need more work and further fleshing-out.”

But Innovation Ohio said several Cleveland Plan ideas are fatally flawed as currently written and should either be modified substantially or jettisoned entirely. Among these are:

• A proposal to allow the transfer of local property tax revenue to Charter schools;
• The transfer of school oversight and other functions from the Cleveland School Board (accountable to the Mayor) to an unelected and less accountable “Cleveland Transformation Alliance”;
• A weighted per pupil funding formula with “money following the child” that, in IO’s view, would inevitably end up short-changing either students or schools;
• Several proposals relating to teacher compensation, collective bargaining and accountability, which IO says are exact replicas of provisions in last year’s Senate Bill 5, which Ohio voters overwhelmingly rejected with 61% of the vote in November.

Said IO President Janetta King:

“IO congratulates the authors of the Cleveland Plan for thinking outside the box and being willing to go big. Nothing is more important to Ohio’s future than our schools and our kids. That’s why education reform is so important, and it’s why all of us who truly care about our state, Republicans and Democrats, conservatives, liberals and moderates alike–must be willing to embrace change and challenge the status quo.

“But our goal cannot be change for the sake of change, or change that can’t work and will only make things worse. So Innovation Ohio has tried to be constructive in our analysis. Where we’ve been critical of the Cleveland Plan, we’ve offered alternative ideas and proposals that we believe are more likely to achieve the desired goals.

“But we recognize that we don’t have all the answers. Frankly, neither do the people who put the Cleveland Plan together. And that is why we believe any serious school reform discussion should and must include the voices of professional educators, parents, and other members of the community. We hope their exclusion will be rectified in the weeks and months ahead.

“So what is Innovation Ohio’s bottom-line take on the Cleveland Plan? We believe the Plan as written is a reasonable place to start, but would be a terrible place to end up. It needs work and IO stands ready to help any way we can.”