How Should Educators Interpret Value-Added Scores?

Via

Highlights

• Each teacher, in principle, possesses one true value-added score each year, but we never see that "true" score. Instead, we see a single estimate within a range of plausible scores.
• The range of plausible value-added scores -; the confidence interval -; can overlap considerably for many teachers. Consequently, for many teachers we cannot readily distinguish between them with respect to their true value-added scores.
• Two conditions would enable us to achieve value-added estimates with high reliability: first, if teachers' value-added measurements were more precise, and second, if teachers’ true value-added scores varied more dramatically than they do.
• Two kinds of errors of interpretation are possible when classifying teachers based on value-added: a) “false identifications” of teachers who are actually above a certain percentile but who are mistakenly classified as below it; and b) “false non-identifications” of teachers who are actually below a certain percentile but who are classified as above it. Falsely identifying teachers as being below a threshold poses risk to teachers, but failing to identify teachers who are truly ineffective poses risks to students.
• Districts can conduct a procedure to identify how uncertainty about true value-added scores contributes to potential errors of classification. First, specify the group of teachers you wish to identify. Then, specify the fraction of false identifications you are willing to tolerate. Finally, specify the likely correlation between value-added score this year and next year. In most real-world settings, the degree of uncertainty will lead to considerable rates of misclassification of teachers.

Introduction

A teacher's value-added score is intended to convey how much that teacher has contributed to student learning in a particular subject in a particular year. Different school districts define and compute value-added scores in different ways. But all of them share the idea that teachers who are particularly successful will help their students make large learning gains, that these gains can be measured by students' performance on achievement tests, and that the value-added score isolates the teacher's contribution to these gains.

A variety of people may see value-added estimates, and each group may use them for different purposes. Teachers themselves may want to compare their scores with those of others and use them to improve their work. Administrators may use them to make decisions about teaching assignments, professional development, pay, or promotion. Parents, if they see the scores, may use them to request particular teachers for their children. And, finally, researchers may use the estimates for studies on improving instruction.

Using value-added scores in any of these ways can be controversial. Some people doubt the validity of the achievement tests on which the scores are based, some question the emphasis on test scores to begin with, and others challenge the very idea that student learning gains reflect how well teachers do their jobs.

In order to sensibly interpret value-added scores, it is important to do two things: understand the sources of uncertainty and quantify its extent.

Our purpose is not to settle these controversies, but, rather, to answer a more limited, but essential, question: How might educators reasonably interpret value-added scores? Social science has yet to come up with a perfect measure of teacher effectiveness, so anyone who makes decisions on the basis of value-added estimates will be doing so in the midst of uncertainty. Making choices in the face of doubt is hardly unusual – we routinely contend with projected weather forecasts, financial predictions, medical diagnoses, and election polls. But as in these other areas, in order to sensibly interpret value-added scores, it is important to do two things: understand the sources of uncertainty and quantify its extent. Our aim is to identify possible errors of interpretation, to consider how likely these errors are to arise, and to help educators assess how consequential they are for different decisions.

We'll begin by asking how value-added scores are defined and computed. Next, we'll consider two sources of error: statistical bias and statistical imprecision.

[readon2 url="http://www.carnegieknowledgenetwork.org/briefs/value-added/interpreting-value-added/"]Continue reading...[/readon2]

What 10,000 teachers think

Scholastic and Bill & Melinda Gates Foundation have just released a survey of nearly 10,000 public school teachers titled "Primary Sources: America’s Teachers on the Teaching Profession". Teachers were asked about their schools and classrooms, about student and teacher performance and about the ways it should be evaluated, supported and rewarded. They shared their honest, professional opinions on everything from the role of standardized tests to teacher tenure, from family involvement to job satisfaction, from digital content to salaries.

In this survey, teachers told us:

• Raising Student Achievement Requires the Work Of Many

  – Teachers agree that their primary goal is helping all students learn and achieve, but a hardworking, committed teacher cannot do it alone.

  – Other factors that teachers identify as essential to raising student achievement include: family involvement, quality curriculum, and a community of educators and school leaders committed to the success of all students.

• Teaching and Learning Are Too Complex to Be Measured by One Test

  – Teachers are clear in their call for multiple measures of student achievement, and they say that standardized tests do not accurately reflect their students’ growth. In fact, we were surprised to learn that only 45% of teachers say their students take such tests seriously. – They also call for more frequent evaluation of their own practice from a variety of sources, including in-class observation, assessment of student work, and performance reviews from principals, peers and even students.

  – Teachers are open to tenure reform, including regular reevaluation of tenured teachers and requiring more years of experience before tenure is granted. On average, teachers say that tenure should be granted after 5.4 years of teaching, more than the typical two to three years in most states today.

• Challenges Facing America’s Schools Are Significant and Growing

  Teachers are concerned about their students’ academic preparedness. They tell us that, on average, only 63% of their students could leave high school prepared to succeed in college.

  When we asked veteran teachers to identify what is changing in their classrooms, they told us:

  – Academic challenges are growing. Veteran teachers see more students struggling with reading and math today than they did when they began teaching in their current schools.

  – Populations of students who require special in-school services are growing as well. Veteran teachers report increasing numbers of students living in poverty, students who are hungry and homeless, and students who have behavioral issues.

• School and Community Supports Are Essential to Keeping Good Teachers in the Classroom

  When asked to identify the factors that most impact teacher retention, teachers agree that monetary rewards like higher salaries or merit pay are less important than other factors

  – though some of these factors require additional funding – including strong school leaders, family involvement, high-quality curriculum and resources, and in-school support personnel.

Here are some of the interesting highlights from the survey, but be sure to check it all out yourself below.

Teachers work long hours

Teachers work, on average 53 hours a week, or 10 hours and 40 minutes a day.

Of the time during the required school day, here is how teachers reported it being spent

The survey has lots of information, including how teachers like to and would prefer not to, spend their time.

Classroom Issues

The graph below shows how teachers feel about class sizes (hint: they think it is very important)

And what do teachers think impact student achievement? (this one's a bit harder to read, but you can see it in large size in the document below). Common core appears at the bottom, family involvement at the top

On student assessments, teachers are in serious disagreement with current reform trends that lean heavily on standardized testing to measure student achievement.

Contentious Issues

Moving on to more contentious issues, reformers should take note - teachers could be your allies if you ever decide to collaborate. Higher salaries are viewed poisitively, but pay for performance is not seen as an important or driving force

But higher salaries fall well down the list of important issues for improving teacher retention. Parental involvement and quality leadership top the list

Finally, on tenure. Teachers do not think tenure ought to be automatic, or bad teachers protected by it. There should be a lot of opportunity for reformers and educators to collaborate in this area.

PRIMARY SOURCES: 2012

Starving America’s Public Schools

Critics of America’s public schools always seem to start from the premise that the pre-kindergarten-through-12th-grade public education system in this country is failing or in crisis.

This crisis mentality is in stark contrast to years of survey research showing that Americans generally give high marks to their local schools. Phi Delta Kappa International and Gallup surveys have found that the populace holds their neighborhood schools in high regard; in fact, this year’s survey found that “Americans, and parents in particular, evaluate their community schools more positively than in any year since” the survey started.

How could there be such a disconnect between a national narrative about public education and opinions about local schools? The two contradictory narratives draw on completely different sources of evidence.

Read the report.

[readon2 url="http://www.ourfuture.org/report/2011104111/starving-america-s-public-schools"]Read more...[/readon2]

School Cuts Hurt Kids. . . 

. . . By Increasing Class Size

. . . By Eliminating Critical Classes

. . . By Eliminating Great Teachers

How money buys education "research"

The Center for American Progress (CAP), which bills itself as a being dedicated to improving the lives of Americans through progressive ideas and action, recently produced a report titled "Charting New Territory: Tapping Charter Schools to Turn Around The Nation’s Dropout Factories"

The report argues for a more prominent role for charter operators in turning around perennially low-performing high schools. Among its recommendations

the report posits that five steps might improve the likelihood of successful CMO-district partnerships (all of which strengthen the CMO’s position in the district):
1) maximizing theCMO’s autonomy over staffing, budget, curricula, operations, and pedagogy;
2) staffing turnaround schools through creative agreements among education entrepreneurs, unions, charter operators, districts, and states, such as developing thin union contracts;
3) ensuring district financial support for turnaround schools;
4) relaxing state and district administrative regulations around staffing, funding, and school operations; and
5) cultivating public will for such partnerships.

At this point, you might be wondering why a progressive think tank is advocating such right wing policies that have been proven to be unsuccessful. The answer is actually quite simple to descern, and can be found on the very first page of this CAP report.

Paid for by the conservative corporate education reform outfit - the Eli Broad Foundation.

The National Education Policy Center has just released their analysis of this report, and they don't have kind things to say about this Broad funded report.

The report bases the majority of its findings and conclusions on conversations with charter school operators—including those that have not yet engaged in turnaround work—and with school district staff, researchers, and education reformers or consultants. Interview respondents included one professor of educational policy, one researcher from the Center on ReinventingPublic Education, five reformers or consultants from reform organizations or think tanks that advocate for market-based education policies, and three district administrators who were associated with their districts’ charter school partnerships.

Secondarily, the report cites evidence from the popular media, blogs, foundation reports, non-peer reviewed literature, charter operators’ external relations materials, and ideologically identifiable think tanks.

Beyond these citations, the report routinely offers a range of unsubstantiated claims that are not supported by any evidence or that ignore existing evidence to the contrary.

At the same time, no theoretical or substantive rationale behind the report’s sources of evidence is provided to justify why the particular interview respondents or literature sources were selected or how their data were evaluated. The result is a collection of weakly supported claims based on an unsystematic, unsophisticated interpretation of the knowledge base on school turnarounds, charter schools, and charter management organizations.

This is what millions of dollars can buy you. Research and recommendations that lack intellectual rigor. Designed to further corporate education reform agendas at the expense of public education, and the possibility of real reforms and changes that would make a difference to the quality of education students receive.

View REVIEW OF CHARTING NEW TERRITORY

A balanced approach to the budget

The Republican controlled legislature and the Governor have determined that the state budget will be balanced on the back of public education and the middle class. Much of the solution will pass the buck to local government and school districts, by eliminating the tangible personal property tax (TPP) replacement money, leaving many school districts in dire straits. Further costs are passed onto working Ohioans with a 2% pension shift, directly extracting hundreds of millions from paychecks over the next 2 years alone. A significant portion of the money that is left over for public education is diverted to expand charter schools and vouchers. A plan even conservative charter proponents oppose. Much of the rest of the budget gap is filled with one time money from the privatization of Ohio's assets, from roads and prisons, to the lottery and alcohol.

Clearly a more balanced approach would be welcome.

A short while ago we discussed and demonstrated that the income tax cuts made through HB66 in 2004 has had a negligible effect on a typical teachers tax burden. For those tax cuts to make a real tangible difference one would have to earn significantly more than any Ohio teacher - hundreds of thousands of dollars more in fact.

A recent study by the Education Tax Policy Institute (ETPI), found

that Ohio ranks as the 17th lowest state in taxes levied at the state level. Increases in local taxes place Ohio with a total state and local tax burden at about the national average. Ohio's ranking among states is higher when local taxes are included in the comparison because state policies have shifted the responsibility for funding public services, including education, down to the local level.

While the study determined that the reform of Ohio's business tax structure in 2005 improved both the fairness and the competitiveness of the state's revenue system, the authors also determined that the combination of the recent recession and the changes in the state’s tax system in House Bill (HB) 66 lowered state revenue by $3 billion below the levels anticipated in 2005 when the state revamped its business tax structure.

The summary can be read here, with the full report is below. The report demonstrates that Ohio does not have the crushing tax burden that some would have us believe, but instead has areas where improvements could be made that would raise revenue without causing any economic harm.

The following two charts from the report show the sources of tax revenue, the first for the state, the second for local taxes.

State Sources of Tax Revenue

Local Sources of Tax Revenue

As you can see, business pays very little, with the vast majority of taxes being paid by individuals through income tax, property tax and sales taxes. One would hardly have to be draconian to realize significant tax revenues from simply closing outdated loopholes

COLUMBUS, Ohio - Ohio's cash-strapped government could generate an additional $300 million a year by closing tax loopholes that are outdated, benefit a few or do little to spur economic growth, a trio of policy think tanks spanning the political spectrum told state leaders Monday.

You can see some of the crazy tax loopholes in this document

The Tax Expenditure Report Ohio

In total those tax exemptions add to over $7 billion.

The evidence is overwhelming that an alternative, more balanced approach to the budget is available to law makers should they choose. They do not have to continue on this path that seriously harms public education and the middle class. They can solve the budget problems without passing the buck down to local government. The question is, will they chose a new path?

REVENUE OPTIONS FOR OHIO’S FUTURE