The LA Times has provoked the wrath of local teachers by posting a database of teacher performance based on a metric called “value added,” a measurement that supposedly controls for the quality of incoming students, English proficiency, individual behavior problems, and so forth. (Q&A here.) But a new EPI report says that value-added sucks anyway:
One study found that across five large urban districts, among teachers who were ranked in the top 20% of effectiveness in the first year, fewer than a third were in that top group the next year, and another third moved all the way down to the bottom 40%. Another found that teachers’ effectiveness ratings in one year could only predict from 4% to 16% of the variation in such ratings in the following year. Thus, a teacher who appears to be very ineffective in
one year might have a dramatically different result the following year. The same dramatic fluctuations were found for teachers ranked at the bottom in the first year of analysis.
Education expert Kevin Carey agrees that value-added is a lousy metric:
But, and this is an enormous caveat, everything else we currently use is worse. A teacher’s years of experience, their education credentials, their certification status, the prestige of their college or their college GPA, even in-class observations. None of these measures does as good of a job at predicting a student’s academic growth as a teacher’s value-added score. Yet, we continue to use these poor proxies for quality at the same we have such passionate fights about measures of actual performance.
Still, it remains true that virtually everyone agrees that value added is “not all that great” (Carey’s conclusion). So what should we do instead? Carey says we just don’t know. We have lots of minimum standards for teaching quality (no uncertified teachers, no violence, no classroom drunks, no overuse of mimeographed worksheets, etc.) but we don’t know much about what makes a teacher stellar:
What kind of teaching is as good as mimeographed worksheets are bad? We don’t really know. The qualifications-and-competence mindset doesn’t allow us to know. We can’t see it, and so gradually we allow policies and institutions and organizational cultures to evolve that pretend it doesn’t exist.
….There’s a natural tendency to proceed from here to the conclusion that we need to intensively study [great teachers like LA’s Zenaida Tan] so we can help others be more like them. And we should, this will be valuable research. But we ought not expect it will produce a new list of qualifications and competencies to which every teacher must adhere. Just as there are many different kinds of great writers and lawyers and artists, so too does great teaching come in all manner of variations. This should be seen as entirely positive for the teaching profession. The jobs worth having — and worth paying for — are those that can’t be wholly reduced to definable rules.
Yet the union that purports to represent Tan has done nothing but oppose the creation of the only measures that accurately identify her value as a professional. In doing so, it helps depress the public understanding of all teachers as professionals. If the LA Times hadn’t performed these value-added calculations and published them, who would have? How long do great teachers have to wait to be recognized? How long are they going to be held hostage to a mindset that pretends they don’t exist?
This is no surprise, I guess, since we have so little idea of what makes someone great at any profession. What makes a product manager great? Or a CEO? Or a magazine editor? Or a blogger? No one knows. If you go to the business section of your local Barnes & Noble you can find a hundred books with a hundred different vague and unhelpful answers based on little more than the author’s instincts. Talk to business professors and you’ll get some different answers, but probably not ones that are an awful lot more reliable.
But the problem with teachers is that assessing their performance isn’t just hard, it’s even harder than any of those other professions. Product managers interact closely with a huge number of people who can all provide input about how good they are. CEOs have to produce sales and earnings. Magazine editors and bloggers need readers.
But teachers, by definition, work alone in a classroom, and they’re usually observed only briefly and by one person. And their output — well-educated students — is almost impossible to measure. If I had to invent a profession where performance would be hard to measure with any accuracy or reliability, it would end up looking a lot like teaching.
So this means we end up using things like value-added, even though we know they’re not very good. What other choice do we have, after all? Sara Mead recommends greater reliance on “validated and reliable observational tools, such as the Classroom Assessment Scoring System (CLASS), that look at teacher classroom behaviors and measure the extent to which teachers are implementing behaviors linked to improved student outcomes,” and that sounds like a good idea to me. But even if this works, it will take years or decades to produce usable results. What do we do in the meantime?
The criticisms of value-added seem compelling. At the same time, if a teacher scores poorly (or well) year after year, surely that tells us something? At some point, we either have to use this data or else give up on standardized testing completely. It just doesn’t make sense to keep using it if we don’t bother taking the results seriously.
