markgarrison.net » a measure of failure

Review of “A Measure of Failure: The Political Origins of Standardized Testing”

Mark Garrison — Wed, 24 Feb 2010 14:17:31 +0000

The February issue of Counseling Today carries a review of my book by Aaron W. Hughey, professor of counseling and student affairs, Western Kentucky University.

The essay begins: “True or false: Proponents of standardized testing are, unintentionally or otherwise, pushing an agenda that intrinsically advocates racism and socioeconomic oppression. Intrigued? Then read on. It really is an understatement to claim that standardized testing permeates every aspect of our culture. Testing is employed in a variety of settings and affects our lives in ways that most of us do not even realize. Nowhere, however, is its influence more evident than in our educational institutions. Testing has become the driving force behind current efforts to reform our public schools. Unfortunately, it has a sinister dimension that has been beyond the comprehension of most of the general public — until now.”

Read the complete review.

Realism and Social Change

Mark Garrison — Mon, 22 Feb 2010 14:16:32 +0000

In corresponding with Cassiodorus about my book, the question of social change took center stage.

When you argue for social change you inevitably come up against the claims of “realism” — we can’t change this or that because to do so would be “unrealistic.” This is the argument typically favored by the incrementalists: “since we can’t change society as a whole,” they say, “let’s change little things, like the means we use to assess the quality of our public schools or the students entering college.”

My first response is this: the society is changing, and to deny that it is changing, and always changes, is unrealistic! Standardized tests are one tool being used to institutionalize and justify various changes — to curriculum, governance, and to the working conditions of teachers. The political and economic arrangements that were the conditions for the emergence of public education in the United States have been dramatically altered, and so, there is pressure on these institutions to “change” — this pressure is not simply coming from the Manhattan Institute and the Fordham Foundation … it is coming from history itself. Public schools have not been able to “equalize the conditions of man.” But to continue to apply standards of that past era, conditions that gave rise to standardized tests and their flawed assumptions, is unrealistic, and requires everyone to think creatively about alternatives. Incremental change can lead to qualitative change if the incremental change hits at what is key. So, the realism argument misses all this.

More generally, I think the realism argument needs to be to interrogated. What is established as possible (“realistic”) is itself a power play; the statement has multiple meanings. Asking for permission — “is it possible to take the day off”– is different from making an analysis of what the conditions as they exist right now make possible. For example, it is possible to eliminate hunger, in that enough food for all humans can be produced right now. Why this does not occur is mainly a political question. I say: Why limit discussion of alternatives with such talk of “realism”? Why be forced to choose between incremental and fundamental change? Why assume they are necessarily in contradiction with one another? It is only the incremental in place of or against the fundamental that I object to. So, again, I would not advocate right this minute eliminating standardized tests altogether, but I would advocate, as an incremental step, eliminating the use of tests for high stakes purposes. I oppose current merit pay schemes; I don’t oppose discussions about accountability; I do oppose discussions about accountability absent discussions about rights, for responsibilities and rights go hand in hand. I would also advocate that educators think broadly and openly debate standards for education — what kind of society do we want and what kind of education will support that aim? Without that orientation, discussions of the “incremental” will become stale, facile and uninspired. So, the big picture discussion is key to identifying what steps should be taken right now.

Are Tests Measures of Test Taking Ability?

Mark Garrison — Mon, 22 Feb 2010 13:32:15 +0000

In a recent discussion of my book, A Measure of Failure, the typical argument against any critique of standardized testing was issued in response to a favorable review of the book’s main points. In the comments we read: “A math test, such as the math portion of the SAT for instance, most certainly measures a student’s ability to do the math problems on the test. It is impossible to do well on such a test without the underlying skill that is required to do the math.” It seems hard to argue with this.

But the English language does not help the discussion of measurement, as measure can signify both a standard and the process of applying a standard for the purpose of measurement, assessment or comparison. Not all applications of standards produce measurements. Applications of legal standards do not yield measurements of criminality. So, to say that a test is the best available measure may be true if by measure one means the prediction of some performance. But prediction and measurement are not the same thing. Measurement is a very specific thing, a claim that a mathematical system corresponds with the phenomenon of interest. This is the criteria of being isomorphic. Standardized tests do not meet that criteria. And, they do not identify a precise object of measurement. Thus, claiming that one must have real knowledge of mathematics to perform “well” (high rank performance) on some math test is not the same as the claim that the math test produces a measurement of math ability. Of course one must have some related skills and general intellectual development to engage with the test in a way society renders valuable. But the outcome of that exercise does not constitute a measurement.

In the course of the discussion, it was argued that test scores are at least measures of test taking ability. My claim is that tests currently in use do not meet the criteria of measurement, and that this fact is hidden, covered over, but in reality, known to psychometricians. My claim is that these tests do not produce measurements of any kind (Walt Haney tried to convince me that they are “weak” measures, which created new problems). This is why I go to great lengths to distinguish between assessment and measurement. Standardized tests are obviously tools for making assessments. They’re just not measurements, and my claim is that this distinction is very significant.

I suppose that part of what is troubling about my argument is my strict use of the word measurement. So, for example, I would agree that a score on a standardized test is a “useful indicator” of how proficient a person is at taking standardized tests in general, but I would object to someone calling that score a measurement of test-taking ability. Creating an indices, Likert scale, etc., with the aid of numbers, may provide “useful” information, and even allow that information to be treated statistically (75% of Americans are opposed to the Iraq war) but the mere assignment of numbers to something in this manner does not in itself constitute measurement. Again, I maintain that the distinction is significant; it is significant that politicians and policy experts routinely call things measurements when the results do not meet the criteria of measurement.

The claim to measurement is made because it enables one to make claims about the origin of social trends. During the rise of intelligence testing, the claim that intelligence was being measured (even though it was known to be a mere classification) enabled reformers to link school performance to what they postulated as variation in intellectual ability (and not ineffective teaching, instruction in a language not spoken by students, or a vapid curriculum). Today, the claim to measurement is required to argue that “teaching ability” or “teaching effectiveness” is the cause of various social trends. No serious scientist believes that student performance on any academic test constitutes a measurement of teaching effectiveness. And, today, even though it is well established that is “normal” for individuals to vary in their rate and depth of learning any content or skill, the useless slogan “all children can learn” is shouted by reformers as if it represents the noblest aspirations of humanity. Even if social inequality were drastically reduced, individual (not group) performance on any valued task — intellectual, social, physical — would vary widely (and this in and of itself is not a social problem).

Finally, as seems to be common when anyone presents a challenge to standardized testing, critics are imputed with the aim of “throwing out the tests.” My book is quite clear that eliminating standardized testing as we know it — while leaving all else intact — would do little good and produce more harm. But blocking the use of high stakes tests would be a positive move. And as for being pegged an anti-tester, I’m the only one (I think) to critique the critics who say standardization is “bad”; again, my aim is to analyze these concepts and structures as they are rooted in definite social and political systems. Standardization in political terms is an advance, and part of the progressive notion of equality. In fact, the tendency now is to undermine, blow off, and ignore standard psychometric procedure (reliability, validity, etc.) and this is destructive and reflective of the larger trend of those in positions of power to act with impunity. As Gene Glass notes, most states don’t even produce the most basic test validation data.

But the actual point is that the standards adopted by a social system change as the system changes; the point is that this is a political fight, and that the fight over standards is political. By political I do not meant to narrowly refer to political parties, but rather I refer to the process by which a society decides who gets what, when, where, and how. Educators can’t wish away this political feature of standards. It is an argument that ultimately says that in order to address the flaws of standardized testing and policy that relies on testing, you have to address the major flaws of the present social system that are reflected in those tools and policies. The failure of “authentic assessment” is as much a political failure as a technical one.

markgarrison.net » a measure of failure

Review of “A Measure of Failure: The Political Origins of Standardized Testing”

Related posts

Realism and Social Change

Related posts

Are Tests Measures of Test Taking Ability?

Related posts