The problem of testing

Testing Issues
Added to the shortcomings of the standards movement is the recent heavy emphasis on high-stakes testing to determine the achievement of the standards. Decades of work have highlighted the effectiveness of authentic assessments (portfolios, student exhibitions, scoring rubrics, etc.) as tools for informing teachers’ instructional practices and methods for communicating to students and parents the knowledge that has been gained. However, the current practice in testing is the standardized, norm-reference test, consisting almost wholly of multiple choice questions. It is problematic that policy makers would on one hand mandate the development of elaborate content standards only to couple such policies with low-level and narrow assessments (Nichols and Berliner, 2007, Dorn 2007)
The long and troubled history of testing and test development in the United States was severely damaged by racism, which the test producers have yet to overcome (Berlak, 2000; Valdéz & Figueroa, 1994). Standardized, usually multiple choice, tests are preferred because they can be mandated by political leaders, implemented, and deliver clear results. But, as in the case of positivism and reductionism (from which these tests come), they are measuring and evaluating only a small segment of the important learning goals of schools. Low level tests do not measure human relations, respect, civic courage, and critical thinking, for example. Standardized testing is a political act that often forces teachers to change their teaching strategies. Teachers need to examine the limits of the testing processes and use classroom based assessments to inform their teaching. In many states, including Texas, California, and Massachusetts, and in many school districts, including New York, Chicago, and Philadelphia, standards and a test driven curriculum have been used to reduce teacher professional choice and decision-making.
The most basic failure of the testing/accountability model was to refuse to recognize that public education is far more than production; it includes, at a minimum, facts, concepts, generalizations, skills, attitudes, critical thinking, and citizenship. Thus, a business-production model, based on low-level, multiple choice testing, only measures a small part of the important issues of schools in a democratic society (see Renzulli, 2002, Dorn 2007). Meanwhile, governors and legislators committed to the testing movement ignore other parts of the business production model, including providing as much support, including tools, training, and technical assistance for the workers as needed.
Testing systems have grown in part because they are very profitable for the companies that produce and score these low-quality tests—companies that lobby the legislatures to establish testing systems. State funding for testing grew in Texas from $19.5 million in 1995 to $ 68.6 million in 2001 and at similar rates in other states (Gluckman, Jan. 2002). Bloomberg News estimated in 2006 that the testing industry makes over $ 2.5 billion per year ( Gloven and Evans, 2006). Funding increases for testing and test preparation usually is matched by a reduction in funding for other classroom items such as textbooks, dictionaries, libraries, and teacher support. In spite of these large investments test-based accountability systems, without major improvements in the quality of testing and investments in teacher capacity-building, will not produce significant improvement in student achievement in high-risk neighborhoods (Kober, 2001; Popham, 2003, Nichols and Berliner, 2007).
Extensive evidence shows that the current testing emphasis has driven instruction away from important issues of developing democratic and multicultural content, away from critical thinking, and away from the development of citizenship and prodemocratic values (Neill, 2003; Renzulli, 2002). Available testing, particularly multiple choice testing, is not the only form of assessment. Other assessment devices include teacher observations, rubrics, student presentations, and portfolios ( Wood, Darling-Hammond, Neil, and Roschewski, 2007 ) These forms of assessment can be used to measure progress on goals of critical thinking, democracy, and important multicultural goals such as mutual respect.
Scores on most standardized skill tests actually teach us very little; they measure very imprecisely. Current objective tests measure whether the student can identify letters, words, and rhyming words, but they do not measure comprehension of a paragraph or the ability to write a creative essay. They measure skills and isolated facts rather than significant academic achievement. Tests are usually not actual measures of competencies, but measures of isolated skills that can be drilled without improving the student’s education. Rather than investing more money in the current low-quality testing systems, we could develop appropriate and useful assessments, including using computer technology, which would help the teacher. There are good uses for standardized testing. They should be short tests given frequently that assist the teacher in making decisions about individual students, teaching and review. But that is now what is happening with testing in k-12 today.
From: Choosing Democracy; a practical guide to multicultural education. 4th. ed. 2010. p. 381.
Duane Campbell

Arne Duncan; does he understand the limits of data?

http://www.ed.gov/news/speeches/2009/07/07022009.html (at the end there are now 6 other replies that precede mine; most are quite good). Monty Neill

Duncan: But sometimes, despite our best efforts, these methods don't work. Today, America has about 5000 schools that continue to underperform year after year, despite our best efforts.

Comment: The number apparently represents those schools not making AYP for 5 years or so. Some of those schools may well be in very bad shape, others may have been making steady progress but started so far behind as to never make AYP. Some may be dysfunctional, others functional but needing more help for the school and in the communities from which the students come. How do we know the difference?
And how do we know they have received "our best efforts," especially since until a year ago the improvement fund for Title I was essentially non-existent? Whose best efforts, how are we to know they were the "best"? Were they strong efforts to do things Duncan listed immediately previous to this point (e.g., "we have tried boosting support for teaching staff and making other changes around curriculum, school day, etc.—and sometimes it has worked. I always favor more support, collaboration, mentoring and time on task")? Is it therefore necessary to take an extreme action, like privatize control of the school, or is it simply time to finally help schools get better on their own?
I am not opposed to extreme measures if a school really is dysfunctional after serious efforts at assistance. The Forum on Educational Accountability, which I chair, has stated that in the end states are responsible for their schools and students don't deserve perpetually non-functional school (see "Empowering Schools and Improving Learning," at http://www.edaccountability.org). But before generally untried nostrums - ones with only anecdotal evidence to support them, ones that seem to fail as much or more than succeed (e.g., charters, on average) - far more careful thought must go into what it takes to improve seriously troubled schools, what kinds of in-school and in-community supports are needed, etc.
Duncan goes on to cite the urgency of the situation. But doing something that has no evidence it will work in terms of improved student learning (more than test scores, however) and completion rates, and that simultaneously undermines democratic control over schools (which turning schools over to private operators very fundamentally does), is to respond via panic not thoughtful action -- if the agenda is systemic, sustained improvement.
Duncan: Now let's talk about data. I understand that word can make people nervous but I see data first and foremost as a barometer. It tells us what is happening. Used properly, it can help teachers better understand the needs of their students. Too often, teachers don't have good data to inform instruction and help raise student achievement.

Data can also help identify and support teachers who are struggling. And it can help evaluate them. The problem is that some states prohibit linking student achievement and teacher effectiveness.

I understand that tests are far from perfect and that it is unfair to reduce the complex, nuanced work of teaching to a simple multiple choice exam. Test scores alone should never drive evaluation, compensation or tenure decisions. That would never make sense. But to remove student achievement entirely from evaluation is illogical and indefensible.

It's time we all admit that just as our testing system is deeply flawed—so is our teacher evaluation system—and the losers are not just the children. When great teachers are unrecognized and unrewarded—when struggling teachers are unsupported—and when failing teachers are unaddressed—the teaching profession is damaged.

Comment: There are many problematic points here. While he describes the current testing system as "far from perfect" and "deeply flawed," he criticizes prohibitions on linking that data to teachers as a way to judge "performance." He seems to equate "data" with test scores, since he at no points suggests data is anything else. Indeed, "teachers need good data," but that must be far more than scores on the mediocre to lousy tests that now exist, from state exams to "benchmark" tests to travesties such as DIBELS.

So when he says, "But to remove student achievement entirely from evaluation is illogical and indefensible," he means that part of evaluating teachers should be their ability to raise test scores. As Alfie Kohn phrased it so sharply, this leads to "raising the scores and ruining the schools." And if some schools are already "ruined," focusing on test scores won't lead to a situation in which, to again cite Duncan, the nation can "give children the very best education possible." Schools that focus on boosting test scores do no such thing. (FairTest regularly summarizes this evidence in our quarterly Examiner newsletter and various reports; www.fairtest.org.)

Duncan then moves on to payment by results - "When great teachers are unrecognized and unrewarded." He has made clear on other occasions he very much supports payment by results, though he regularly cautions he wants to do this with teachers. But what if the teachers refuse? Meanwhile, Duncan has echoed Eli Broad in claiming that payment for performance is common in other fields. As the recent EPI book by Rothstein makes clear, it is not common among professions, and where implemented it brings about goal distortion, gaming the system, and bad consequences. George Madaus has looked at paying teachers for results in Ireland, others have looked at England, and the situation is the same: it does not work.

Again, we are in a situation in which Duncan insists that due to the dire situation, we must "do something." But the something, in this case, not only has no evidence it will not work, it has clear evidence it will not work.

Two points are here inter-twined: payment for results, and using test scores to define the results. The first has not worked in other fields, the second compounds the damage and will further intensify the score inflation now seen across the states (and that will plague a national test as well).

Teachers and their unions should flat out refuse payment for results. They will of course be attacked as protecting themselves at the expense of their students -- but the truth is, they are also protecting the children from the systemic malfeasance of allowing standardized tests to control curriculum and instruction.

Duncan is correct, as many people have pointed out, that evaluation of teachers is largely a farce - for many reasons. It should be greatly improved, first of all in order to help teachers get better. It must be tied to very different forms of professional development than the trivial time-wasters that have given PD a bad name among teachers. And assessment must be overhauled, not to have "better" ways to institute payment for results, but to have good information about student learning that students, teachers, administrators and other professionals, parents, communities and states can use to improve real learning (guide, not drive, action, as Deborah Meier puts it). FEA has much to say about most of these points in our various reports such as Redefining Accountability and the report of the Expert Panel on Assessment, at http://www.edaccountability.org (not teacher or principal evaluation, however; but see the National Staff Development Council materials).

So there is much to do, and the stimulus funds as well as an overhauled ESEA can be used to improve assessment in line with FEA and FairTest recommendations, to overhaul professional development in line with FEA recommendations, to make data mean something far more than test scores, to focus on improving school capacity to serve all children well, to educate the whole child, and to pay attention to the consequences of racism and poverty that plague so many communities and their schools. Payment for "performance" is ultimately a distraction, as are national standards and a national test, from the real work of improving schools. It is time to re-focus and move in more useful ways.

--
Monty Neill, Ed.D
Deputy Director
FairTest
15 Court Sq, Ste 820
Boston, MA 02108
monty@fairtest.org
857-350-8207; fax 850-357-8209
www.fairtest.org

Getting Accountability Right

Getting Accountability Right

By Richard Rothstein
The federal No Child Left Behind Act has succeeded in highlighting the poor math and reading skills of disadvantaged children. But on balance, the law has done more harm than good because it has terribly distorted the school curriculum. Modest modifications cannot correct this distortion. Designing a better accountability policy will take time. We cannot and should not abandon school accountability, but it's time to go back to the drawing board to get accountability right.

The first step is to understand today's curricular distortion. It has arisen because No Child Left Behind holds schools accountable for only some of their many goals. When we demand adequate math and reading scores alone, educators rationally respond by transferring resources to math and reading instruction (and drill) from social studies, history, science, the arts and music, character development, citizenship education, emotional and physical health, and physical fitness.

This shift has been most severe for the disadvantaged children the law was designed to help, because they are most at risk of failing to meet the math and reading targets. But they are also most at risk of losing curricular opportunities in other domains. In these other areas, NCLB has widened the "achievement gap."

President Barack Obama has vowed to correct this distortion. He has noted that NCLB "has become so reliant on a standardized-test model that ... subjects like history and social studies have gotten pushed aside. Arts and music time is no longer there. So the child is not having the well-rounded educational experience I benefited from and most in my generation benefited from." We must change No Child Left Behind, he has said, "so that the assessment is one that takes into account all the factors that go into a good education."

Although some Democrats and Republicans want to ignore the law's goal distortion, observers with varying policy perspectives share the new president's view that NCLB requires a radical reconsideration. The Center on Education Policy, headed by Jack Jennings (formerly an aide to Democrats on the House education committee), has publicized the loss of instruction in social studies, science, the arts, and physical education, especially for disadvantaged children. Chester E. Finn Jr. and Diane Ravitch, who served as federal education officials in Republican administrations, complain that present policy means only "top private schools and a few suburban systems will stick with education broadly defined." While rich kids study a wide range of subjects in depth, they write, "their poor peers fill in bubbles on test sheets." There is a "zero sum" problem, Finn and Ravitch say, because "more emphasis on some things ... inevitably mean[s] less attention to others."

Yet public discussion of the law's upcoming reauthorization focuses almost entirely on correcting flaws in math and reading measurement: substituting "growth models" for fixed levels, modifying the 2014 deadline for attaining student proficiency, standardizing state definitions of proficiency, modifying "confidence intervals" in reporting. While these steps may improve the sophistication of math and reading data, none addresses the goal distortion caused by exclusive accountability for basic skills.

Designing accountability tools that require satisfactory performance across a balanced set of outcomes requires a significant federal research-and-development effort, which could build on prior experience. When the National Assessment of Educational Progress was developed in the 1960s, it measured a broad range of cognitive and noncognitive knowledge and skills. NAEP abandoned that breadth when its budget was slashed in the 1970s, however, and never restored it.

To see whether students learned to cooperate, for example, the early NAEP program sent trained observers to sampled schools. In teams of four, 9-year-olds were offered prizes (such as yo-yos) for guessing what object was hidden in a box. Students could ask yes-or-no questions, but all team members had to agree on each question asked. NAEP rated the students on whether they suggested new questions, gave reasons for viewpoints, or otherwise demonstrated cooperative problem-solving skills. It then reported to the nation on the percentage of children capable of cooperative problem-solving.

For teenagers, NAEP assessors provided lists of issues about which young people typically had strong opinions. Students had to collaborate in writing recommendations to resolve them. For 13-year-olds, lists included topics such as whether they should have curfews for getting home, and for 17-year-olds, the age eligibility for voting, drinking, or smoking. NAEP rated students on whether they took clear positions, gave reasons for viewpoints, helped organize internal procedures, and defended another's right to disagree.

Early NAEP understood that teaching civic responsibility involved more than having students memorize historical facts. So in 1969, during the era of the civil rights revolution, the assessment asked teenagers what they felt they should do if they saw black children barred from entering a park. NAEP reported that 82 percent of 13-year-olds and 90 percent of 17-year-olds knew that they should do something constructive, such as tell parents, report it to a civil rights or civil liberties organization, write letters to the newspaper, or take social action such as picketing or leafleting.

The early version of NAEP also assessed 17-year-olds' ability to consider alternative viewpoints, by asking them to state arguments both for and against a heated public issue of the time, such as whether college students should be drafted. It asked 9- and 13-year-olds if something reported in a newspaper might be untrue. It also asked teenagers if they belonged to any nonschool clubs or organizations; interviewers followed up with questions to verify answers' accuracy.

To assess commitment to civil liberties, NAEP asked teenagers if someone should be permitted to say on television that "Russia is better than the United States," that "some races of people are better than others," or that "it is not necessary to believe in God." The assessment reported the discouraging result that only a small minority of the teenagers thought all three statements should be permitted.

The early NAEP program also assessed personal responsibility. Seventeen-year-olds were asked what to do if, when visiting a friend, they noticed her 6-month-old baby was bruised. The correct answer was "suggest that your friend call her baby's doctor." Incorrect choices included "ignore the bruises because they are none of your business." A follow-up prompt said that at a later visit, bruises remain and "you are now suspicious that your friend may have hurt the baby." Students were asked what to do now. The correct choice was "call the local child-health agency and report your suspicions."

Certainly, if school systems were evaluated by such results, not simply by math and reading scores, incentives would shift. National reporting of low scores on the civil liberties questions, for example, could spur demands that schools do a better job on citizenship; then, the incentive to drop cooperative learning in favor of test prep in math and reading would diminish.

Designing a new accountability system will take time and care, because the problems are daunting. Observations of student behavior are not as reliable as standardized tests of basic skills, so we will have to accept that it is better to imperfectly measure a broad set of outcomes than to perfectly measure a narrow set. We will have to resolve contradictory national convictions that schools should teach citizenship and character, but not inquire about students' (and parents') personal opinions. To avoid new distortions, we'll need to make tough decisions about how to weight the measurement of the many goals of education.

The time to start on these difficult tasks is now, but the new administration won't have to begin with a blank slate. Looking back at the early National Assessment of Educational Progress can start us on a better path.

Richard Rothstein (riroth@epi.org) is a research associate of the Economic Policy Institute. This article summarizes an argument from his recent book, co-written with Rebecca Jacobsen and Tamara Wilder, Grading Education: Getting Accountability Right (Teachers College Press).
Published in Ed Week.

Unfortunately legislators and media writers have too often accepted claims of accountability rather than serious study of accountability measures. This year the California Legislature and the Governor provided $10 million for accountability in Teacher Performance Assessment. The TPA/PACT process is neither valid, nor reliable. However, since it goes under the frame of accountability it has been funded even while schools are cutting classes, increasing class sizes and closing schools.
To read more about the problems of TPA/PACT go to http://sites.google.com/site/assessingpact/

The Legislature, the schools, and accountability

Budget stalemate: Day 70
The California legislature since 1994 has imposed accountability on the public schools. They have spent millions on tests. They have insisted that thousands of hours of professional work be dedicated to accountability. Their accountability work has a very weak record of reliability and validity ( See Collateral Damage: How High-Stakes Testing Corrupts America’s Schools. Nichols and Berliner.

Now, the legislature does not pass a budget for over 70 days.
California’s schools are open and the teachers are at work. Over 6 million children have returned to school. Some 477,000 will be entering first grade in over 5,000 schools. Each of these schools have a budget and each of these budgets are in confusion while the state decides what to do about their budget crisis. At least 25% of the schools will not be ready for the students because the school doesn’t know what its budget will be.
Will the school have an ELL teacher or two?
Will there be a reading coach?
Will class size be 24 or 32? Which really means we will have to re-organize each of the classes and the teachers.
What will happen to the new programs established last year under the Quality Education Act?
Shall the district hire a new teacher or only a 30 day substitute ?
Do we have the money for an ELL specialist or will the money be for an Algebra teacher? And when we finally hear if we have the money, will the well qualified Algebra teacher have moved to another state where this annual disruption of their lives does not occur? Really, would you wait 2-3 months each year to see if you had a job?
And, even in mainstreamed classes, will there be two English Learners or eight?
These are but the start of the many decisions that need to be made. Rather than beginning school in late August, far too many classrooms will have to wait until October while the budget gets decided and allocations are made.

This is a state that ranks 47th. in math and about 48th in reading. A state budget impasse each year creates 4-6 weeks of school disruption, confusion, and disorganization.
And then the legislature calls for school accountability? Where is their accountability?
Duane Campbell

Accountability and Schools

Accountability
The school systems in most of our cities in California are currently at a crisis point. Schools can continue as they are. A segment of society will be well-educated, another segment will continue to fail. The economic crisis for working people and people of color will continue to grow (Mishel, Bernstein, Allegretto, 2007). Alternatively, schools can be transformed into places where education is a rich, compelling, and affirming process that prepares all young people to make thoughtful contributions to their community in economic and civic terms.
Reforming the schools requires money. And, the Legislature is currently deadlocked. Their issue is how much to cut this year .
Frederick Douglass spoke to this issue in 1849 when he wrote the following:
The whole history of the progress of human liberty shows that all concessions yet made to her august claims have been born of earnest struggle. The conflict has been exciting, agitating, all absorbing, and for the time being putting all other tumults to silence. It must do this or it does nothing. If there is no struggle there is no progress. Those who profess to favor freedom, and yet depreciate agitation, are men who want crops without plowing up the ground. They want rain without thunder and lightning. They want the ocean without the awful roar of its many waters. This struggle may be a moral one; or it may be a physically one; or it may be both moral and physical; but it must be a struggle. Power concedes nothing without demand. It never did and it never will. (Douglass, 1849/1991, p. vii)

WE need to invest in California schools, provide equal educational opportunities in these schools, and recruit a well prepared teaching force that begins to reflect the student populations in these schools. We must insist on equal opportunity to learn, no compromise. When we do these things, we will begin to protect the freedom to learn for our children and our grandchildren, and to build a more just and democratic society.
The struggle for education improvement and education equality is a struggle for or against democratic participation. The struggle for multicultural education, based in democratic theory, is an important part of the general struggle against race, class, and gender oppression.
Schools serving urban and impoverished populations need fundamental change. These schools do not open the doors to economic opportunity. They usually do not promote equality. Instead, they recycle inequality. The high school drop out rates alone demonstrate that urban schools prepare less than 50 percent of their students for entrance into the economy and society. A democratic agenda for school reform includes insisting on fair taxation and adequate funding for all children. Political leaders in California have not yet decided to address the real issues of school reform. We cannot build a safe, just, and prosperous society while we leave so many young people behind.
The conservative/ media emphasis on accountability is a distortion. We know which schools need improvement, and we know how to improve them.. Teachers and parents together face a political choice. Shall we continue to call for high standards without providing the necessary resources for all schools to have a reasonable chance to attain such standards? Shall we continue to punish schools and their staffs for low test scores, even when we know that the tests are poor instruments for measuring learning and that their construction guarantees the failure of many students? Is increased competition and privatization the answer for schools when it has not been the answer in other sectors of our society, particularly for low income and diverse people?
The problem is to provide the resources, including well prepared teachers with adequate support, needed to make the current schools successful. We face a choice between providing high-quality schools only for the middle and upper classes, and underfunded, understaffed schools for the poor. Or, we can also choose to work together to improve schools that are presently failing.

Lessons from Finland: The way to education Excellence

Published on Wednesday, February 27, 2008 by The Providence Journal (Rhode Island)

Lessons From Finland: The Way to Education Excellence

by Walt Gardner

When Finland’s 15-year-olds recently placed No.1 in math and science on the recent Program for International Student Assessment, the news of the coup was received in Helsinki with characteristic reserve. For the Finns, whose schools are considered the best in the world, the scores stood as a redundant confirmation of the success of their policies.

But in the U.S., the frustration was palpable. Despite persistent attempts to bring equity to the wildly uneven quality of our schools, reformers have not been able to produce the intended results. That’s why they’ve begun to look even more closely in this presidential election year at Finland for lessons that can be applied here. What they will find in the end serves as a cautionary tale for strategies that we proudly consider cutting edge.

At the heart of Finland’s stellar reputation is a philosophy completely alien to America. The country of 5.3 million in an area twice the size of Missouri considers education an end in itself - not a means to an end. It’s a deeply rooted value that is reflected in the Ministry of Education and in all 432 municipalities. In sharp contrast, Americans view education as a stepping stone to better-paying jobs or to impress others. The distinction explains why we are obsessed with marquee names, and how we structure, operate and fund schools.

The headlines notwithstanding, misconceptions about Finland’s renown as an educational icon abound. The Finns spend a meager (compared to the U.S.) $5,000 a year per student, operate no gifted programs, have average class sizes close to 30, and don’t begin schooling children until they are 7. Moreover, Finland is not the homogeneous nation of lore. While still not as diverse as the U.S., the number of immigrant students in Helsinki’s comprehensive schools is exploding, with their numbers expected to constitute 23.3 percent of the city’s schools by 2025. At present, about 11 percent are immigrants, compared with just 6 percent in 2002. According to the City of Helsinki Urban Facts, by 2015 there will be schools with more than half of the student body from abroad.

Not surprisingly, in a land where literacy and numeracy are considered virtues, teachers are revered. Teenagers ranked teaching at the top of their list of favorite professions in a recent survey. Far more graduates of upper schools in Finland apply for admission to teacher-training institutes than are accepted. The overwhelming majority of those who eventually enter the classroom as a teacher make it a lifelong career, even though they are paid no more than their counterparts in other European countries.

One of the major reasons for the job satisfaction that Finnish teachers report is the great freedom they enjoy in their instructional practices. As long as they adhere to the core national curriculum, teachers are granted latitude unheard of in the U.S. The scripted lesson plans that teachers here are increasingly being expected to follow would be rejected out of hand as an insult by teachers in Finland and by their powerful union, which has a growing membership of some 117,500 members.

If none of these facts are enough to raise doubts about the policies the U.S. has in place or on the drawing board, Finland’s testing practices should raise a final red flag. The Finns do not administer national standardized tests during the nine years of basic education. Instead, the National Board of Education assesses learning on the basis of a sample representing about 10 percent of a stipulated age group. Individual school results are strictly confidential, and schools are neither ranked nor compared. The data collected are available only to the schools in question and to the National Board of Education, which use them to help improve instruction. The naming and shaming that No Child Left Behind relies on in its obsession with quantification would be unthinkable.

What ultimately emerges from studying Finland is the realization that the reform movement in America is based on a business model fundamentally at odds with the education model used by a country with the world’s finest schools. While it’s always risky to attempt to apply findings from one country to another, particularly when the two are so different, it’s a mistake to turn our backs on Finland’s approach.

Walter Gardner taught English for 28 years in the Los Angeles Unified School District and was a lecturer in the University of California at Los Angeles Graduate School of Education.

© 2008 The Providence Journal Co.

School Accountability

Great podcasts on a complex topic.
http://www.accountabilityfrankenstein.com/

Accountability tests, and NCLB, fundamentally flawed

Published Online: November 13, 2007
Published in Print: November 14, 2007
Commentary
Accountability Tests’ Instructional Insensitivity: The Time Bomb Ticketh
By W. James Popham


Would you ever want your temperature to be taken with a thermometer that was unaffected by heat? Of course not; that would be dumb. Or would you ever want to weigh yourself with bathroom scales that weren’t influenced by the weight of the person using them? Of course not again; that would be equally dumb. But today’s educators are allowing their instructional success to be judged by students’ scores on accountability tests that are essentially incapable of distinguishing between effective and ineffective instruction. Talk about dumb.
What’s worse is that we are now racing toward the 2014 deadline of the federal No Child Left Behind Act, the point at which all students are supposed to have attained test-based “proficiency.” But the 2002-2014 schedules that most states devised when establishing their goals for annual required numbers of proficient students will soon demand some staggering increases in how many students must earn proficient scores on state NCLB tests each year. These balloon-payment improvement schedules were, in most instances, adopted as a way of deferring the pain stemming from having too many state schools and districts flop in reaching their goals for adequate yearly progress, or AYP.
Such cunningly crafted, soft-to-start improvement schedules will lead in a very few years to altogether unrealistic requirements for improved test scores. Without such improvements, huge numbers of U.S. schools and districts will be seen as AYP failures. If the American public is skeptical now about the quality of public schools, how do you think citizens will react when, in the next several years, test-based AYP failure becomes the rule rather than the exception? Can you hear the ticking of this nontrivial time bomb?
How could American educators let themselves get into a situation in which the tests being used to evaluate their instruction are unable to distinguish between effective and ineffective teaching? The answer, though simple, is nonetheless disquieting. Most American educators simply don’t know that their state’s NCLB tests are instructionally insensitive. Educators, and the public in general, assume that because such tests are “achievement tests,” they accurately measure how much students have learned in schools. That’s just not true.
Two types of accountability tests are currently being used to satisfy the No Child Left Behind law’s assessment requirements. About half of the nation’s NCLB tests consist of traditional, off-the-shelf, standardized achievement tests, usually supplemented by a sprinkling of new items, so that the slightly expanded tests will supposedly be better aligned with a particular state’s content standards. Other NCLB tests are made-from-scratch, customized standards-based accountability tests, built specifically for a given state. Let’s see, briefly, why both these types of tests are instructionally insensitive.
Traditional standardized achievement tests, such as the Stanford Achievement Test-10th Edition, are intended to provide comparative information about test-takers. So the performance of a student who scores at, for instance, the 96th percentile can be contrasted with that of students who score at lower percentiles. To accomplish this comparative-measurement mission, these tests must produce a substantial degree of “score spread,” so there are ample numbers of high scores, middle scores, and low scores. Most items on such tests are of middle-difficulty levels because such items, statistically, maximize score spread.
Over the years, however, many of these middle-difficulty items turn out to be closely linked to students’ socioeconomic status. More-affluent kids tend to answer these socioeconomically linked items correctly, while less-affluent kids tend to miss them. This occurs because socioeconomic status, or SES, is a nicely distributed variable, and one that doesn’t change rapidly; SES-linked items help generate the score spread required by traditional standardized achievement tests. When such tests are used as accountability assessments, however, they tend to measure the socioeconomic composition of a school’s student body, rather than the effectiveness with which those students have been taught. The more SES-linked items there are on a traditional standardized achievement test, the more instructionally insensitive that test is bound to be.
How could American educators let themselves get into a situation in which the tests being used to evaluate their instruction are unable to distinguish between effective and ineffective teaching?
The other type of NCLB accountability test used in the United States is usually described as a “standards-based test,” because such tests are deliberately built to assess students’ mastery of a given state’s content standards, that is, its curricular aims. In all but a few states, though, the number of content standards to be assessed is so large that there is no way to accurately assess—via an annual accountability test—students’ mastery of this immense array of skills and knowledge. Instead, each year’s accountability test must sample from the profusion of the state’s curricular aims. Such a sampling-based approach to annual assessment means that teachers end up guessing about which curricular aims will be assessed each year. And, given the huge numbers of potentially assessable curricular targets, most teachers guess wrong.
After a few years of incorrect guessing, many teachers simply give up on trying to mesh their teaching with what’s to be assessed on each year’s accountability tests. And when this happens, it turns out that the major determinant of how well a school’s students perform on accountability tests is the very same factor that governed students’ performances on traditional standardized achievement tests: socioeconomic status. Thus, even on customized standards-based tests, a school’s scores are influenced less by what students are taught than by what the students brought to that school. Most standards-based accountability tests are every bit as instructionally insensitive as traditional standardized achievement tests.
The instructional insensitivity of accountability tests does not represent an insuperable problem, however. Remember when, several decades ago, we began to recognize that there was considerable test bias in our high-stakes educational assessments? Once this difficulty had been identified, it was attacked with both empirical and judgmental bias-detection procedures. As a consequence, today’s educational tests are markedly less biased than were their predecessors. Once the test-bias problem had been identified, we set out to fix it—and in less than a decade, we did.
That’s precisely what we need to do now. Using a mildly technical definition, a test’s instructional sensitivity represents the degree to which students’ performances on that test accurately reflect the quality of instruction specifically provided to promote students’ mastery of what is being assessed. We need to discover how to build accountability tests that will be instructionally sensitive and, therefore, can provide valid inferences about effective and ineffective instruction. It may take several years to get the required procedures in place, but we need to get started right now.
In the short term, though, we must make citizens, and especially educational policymakers, understand that almost all of today’s accountability tests yield an invalid picture of how well students are being taught. Accountability systems based on the use of such instructionally insensitive tests are flat-out senseless. We need accountability tests capable of distinguishing between students who have been properly taught and those who have not. Until such tests are at hand, we might as well relabel our accountability systems as what they are—elaborate and costly socioeconomic-status identifiers.
W. James Popham is a professor emeritus in the graduate school of education and information studies of the University of California, Los Angeles. He now lives in Wilsonville, Ore.
Vol. 27, Issue 12, Pages 30-31

U.S. takes lousy care of our children

The nation, not schools, takes lousy care of our children


From the beginning of the educational “accountability” movement in the mid-1990s, the demand that schools “close the achievement gap” has set educators’ teeth on edge. The “gap” refers to the wide discrepancy between the test scores of middle-class white children and those who are low-income and non-white.

Educators know first hand that less-privileged students — an ever-growing number, seemingly — enter school at a significant disadvantage compared to their more privileged peers. That gap opened up long before the school bell tolled. Even in schools where the low-income children have made strong gains, the gap persists. Schools have little impact on poverty or the lack of good health care, decent jobs for parents, affordable housing and other social factors that contribute to a child’s readiness to learn.

Educators who voiced these concerns were often chastised as racist, class-biased or indulging in the “soft bigotry of low expectations.”

And it’s true that the schools that educate most urban and poor children have become enmeshed in political power struggles unrelated to helping students. They can’t in good conscience point to their work and say: See? We’ve given these students the very best, and we’ve made gains, but the gap will continue to persist until conditions improve in their home lives and neighborhoods.

In his most recent book, Richard Rothstein, former education columnist for The New York Times, catalogs an array of social conditions that contribute to the achievement gap in exhaustive and fascinating detail. Class and Schools — Using Social, Economic, and Educational Reform to Close the Black-White Achievement Gap does not let the schools off the hook. But it does argue that with all the negative social forces at work, we are kidding ourselves if we think that schools are going to do this job by themselves.

Here are three of Rothstein’s examples illustrating the profoundly different backgrounds of high-, middle- and low-income children:

Researchers Betty Hart and Todd Risley “found that, on average, professional parents spoke more than 2,000 words per hour to their children, working class parents spoke about 1,300, and welfare mothers spoke about 600. So by age 3, the children of professionals had vocabularies that were nearly 50 percent greater than those of working-class children and twice as large as those of welfare children.”

In a school’s regular day and year, teachers cannot contribute enough to low-income children’s education that would allow the students to catch up to their middle-class peers. Middle-class kids are also learning during that same day and year, as well as attending after-school enrichment activities.

Second, consider the cultural difference between professional and working-class jobs. Parents who are working professionals have authority and responsibility, so they are used to exploring alternatives and negotiating compromises. At home they talk their kids through solving problems and give reasons for their decisions or actions. Their children learn to negotiate what they want and feel entitled to do so.

“But parents whose jobs entail following orders or doing routine tasks show less sense of efficacy. They are less likely to encourage their children to negotiate over clothing or food and more likely to instruct them by giving directions without extended discussion. Following orders, after all, is how they themselves behave at work.”

Many people, including me, believe that learning good negotiation skills more positively affects later academic, career and personal success than the learning that gets good test scores.

The best schools explicitly teach manners, negotiation skills and how to handle feelings in acceptable ways — called a social-emotional curriculum. This ensures that all children learn these important skills, but it still can not make up for the practice the middle-class child has at home, reasoning with elders and being encouraged to solve problems.

Lastly, affordable housing has become increasingly scarce, exacerbating the extent to which low-income people have to move. Changing residences often affects a family’s ability to function well and changing schools disrupts the continuity of a child’s education.

This relatively minor example illustrates the extent to which public policy makes a bad situation worse. A child recently uprooted from home and school often cannot pay attention to lessons in the new school. So the child falls further behind, exacerbating the achievement gap. Cities could pay for transportation to keep the child in his old school, with friends and teachers, if the residential move is local. The cost would be modest, but cash-strapped cities face an endless menu of hard choices.

Rothstein says, “The connection between social and economic disadvantage and an academic achievement gap has long been well known.... Calling attention to this link is not to make excuses for poor school performance. It is only to be honest about the social support schools require if they are to fulfill the public’s expectations that the achievement gap will disappear.”

Rothstein’s book unpacks for us the specifics of such supports as access to health care, housing, after-school and summer enrichment programs and preschool.

But Rothstein’s book begs the question as to whether the American public really wants to close this achievement gap. With calm rhetoric and rich data, he lays out the problems, solutions and choices in front of us.

Schools may exacerbate the achievement gap, but they didn’t create it in the first place. As a nation, we are shockingly content to tolerate widespread poverty among our fellow citizens. We are the richest country in the world, but one in five children is brought up in a family living at the federal poverty line. The quintile above them is not much better off.

In short, we take lousy care of our kids, but find it convenient to blame the schools.

Julia Steiny is a former member of the Providence School Board; she now consults and writes for a number of education, government and private enterprises. She welcomes your questions and comments on education. She can be reached by e-mail at juliasteiny@cox.net or c/o EdWatch, Education and Employment, Providence Journal, 75 Fountain St., Providence, R.I. 02902.

NCLB: How Karl Rove reframed the debate on school reform

'Framing' the Debate over NCLB
The No Child Left Behind Act – though a policy failure in many ways – has nevertheless been a rhetorical triumph. For NCLB proponents, the emphasis on overcoming racial "achievement gaps" has served as a moral high horse, enabling them to gallop roughshod over critics while decrying "the soft bigotry of low expectations." Accusations of "making excuses for failing schools" and "believing that minority children can't learn" have proved to be effective weapons in the debate.

As a result, most organizations representing educators find themselves on the defensive whenever they raise concerns about NCLB's impact. Who wants to be labeled as "against accountability," much less "bigoted" against minority children? So those favoring fundamental changes in NCLB often find themselves at a tactical disadvantage in enounters with Stay the Course forces.

Credit for all this belongs largely to Karl Rove, White House adviser and Republican apparatchik extraordinaire, who has been deservedly dubbed "Bush's Brain." It was Rove who made NCLB, both the slogan and the concept, a centerpiece of the 2000 presidential campaign. He saw it as a way to position George W. Bush as a "compassionate conservative," to soften his party's hard-hearted image, and to outflank Democrats on a key domestic issue.

In effect, Rove "reframed" the issue of school reform to gain a political advantage for Republicans. Whereas conservatives had traditionally opposed a strong federal role in education and had tended to stress academic excellence over equity, the Bush Administration took up the cause of "disadvantaged" students in order to advance other Republican objectives, such as the privatization of public schools. Using the rhetoric of civil rights and anti-poverty, it enlisted many Democratic followers as well. Hence the overwhelming bipartisan support for NCLB in 2001.

New Terms, New Politics
Terminology is central to framing, as the linguist George Lakoff explains:

"Frames are mental structures that shape the way we see the world. When you hear a word, its frame is activated in your brain. Reframing is changing the way the public sees the world… New language is required for new frames. Thinking differently requires speaking differently."

NCLB has relied on several terms to reframe the debate over school reform, including accountability, adequate yearly progress, scientifically based research, and perhaps most important, achievement gap.

Disparities in test scores between socioeconomic groups are nothing new, of course. (Nor are they restricted to American schools.) But when did "the achievement gap" become a focus of public discourse? Quite recently, it turns out. An archive search of the New York TImes found that the term appeared rarely in the 1980s, sporadically through most of the 1990s, and then frequently beginning in 1998 – often in reference to the Bush presidential campaign and later, of course, in connection with NCLB.

Meanwhile, a long-established term – equal educational opportunity – was going out of common usage. From 2001 to 2005, it appeared in just 10 articles, as compared with 173 articles mentioning achievement gaps. The shift in terminology, albeit subtle, signaled a reframing of how the American public thinks about school reform.



Whereas "equal educational opportunity" had framed the issue in terms of educational inputs – resources, curriculum, facilities, materials, teacher training, best practices – "achievement gap" now highlights only educational outputs, as measured by standardized tests.

Responsibility for inputs is broadly shared among policymakers at all levels. Outputs are seen as the job of schools, which will be "held accountable" for achievement gaps. Hence NCLB's constricted version of accountability. The law essentially removes political leaders from the picture, ignoring their responsibility to provide adequate and equitable resources, and shining the spotlight on educators alone.

Another sign of reframing: The phrase “failing schools” appeared in 306 New York Times articles between 2001 and 2005, as compared to just 69 between 1991 and 1995.

James Crawford

Unstated Assumptions about NCLB-Style Accountability
'Accountability' at What Price?
What Kind of Accountability?
A Better Way To Hold Schools Accountable for ELLs

Copyright © 2007 by the Institute for Language and Education Policy. All rights reserved. Permission is hereby granted to reprint or repost material from this site unless copyrighted by third parties, for educational, advocacy, and other noncommercial purposes. All other permission requests should be directed to the Institute at bilingualed@starpower.net.

Schools and democracy : Deborah Meirer

Posted by Deborah Meier at 4/17/07 6:00 AM
Tags: Education Policy

Some people wake up with great ideas. But I’m a night person. Right before I fall asleep I think I’ve finally found just the right way to say what it is I’m thinking. Often when I wake up I've either forgotten it or it seems banal.

But here are two ideas that keep reoccurring, and it is morning now so I’m going to try to capture them.

Great Idea 1:

The whole point of public education (vs job training or even some forms of private education) is to prepare a public for its responsibilities which, in a nutshell come down to exercising careful, thoughtful and reasonable judgment in the face of complex evidence. In the two tasks that confront 18 year olds--voting and serving on juries--these are the presumptions that lie behind the privilege. Our best judgment is what in the end we bring to the table. Note there are neither admissions tests, nor licensure requirements for either voting or serving on a jury. It's the unspoken and awesomely heavy presumption and also the most irrational facing governance by democratic principles and practices. It makes no sense, except (as Churchill said) it's better than any of the alternatives. But for every problem confronting this absurd idea--that everyone has a "right" to such power--there is a solution. Better education.

Not just formal K-12 schooling, surely not just or even college—which comes too late for many voters and jurors and is not open to everyone. But, as the slogans say, our aim is “lifelong” learning, on-going adult education. Newspapers are one of these educating forces, as are all the new technologies. Public access to books, libraries full of resources for getting at "the truth", public spaces for communicating one’s ideas, and for demonstrating on behalf of them, etc, etc. I’m enamored even of the idea of subsidizing adults for going back for a liberal arts education later on, when they are more likely to appreciate its usefulness. But the one and only institution we set aside for this and only this purpose--with no obligation to make a buck in return--is our K-12 system of schooling.

I challenge any of us to spend a day with a kid in an average school and try to connect the dots between what is being learned there--formally ad informally--and what a citizen of a democracy requires (in contrast to citizens perhaps of countries that don't even pretend to be democratic). The world is full of virtues. And economic necessities. But what are the explicitly democratic predispositions, skills, habits of mind and heart that we are not born with, but could learn in a setting devoted to such a purpose?

Great Idea 2:

Then, one night it occurred to me, that for all my ranting and raving against the term accountability, in fact the idea of being accountable lies at the heart of democracy. Democracy is a form of accountability--a concept intended to hold the powerful’s feet to the fire. Naturally as our schools have moved further and further away from being attached to their publics, it has become more and more important for us to invent other non-democratic, bureaucratic, “mandarin” forms of accountability. When there were 200,000 school boards serving a population less than half the size of today’s, a lot of people knew who was making judgments about their schools. Today with as few as 10,000 school boards, and with some of them having almost no realistic power over anything but floating bonds, well.... No wonder! There ought to be a half million school boards or more, if--big if--we really believe in democracy as our most special and effective form of accountability.

Given that I'm not a fan of many of the decisions reached by democratic decision making bodies--including many school boards as well as state legislatures and Presidents--this is a leap of faith. I make it because I still agree with Churchill about the alternative to holding on to this often counter-intuitive and even counter-reasonable faith. Neither various forms of benign dictatorship or market-place utopias seem more reasonable . Although if I got to choose the dictator it does some nights appear to be the solution. But by morning I have to face the fact that it’s unlikely to be someone of my choice; and if it were I’d probably be in the opposition the day after—coercion just has its limits when it comes to the important stuff—the stuff inside our hearts and minds.

These two ideas have become more than nighttime fantasies, but daytime ones too. I long for a more robust discussion. We confront the increasing daily power of BOTH my dystopias-- increased centralization of public schools in the hands of the few, and increased “selling off” of our schools to private interest groups. And yet we confront a very thin response to both.

What, Forum readers, would we have to do to make these issues part of the conversation about K-12 schooling among our friends, parents of our children’s friends, colleagues, fellow citizens?

copyright © 2007 The Forum for Education and Democracy