Standards and Assessments

Research and Best Practices

A Policymaker's Guide to Standards-Led Assessment Executive Summary by Robert L. Linn and Joan L. Herman February 1997 National Center for Research on Evaluation, Standards and Student Assessment and Education Commission of the States (Posted with permission from the Education Commission of the States) States across the country are setting tough new standards, defining what students should know and be able to do. To help students meet these standards - and to measure their progress in doing so - many states are also designing and implementing new assessment systems. Assessments play a pivotal role in standards - led reform, by: Communicating the goals that school systems, schools, teachers and students are expected to achieve; Providing targets for teaching and learning; and Shaping the performance of educators and students. Coupled with appropriate incentives and/or sanctions - external or self-directed - assessments can motivate students to learn better, teachers to teach better, and schools to be more educationally effective.

What's Different about Standards - Led Assessments?

Unlike more traditional assessments, standards - led assessments are closely linked to curriculum, producing a tight coupling between what is taught and what is tested. Unlike norm-referenced tests, which compare each student's performance to that of others, standards -led assessments incorporate pre-established performance goals. And unlike multiple - choice exams, many standards -led assessments require students to demonstrate a broad range of problem-solving skills- the very skills students will need for future success. These "authentic" or "performance" assessments typically engage students in real-world problems, rather than artificial exercises. Such assessments not only measure students' ability to master complex tasks but also model those tasks for teachers, providing examples for use in the classroom. Performance assessments that require extended responses must be scored by expert judges, using clearly specified scoring guides. The development and application of such scoring guides presents teachers with a rare opportunity to discuss new standards and performance expectations. Examining actual responses helps teachers understand the strengths and weaknesses of their students' learning and plan appropriate instructional activities. What makes for a sound assessment? Two major criteria are typically cited: validity, the degree to which particular uses and interpretations of assessments results are justified; and reliability; the degree to which scores are free of measurement error. For standards-led assessment, another key is alignment - the degree to which the assessment adequately reflects the standards on which it is supposed to be based. An assessment that is mismatched with a given set of standards may undermine learning, by focusing attention on less important skills or knowledge at the expense of others and more important ones.

Challenges for Standards - Led Assessment Systems

Building state and local consensus

If public opinion polls are any indication, the concept that students should be held to high academic standards enjoys broad support. Experience shows, however, that such support can be fragile. The diversity of opinion on what students should learn and schools should teach makes it imperative to involve the public in the development of standards and assessments. Building a broad consensus requires not just a series of public hearings and opportunities for input and review but a comprehensive process that fully involves the public, ensuring that its concerns are understood and addressed.

Providing strong standards

Achieving consensus on standards that are broad and vague is no challenge - who would disagree that all students must be able to "communicate effectively"? But when standards are stated in such general terms, they offer little help for the students who must meet them or for the teachers and schools attempting to assess student progress. Available evidence suggests that many states' current standards are not strong enough to support rigorous content-based assessment.

Aligning standards with assessment and instruction

Many states and localities develop standards and assessments at the same time, rather than following the more logical sequence: standards first, assessments second. Indeed, some states patch together assessment systems using whatever assessments are available, sacrificing the "custom fit" they would gain by developing assessments from scratch. Systems that rely exclusively on multiple-choice exams cannot show how well students are performing on the full range of shills and understandings covered by standards. Ultimately, classroom curriculum and instruction should be aligned with standards and assessments. Yet this alignment depends in turn on teachers' ability to understand - and obtain the resources and expertise to help their students meet - the expectations embodied by new assessments. Fairness demands that students not be held accountable for goals they have had an inadequate opportunity to reach.

Assuring accurate measures

Performance assessments, which ask students to create a response rather than choose one from a list, generally provide a better gauge of complex thinking skills. But scoring such assessments requires more time, usually more money, and consensus among judges on the quality of the response. To furnish a stable estimate of student capability, most assessments now being developed incorporate a broad range of tasks, reflecting the full scope of the standards. When measuring the performance or progress of a school or district rather than an individual student, assessments can also assign different tasks to different samples of students (a practice known as matrix sampling).

Defining progress

The progress of schools, districts and states is typically defined by the performance of successive cohorts of students: Are more fourth-graders, for example, demonstrating proficiency in math standards this year than last? Federal law requires that states define "adequate yearly progress" in terms of students' performance on the states' standards -led assessment, determine whether their schools are making such progress, and target an "appropriate" date by which all Title I students will perform at either the proficient or advanced level. States must then set an annual rate of improvement that is both "substantial" and sufficient" to achieve that goal.

Setting the stakes

What schools do with assessment results - whether simply reporting them, at one end of the spectrum, or making graduation contingent on them, at the other - can have profound effects on students. Assessments also can be used to hold educators and schools accountable for students' performance. Districts may use the results as the basis of explicit rewards (e.g., cash grants) or sanctions (reassignment of dismissal of staff, administrative take-overs).

Including all students

Standards are designed to raise expectations for all students. Including limited English speaking students and students with disabilities in an assessment may require a variety of different accommodation strategies, from the allotment of extra time to the provision of oral assessments or translation to other languages. Students with learning disabilities - who account for the largest group historically excluded from assessments - may be able to complete assessments, in part or in full, without special accommodations. (Those with severe cognitive disabilities may require a separate system of assessments.)

Estimating costs

While the costs of assessments vary widely, those requiring extended student responses - to be judged by teachers or other subject - matter experts - costs substantially more than multiple - choice tests. Administering a machine-scorable test may cost between $5 and $8 per student; assessments that require a mix of short answers and extended written responses can easily cost two or three times as much. Estimates for more elaborate performance assessments range from $30 to $70 per student.

Addressing legal challenges

Assessments are most likely to face legal challenges when high stakes - whether to graduate a student, whether to endorse a diploma - are attached to the results. Challenges also can be expected when assessments produce an adverse impact on historically disadvantaged groups: substantially higher failure rates for African American or Hispanic students, for example. Such evidence does not, by itself, establish the unfairness of an assessment or an intent to discriminate. But the identification of an adverse impact can - and often does - trigger a legal challenge. Among the other most likely triggers: The "use of processes perceived to be unfair, arbitrary, or capricious" The "suggestion that specific attitudes or values are being assessed" The "failure to provide all accommodations requested by the disabled" The assessment of "knowledge or skills that examinees have not had the opportunity to learn." The likelihood of legal challenges argues against attaching high stakes to assessment results too soon. Designing a reliable standards-led assessment system is a complex and time-consuming process. It will take just as much time for teachers, schools and students to understand the expectations such a system raises--and to meet them.

Building local capacity

Research shows that most teachers treat performance assessments seriously and incorporate the underlying goals in their instruction. At the same time, though, many principals and teachers report serious concerns about the demands new assessments place on themselves and their schools. In particular, they report, teachers need time to become familiar with new standards, assessments and administration requirements; to understand how new forms of assessments are developed and scored; to apply criteria for assessing students' work; and to acquire enough information and pedagogical knowledge to change their practices. Providing appropriate resources and sufficient opportunities for professional development is equally important.

Distinguishing assessments

An assessment that attempts to perform too many functions -- student diagnosis, curriculum planning, program evaluation, instructional improvement, accountability, certification, public communication -- will inevitably do none well. It is important, therefore, to distinguish appropriate roles for different assessments, at the district, school and classroom level. A cohesive system ensures that teachers and students understand what is important to learn and how well they are doing. Teachers routinely use a wide variety of formal and informal assessments to gauge student progress, assign grades, motivate attention, provide feedback and adapt instruction to student needs. Similarly, students regularly engage in self-assessment, as they study and attempt to solve problems and monitor their own progress. Together, all of these assessments provide teachers and students with the detailed understanding and continual feedback they need to guide effective, ongoing learning. It is essential that these assessments reflect state standards. Copies of this guide are available for $10 plus postage and handling from the ECS Distribution Center, 707 17th Street, Suite 2700, Denver, Colorado 80202-3427; T: 303-299-3692. Ask for No. SI-97-3.