At times it seems that everything there is to say about testing and assessment in adult literacy has been said. By now, practitioners and administrators alike can cite the shortcomings of standardized tests using multiple-choice formats and are familiar with the inadequacy of grade levels as indicators of what adult learners know and are able to do. Yet, multiple-choice, pencil and paper tests continue to be used not only as placement instruments but as measures of learner gains and evidence of program success. Given current reporting requirements, their use is likely to increase, at least in the near future. From the perspective of programs, there seem few viable alternatives that would meet the information needs of funders interested in reliable data that indicate how a program is doing overall. Portfolio approaches, for example-considered the last great hope a few years back-have not quite matured to the level where they might be used as a means to report and aggregate learner gains by group (although they are invaluable as evidence of individual learner progress), largely because the field has not invested in the development of benchmarks and rubrics. Local approaches have remained just that, local approaches, primarily for two reasons: (1) there has not been enough field testing to establish the reliability of these measures and (2) there have not been sufficient efforts to implement alternative assessments across programs. It is easy to see how even programs that have been enthusiastic about developing an assessment system that captures worthwhile outcomes are becoming distressed about the prospects of an alternative system being able to rival the standardized tests currently in fashion.

All Is Not Lost
Yet, the picture is not as dim and grim as it might first appear. Indeed, it may be premature to give in to cynicism ("it's all a sham and no one really cares"), paranoia ("next year, all funding will be tied to the results of standardized tests"), and paralysis ("in the end, no one will care about alternative assessment, so let's just sit and wait to see what comes down the pike"). Since a Pollyanna attitude does not appear to be justified either, given recent legislation, perhaps it is time to take an existen-tialist perspective where we commit ourselves to forge ahead although (and even because) life in adult literacy does not always make sense, but what else are we going to do to stay sane? Let's ask then if there is anything positive happening in assessment, and how we can help shape new directions on the national or state level, while continuing to strive for sane assessments within and across local programs.

The Federal Outcome Reporting System
You may have heard that the US Department of Education has mandated a uniform outcome- based reporting system that requires that all states send data for all programs funded under Adult Basic Education (ABE) to the Department of Educa-tion in Washington. Assessments for capturing outcomes must be "valid and reliable." States (and the programs they fund) will be asked to report "learner gains" in reading, writing, speaking, and listening (and possibly additional skills related to workforce development) and show that learners are advanc-ing across levels.

To understand the thinking behind the initiative, it is important to keep in mind that the primary focus is neither curriculum reform, nor program improvement, but rather an accountability measure to bring adult literacy in line with the requirements of GPRA -- the Government Performance and Results Act. GPRA requires that all federal agencies show that they, as well as the agencies and programs they fund, are achieving results or else risk loss of funding. Funders will want to know how a program is doing overall (i.e., whether it is positively affecting literacy skills), and they expect to see numbers in aggregate form. While in many ways, documenting the kinds of outcomes required by the new reporting system is "doable," two dangers loom as programs try to show gains and as results are increasingly tied to funding. There is a risk that programs will be (1) tempted to manipulate assessment results in their favor and (2) succumb to a practice known as "creaming."

The Dangers of Creaming
It is an unfortunate fact of adult literacy that programs that help those "hardest to serve" (e.g., learners who are both new to English and new to literacy) have the greatest difficulties showing gains, not only because their learners need a great deal of time until progress is evident, but because the kind of progress they are making is not easily captured by standardized multiple- choice, paper and pencil tests. In addition, programs who serve these students don't have the resources to set up testing alternatives appropri-ate for a low literacy population. There is a danger, then, that programs not fully committed to serving learners who need both special support and extended time will decide to focus their efforts instead on those students who most easily advance, since the incremental progress of "slower" students only makes the program "look bad." Thinking along those lines, ESOL programs, for example, might decide to focus the curriculum on immigrants with higher levels of education, rather than serving ESOL literacy students. This process of focusing on participants who are easy to serve is known as "creaming" and has long been decried as an unintended outcome of programs that have signed performance- based contracts, where funds are linked to learner outcomes and program impacts, such as job placement.

So far, not many public debates have taken place around this issue in adult literacy on the state level, but concerns are sure to arise as programs realize the difficulties they face in reporting progress across levels in the time periods envisioned by the reporting system.

So Why Not Ask for an Exemption?
Two solutions to the problem of creaming seem possible: (1) set aside monies so programs can develop an alternative assessment for lower level students, or (2) ask that learners who have difficulty negotiating paper and pencil tests be exempted from testing. In my view, exemptions, as attractive as they may seem, are not the best solution in the long run, since we may end up marginalizing both this group and programs that serve them. I believe that, rather than asking for exemptions for students who cannot cope with the standard-ized tests approved by a state, we are better off advocating for the development of an alternative assessment framework for this group. Once such an assessment is developed for one group, it is easier to acquire the resources to extend it to other levels and other populations.

Alternative Testing for Low Literacy Students
What might an assessment that measures the incremental changes occurring at the initial levels of language and literacy development look like? It is entirely possible to design a framework allowing learners to demonstrate what they can say and understand in English despite limited proficiency (in fact, the oral interview component of the BEST test does just that). It is also possible to design a "can-do" literacy assessment based on the kinds of texts and tasks that those new to literacy deal with every day.

If a program wants to create an assessment that works double duty (as a basis for program im-provement and for accountability), a further step is necessary: the development of scales, rubrics, and benchmarks indicating the expecta-tions for any given level, and to what degree learners are close to acquiring the kinds of knowledge, skills, and strategies that are a core part of our curriculum.

As funding for adult literacy is increasing, the old refrain of "there is no money to do this," no longer holds true. There are alternatives to multiple-choice tests, and we must advocate for their development and use if we are serious about documenting progress for all learners, including those who still struggle with basic literacy.

Top of Page

Building an Assessment Framework That Yields Worthwhile Results
Developing an assessment that captures gains at the lower levels is only the starting point in a larger effort to build a system that works. Other efforts are needed, at both the local and the state levels, so that we don't end up with an accountability system driven in large part by what current standardized tests are able to measure. If we want the quality of adult literacy to increase, we need an approach that measures to what extent learners are acquiring the knowledge, skills, and strategies that matter in the long run. How can this be done? At the local level, a three-pronged approach might be necessary: (1) finding a way to live with the currently available standardized tests, selecting the "LOT" -- the least objectionable test -- and keeping in mind the principle of "first, do no harm" to students; (2) convincing the state that the data a program has provided over the years are at least as valid and reliable as standardized tests such as the TABE, and therefore the process should continue; and (3) work with others to develop an assessment system reflecting the realities of adult learners' lives and focusing on what participating programs have deemed to be the core sets of knowledge, skills, and strategies important enough to teach and test.

Components of an Alternative Assessment System
What might be the components of such a system? To start with, any program concerned about serving different groups of learners equally well, needs to collect demographic information capturing the kind of learner characteristics and exper-iences that may have a bearing on school success. After all, only by having rich descriptive information can we know what learners want and need to do with English and literacy, how much schooling they have had, and what the print and communica-tion challenges are that they face in their everyday lives. Having descrip-tive information of this kind is invaluable, since it allows us to see which learners are succeeding in our programs, and which are lan-guishing (or leaving) because their needs are not met. This information can be collected in the form of profiles that travel with the student and to which teachers and learners contribute on an ongoing basis. In addition to background variables such as age, employment status, years of schooling, country of origin and languages spoken, these profiles can: (1) capture current literacy practices; (2) chart shifts in learner goals; and (3) record changes in life circumstances.

In these profiles, progress can be captured as it occurs. Profiles have the added advantage of encouraging teachers to create opportunities for learners to discuss what is happening in their lives, so they can spend some time observing. Profiles of this sort (also known as "running records") can be connected with portfolios that demonstrate student progress through writing samples, reading inventories, and various types of performance tasks. If a stan-dardized test is used, results can be included in the profile as well, helping to flesh out the general picture of achievements and struggles.

From Learner Success to Accountability
This must be said: While an approach that combines rich profiles and individual portfolios will produce important information on individual students and provide insights into the relative success of certain learner groups, it does not, in and of itself, yield the kind of data needed for accountability. After all, we cannot ship boxes of profile folders to funders to have them realize what a great job we are doing. To make profiles work for funders, a further step is needed, one that yields data in aggregate form so that policy-makers can get a picture of the shape and size of the forest, not just a close-up of the trees.

To measure progress and report to funders, profiles need to include the following: a broad set of language and literacy tasks that are accompa-nied by rubrics, scales, and bench-marks for transition. Rubrics are used to indicate what expectations are for any given area (face-to-face communication, dealing with print, accessing resources, etc.) and what evidence of success might look like. The scales that accompany the rubrics allow us to document where learners fall on a continuum of proficiency, documenting what they can do with relative ease, where they succeed with some help, and where they are struggling. Since rubrics and scales can be designed for different skill domains (SCANS skills, communication strategies, navigating systems, etc.) and for various contexts (school, family, community), they can easily be matched to the goals of learners, and adapted to the focus of a particular program.

Once rubrics and scales are in place, meeting accountability requirements calling for aggregate data becomes relatively easy. Since the descriptors on a scale can easily be numbered (from 1 for "struggles" to 6 for "no problem") assessment results can be easily compiled, summarized, analyzed, and reported out. If matched with demographic profiles, they allow a program to see which groups of learners are being served well by the program and where program changes are in order because success is lacking.

The beauty is that this kind of approach fulfills the same function as standardized tests: learners are assessed on a variety of skills under standard conditions with common instru-ments on similar tasks. But unlike the standardized tests currently available, profile assessments do not rely on multiple-choice, paper and pencil items. Rather, they give learners the opportunity to demon-strate what they can do with language and literacy through more open-ended assignments. Furthermore, profile approaches to assessment can be adapted for certain learner groups and modified to match the focus of a particular program (e.g., workplace, family literacy, citizenship). Most importantly, they provide rich information that makes sense to teachers and learners, information that is useful to programs, not just funders.

Why then, are we not seeing more of these kinds of assessments? While extremely worthwhile and high in validity, these types of assessment carry a significant burden: they require consensus building on what is worth teaching and learning, and a common under-standing of what the evidence of success might look like for any given skill domain. To be successful, profiles and portfolios have to be integrated into the curriculum, and ongoing assessment must either be part of the day-to-day teaching we do, or time must be set aside at intake to establish a baseline, and toward the end of a teaching cycle, to document progress. If that means the end of open-entry/open-exit as we know it, and if it forces us into shorter instructional cycles that have a clear teaching/learning focus, so be it.

To give such a framework a chance, a significant amount of teacher orientation, training, and buy-in will be needed. Clearly, there are not many adult literacy programs that have the commitment, energy, and resources to embark on that endeavor, although some, like the Arlington Education and Employment Program in Virginia, are well on their way. But, given sufficient advocacy from local programs, along with a modicum of political will on the part of state directors and other funders, working groups and consortia could be set up to develop an assessment framework that, if not based on profiles, at least includes them. In fact, the National Institute for Literacy is moving in that direction, developing an assessment framework combining the use of alternative assessments with standardized tests where appropriate, in order to capture the gains that learners make who are part of the Equipped for the Future initiative.

What then is the bottom line, given the current climate of accountability for accountability's sake? We have several options: we can decide that cynicism is the only sane response to the current requirements, live with standardized tests as best as we can, try to lay low, figuring "this too shall pass," or commit ourselves to fighting for a saner system for our own sake and that of our students. On the local level, we must be prepared to work with others to decide on the focus of our programs, and be willing to map out a core set of knowledge, skills, and strategies that matter. At the federal level, we must push for an accountability system that is driven not by what the current standardized tests are able to assess (which is rather limited), but by outcomes that reflect what sound adult literacy programs should be all about. Furthermore, if we are asked to show accountability related to outcomes and impacts, we must be given the resources to document success in meaningful ways. Finally, while we may need to play the accountability game for the time being, we can also work toward a system that measures effectiveness where it counts: adult learners acquiring the kinds of knowledge, skills, and strategies that are important to them now and that matter in the long run. If we give up too soon, we will only marginalize adult literacy further.

Heide Spruck Wrigley is a Senior Research Associate at Aguirre International in San Mateo, CA. She can be reached by email at hwrigley@aiweb.com

This article was excerpted from Adventures in Assessment, Volume 11 (Winter 1998), SABES/World Education, Boston, MA, Copyright 1999. The original version is available on the Web at: www.sabes.org/resources/publications/adventures/vol11/11wrigley.htm