This is Only a Test…

Janet Isserlis
Literacy Resources/Rhode Island

I first became aware of assessment in adult education during the initial de-
velopment of the BEST, in the early ‘80s (see AIA, Volume 10, December, 1997, for a current consideration of the BEST). I had been working at a site in Rhode Island that had participated in the Center for Applied Linguistics’ overall development of both the test and the performance levels it measured, but never clearly saw the connection between the test itself and educational practice. I began to understand the notion of accountability, but I could never really tell how a particular score on the BEST would inform my teaching.

I still can’t.

I can, of course, get a sense of what a learner’s ability to take a test such as the BEST might be like based on his/her scores on one. I can see the range of content (life skills, math, science, reading comprehension) she knows and the kinds of items (essay, multiple choice, true/false, yes/no questions) that s/he is able to successfully complete or that s/he might miss. I can see if that learner has ‘done school,’ and to what extent, but for the most part, when working with adults with limited English language abilities and little prior schooling, most standardized tests are not teaching me very much about learners’ strengths or needs.

I can most ably assist a learner in finding an appropriate class/level placement and meeting her educational goals by speaking to her, by looking at a writing sample, by learning about what she knows about in the world, where or if there’s been prior schooling and in which language(s). This information gives me a sense of what a learner knows about using print as a resource for learning, and about her/his sense of what school can or should be. Learning about what s/he has done before coming to the learning center gives me a sense of what strengths and needs s/he’s bringing to the learning process.

In some settings, placement tests are helpful to practitioners in appropriately directing learners into the levels offered within particular programs; in and of themselves, however, the tests have limited value. Some programs have developed their own tests, others rely on standardized tools. For many learners, though, any test is daunting. Learners who may already be nervous about coming to school really don’t need to have an additional element of risk added to their initial encounter with the learning program.

In other settings, standardized tests are not useful to educators for placement or for progress assessment. They’re either too difficult (so that beginning level learners ‘bottom out,’ gaining scores that hardly indicate anything about their abilities) or do not measure what learning has actually occurred within a classroom. Programs feel caught between a rock and hard place; they have to show some pre- and post-learning movement, and tend to minimize the test itself to learners. (“It’s just to help us help you find the right level/get a sense of where you are now/show that you’ve been here...”).

Many programs have taken the time and energy to explore how testing can be used more appropriately, or how alternatives can be developed. Others report that learners who do show gains in standardized test scores feel encouraged by those gains.

But many practitioners indicate that tests terrify their students, or waste their time or that they can’t possibly prepare students to take so many different tests. (“I can’t send one out to a program that gives the TABE, another to a CASAS-based program, and yet others on to the GED…”).

I maintain that you can, but that’s not the point. If a learner identifies an academic goal for which testing will be an important and ongoing part of learning, there are strategies and approaches to prepare students to take tests and to pass them. Looking at the language of tests, the kinds of reading strategies that learners can use to make meaning and then to translate their understanding of that meaning into essay, multiple choice, true/false, or yes/no answers is possible. More important, however, is the question of what it is the students want to do.

As a state literacy resource center director, I not only work with and think about learners; I’m concerned, too, about program progress overall. I need to look at and think about ways to measure progress beyond gains made by individual learners, and to see how groups of learners progress over time. Looking beyond pedagogical concerns, I also need to be able to demonstrate learners’ progress to those who fund programs and to those who make policy.

As a vehicle of accountability, and of reporting larger spheres of need (and accomplishment) to those who do not work directly with learners but who do determine funding patterns and policy, the National Adult Literacy Survey (NALS) has most recently become the standard against which everything we do is measured in some way. The NALS indicates that a sizable percentage of the population of this country is ill prepared to engage in higher order thinking skills, having assessed those skills on a pencil/paper test. The underlying assumption here is that higher order thinking can (only) be demonstrated in writing, and that this is an appropriate way of measuring such thinking. The NALS claims that it assesses real life skills, yet it measures an adult’s ability to use those skills through a print-based medium. This is neither the time nor the place for a lengthy analysis of the NALS or its methodology. Rather it is a time to consider how it has been taken up as the largest umbrella under which all our other attempts at using test scores seem to fall.

The NALS, understandably, has been used to establish the need for increased services for learners. However, as a practitioner committed to not perpetuating deficit views of adult learning needs, I find it troublesome to contemplate the way these needs are flaunted through a rendering of NALS scores. Such renderings tend to emphasize adults’ lack of abilities with literacy in order to play up the need for increased access to educational opportunities for adult learners. In addition to flogging NALS findings within their states, programs also grapple with using learner outcomes that are based on a variety of tests — using gains in test scores to prove their effectiveness as educational providers; using low scores to speak to the need for yet more services to be provided. None of us uses the NALS to measure our own learners’ actual abilities, however, even if we accept that the NALS could provide us with useful information about those abilities.

Top

At a recent state-level policy meeting, a suggestion was made that we actively seek correlations between the NALS and other standards (such as the GED), in order to set more precise goals around learner outcomes. Such goal setting in and of itself does not improve learning, though; it merely shifts the focus (yet again) in terms of what learners and educators are being asked to demonstrate.

It seems I’m really addressing two broad issues: actual learning and accountability and the endless need for funding. Both interest me in different ways. How can adminstrators (and teachers and learners for that matter) argue intelligently for a meaningful way of assessing progress while also dancing to the multiple tunes played by the funders to whom they are accountable? How do we translate what we know about learning into a form that can be understood readily by those who fund and support our work?

Again, it seems that speaking about testing and accountability is speaking about reporting, translating gains that students make on a particular test to funders or interested others to whom that information is important. The work done within the last decade on broadening the range of what counts and how we count it has contributed significantly to the field’s understanding of the multiple ways in which learning can be measured. We still seem woefully incapable, however of translating actual learning gains into something that funders know, understand and appreciate as indications of learning having taken place. With constant turnover in the field itself, as well, we often need ongoing ways to assist ourselves, as practitioners, to learn to closely observe learners’ work and to translate our own observations into better understandings of their strengths and abilities. Such understandings can inform our attempts to improve practice to meet learners’ needs more appropriately. Adventures in Assessment is one way through which this work is being carried out, but more work on a program-based level is required if we really are to connect assessment and learning and move away from arbitrary and capricious testing and more closely into assessment that informs both learners and practitioners in meaningful ways.

Heide Wrigley (personal communication, August, 1998) suggests that we need to understand not only what works in terms of teaching/learning approaches and progress, but also “the implications ... for test development or assessment both large scale – program wide – and small scale, in the classroom.” This close learning can barely be gleaned from a barrage of test scores – even tests that look at ‘life skills.’ (Both the BEST and CASAS have ‘authentic’ tasks, if you consider it authentic to talk about how you go to the bank or to the laundromat, or whatever else one does without explicit talk in L1 or L2.) Yet, again and again we insist on using standardized tests to get a sense of a group of learners’ abilities so that we can report these abilities to funders. Nonetheless, I realize the significance of Heide’s suggestion that standardized tests can show us progress across groups more clearly than they give us useful information for teaching. Information gleaned from test scores can help us to see how groups of learners perform over periods of time. They give us a sense of a body of learners responding to tests of a body of knowledge.

Increased assessment — giving more tests — however, can not improve programs or learning until and unless those assessment instruments tell teachers about how their learners learn and don’t learn. Heide Wrigley’s article in this issue addresses those broader program and policy concerns, those big picture pieces that classroom teachers may not be aware of but which drive their work, one way or another, through shifts in policy, funding and mandates.

In terms of local programming and professional development, Heide’s suggestion that “we can either go to a profile system or we can develop a framework that combines some deeper classroom-based assessments with some quicker assessments that are done across classrooms and can be compared across programs (writing samples, for example, or participation in an interview or a discussion)” makes good sense. As Heide suggests, if we believe that something is worth teaching, we then need to know if students have learned it. How we go about getting that information is a central question.

Top

Professional Development
As indicated above, the work of learning how to understand learner progress needs to be integrated into ongoing professional development. We need to know what connecting assessment to learning looks like, how to help learners see their own progress through multiple lenses (longer journal entries, fewer mistakes at the blackboard, recognition of positive changes from teachers and from fellow learners). Several years ago, Francine Collignon and I undertook an action research process to develop one form of qualitative assessment of a group of beginning level ESOL literacy learners at the International Institute of Rhode Island. The resulting grid was a useful vehicle for us to review learner progress on a monthly level and to show our funder ‘evidence’ of that progress. The process of reviewing notes on classes, learners’ writing, interaction in class and general activity informed those grids and helped us shift our teaching practice accordingly. We never found a standardized test to help us get this information, nor were we obliged to submit scores throughout the course of that particular three-year grant. The work of documenting classes, reviewing those notes and discussing learner gains was time consuming. We had a relatively small number of learners, and time for preparation and analysis of our work. Most programs do not have these necessary luxuries.

There are ‘down and dirty’ ways of looking at learner progress. In addition to deciding what it is we want to assess (progress with writing? pronunciation? mastery of certain grammatical forms?), there are other factors that contribute to learning: class size, previous education, situational and dispositional barriers, gender, age and myriad other realities of adults’ lives. Classroom teachers know that these factors are all as important to learning as the quality of teaching itself. We still see, though, that a better test doesn’t give us a better program. So where are we now?

It seems that practitioners and administrators are still grappling with the issues I’ve raised above. Nancy Fritz, an ESOL teacher at the Genesis Center in Providence, is working on a mini-grant to study existing assessment protocols, in the hopes of adapting one to use with her students and to share with colleagues as the Center considers revising its curriculum. Nancy speaks to her concerns as a classroom teacher and as a member of a particular educational community:

During the 1997-98 academic year, I taught an intermediate level class of ESOL adult learners in a program that provides ESOL instruction at four different levels for five hours a day, five days a week. During the year, I came to realize some of the problems inherent in trying to assess our learners’ progress. This experience has led me to try to find or develop some instruments that will be helpful to me, other teachers and to our students in measuring growth in English. I’m doing this with the support of a mini-grant from the Literacy Resources/R.I. – the state’s literacy resource center.

In our ESOL classes we currently use the written form of the BEST test which is administered at the beginning of the year and again at the end in June. The problem that I see with the BEST test is that it is very difficult for many of our beginning level students. They have to guess at many of the answers and if they happen to guess better in September than they do in June, their scores actually go down. I saw this happen to a few students who had shown a great deal of improvement in reading, writing, and speaking English during the year. The BEST test did not reflect the progress.

In addition to the BEST test, we kept an ongoing portfolio assessment for each student. I think the idea of portfolio assessment is great, but the lack of time and the necessity to teach a large number of students made it impossible for me to do a good job. In putting together the portfolios, I also felt that we lacked any means of measuring the progress that had been made. Progress was evident if one took the time to read the material in the portfolio, but for a quick review this is difficult.

I am looking for assessment instruments that will yield a quantifiable result. Many of my students asked me during the year for ‘grades’ or ‘report cards’ like their children receive in school. I am not interested in such a cut-and-dried approach to documenting progress, but I would like to be able to show measurable results to students. In spite of the fact that they knew their English was getting better, they could provide numerous examples of things they could do now that they couldn’t do before, and they could see their progress in their portfolios, they wanted this tangible ‘measure’ put on their English progress. I would like to be able to take two pieces of writing done at intervals of several months and compare them according to criteria that are realistic for our learners and that also serve as a guide to good writing for them. The writing samples would receive a score based on how well they measure up to the criteria. In think this procedure could also be done with samples of spoken English. I think that this type of assessment would satisfy our desire for holistic assessment while at the same time provide a quantifiable result that would please the students, give them a sense of accomplishment, and make a quick review possible for teachers. I’m hopeful that I can find such instruments to use during the coming year.

Top

Nancy worries that “wanting some quantifiable assessment isn’t too contrary to the ideas in Adventures in Assessment. I’m not really obsessed with numbers and I understand that there are lots of drawbacks. I just think it would be helpful to have.” This concern indicates to me part of what’s been ingrained in adult education people for many, many years. The need for numbers exists in constant tension with the need for actual information with which learners can see their own progress, teachers and learners can talk about that progress and most importantly actual learning can be seen and increased. What we need seems light years away from what we’re finding on the TABE, on the BEST, through CASAS and even the GED.

Nancy will most likely develop an adapted tool from the many instruments she’s examining and will share her findings with her colleagues at the Genesis Center and with others in a workshop later this fall. I wonder, though, how far this learning will be able to assist us as a state (and by extension, I hope, as a field) in finding a more satisfactory way of reporting learners’ progress. Meeting with sharing/discussion groups, posting findings on Literacy Resources/RI’s website, and informally communicating with colleagues are all ways to consider getting the word out about Nancy’s work and about other possibilities around assessment. We seem to endlessly consider, though, what useful assessment does and looks like, with little hope of implementing ‘official’ assessment methods that funders can accept (see, for example, NCLE’s posting on research into assessment, 9/98).

One hopeful movement across this state and across the country is the evolution of the standards designed through Equipped for the Future, an undertaking of the National Institute for Literacy (NIFL). Sondra Stein, NIFL Senior Research Associate and Director, Equipped for the Future, relates that Equipped for the Future lessons “start from what adults need to know and be able to do in REAL LIFE, integrate skills instruction into work on meaningful projects, and assess learning results by looking at the application of skills in the context of carrying out real life tasks.” Dr. Stein acknowledges that there is “a ways to go before this translates into a framework for accountability” and proactively invites the field to contribute to and participate into this next step of the project.

Heide takes up the difficulties inherent in assessing even skills and tasks that learners themselves identify as being important. While not wanting to lose sight of those difficultires, I am also hopeful about the learning that may occur, at least within programs and classrooms, as a result of the work on the EFF standards as part of a broader ongoing series of professional development events and activities. These events and activities can include practitioner inquiry, development of in-house assessment tools, attention to active listening and observation, work with learners in naming goals and learning about how progress occurs, and investigations into learning styles. The possibilities are many.

In the long run, though, the same thorny issues remain. No matter how well teachers learn how to meet their learners’ needs and work from their strengths, it seems there will always be someone external to the learning process who will want to know how it is we know that learners have learned something and what it is that they have learned. Testing might give us some information about how well they’ve learned whatever content/skills we hope to have taught, but we need to be mindful of what is it learners are hoping to gain, what we want these tests to do, and to tell us. An increase in testing, or a shift in the focus of testing will not in and of itself improve learning. It might improve reporting, but will anyone learn more because their teachers are using new or different vehicles of testing and reporting? Keeping these questions in the foreground might help program workers as they continually work on the dilemma of measuring learning in ways that keep everyone informed, funded, and most importantly, learning.

Web Resources of Interest

Handouts for “How Do We Measure Progress in Adult ESL?”
http://www.cal.org/discuss/adultesl/eslprog/handout1.htm
From an online discussion about adult education through the National Clearinghouse for ESL Literacy Education (NCLE).

Let’s Get Started: An initial assessment pack for adult literacy programs.
http://www.nald.ca/clr/getstarted/cover.htm
An online booklet of information intended to help a tutor get started with a new learner from Manitoba Education and Training.

Portfolios in Practice. Andrea Leis with Rosemary Rognvaldson.
http://www.nald.ca/clr/portfolio/cover.htm
Hard copies, complete with graphics not available on-line, can be purchased for $5.60 each from Conestoga College. Send a check made out to: “Conestoga College” to: Bob McIver, Conestoga College, 435 King Street, North Waterloo, Ontario CANADA N2J 2Z5
Phone: 519-885-300, x241, Fax: 519-747-1195, email: bmciver@conestogac.on.ca Manuals will be sent COLLECT via Purolator.

http://www.cal.org/ncle/agenda
Research agenda for Adult ESL, prepared by NCLE in collaboration with the National Center for the Study of Adult Learning and Literacy (NCSALL) with support from Teachers of English to Speakers of Other Languages, Inc. (TESOL). The paper identifies issues recommended for further research and development in the field of adult ESL education.

http://www.brown.edu/Departments/Swearer_Center/Literacy_Resources/index.html
Janet Isserlis’ website.

Top

Originally published in Adventures in Assessment, Volume 11 (Winter 1998),
SABES/World Education, Boston, MA, Copyright 1998.

Funding support for the publication of this document on the Web provided in part by the Ohio State Literacy Resource Center as part of the LINCS Assessment Special Collection.