Using the REEP Assessment for ESOL and ABE Classroom Instruction

Joanne Pinsonneault and Carey Reid

The REEP is the mandatory assessment in Massachusetts for measuring adult ESOL learner gains in the area of writing. After administering the REEP twice to my class of mid-level ESOL learners, I discovered the test --with its warm-up activities, familiar essay prompts, and simple scoring rubric--to be very learner friendly. As long as I did not use official testing prompts in my classroom, there was no reason why I could not share useful aspects of it with my learners. After beginning where I usually do (by talking to my colleagues), I eventually wrote
a lesson plan. My initial objectives were to familiarize the learners with the REEP Writing Assessment's pre-writing activities, essay prompts, and scoring rubric as one way to help me determine the best course for teaching writing. I decided to walk them through the testing procedure, using my own pre-writing questions and writing prompts and discussing the content and purpose of each step as they actually completed it. I began with the rubric because I wanted them to understand how their writing was being judged.

Step 1: Helping learners to understand the REEP testing process

I gave each learner a copy of the REEP rubric. I then explained the essay-scoring process: Two teachers who do not know the learner read and score the essay by using the rubric. If the scores are within a one-point range, they are averaged. If the range is greater than one point, a third reader provides a score that must be within one point of either of the first readers'. Those two scores are then averaged. I demonstrated how the averaging system works.

Next, I wrote the five scoring categories and my explanations of them
on the board:

1. Content and Vocabulary (addresses task, answers questions, varies
vocabulary)
2. Organization and Structure (paragraph formation, details, essay structure)
3. Structure (sentence structure, grammar)
4. Mechanics (punctuation, capitalization, spelling)
5. Voice (addresses audience, engages/
persuades reader, embellishes if necessary to make interesting)

We discussed each category in general, and I described the expectations for each category at every level on the point scale.

Step 2: Administering a practice test

I asked the learners to recall the testing procedure: introduction, brainstorming, conversation, prompt reading, and writing. I explained standardization and the rules for administering this test. Now the learners were ready to take a practice test.

Step 3: Brainstorming

We discussed the purpose of brainstorming (to get ideas flowing), and, keeping within the five-minute time limit that the REEP allows, practiced brainstorm listing with a prompt that I created: What kinds of challenges do new immigrants encounter when they first come to America? Afterward, I asked the learners what they had learned from the process. At first, many had a hard time differentiating between the items on the board and the process that led to the discovery of those items. Then they began thinking about personalizing the list, discarding irrelevant information, and grabbing onto others' ideas. I modeled how they could create their own lists in their minds by adopting or discarding other learners' ideas.

Step 4: Conversation practice

I began to wonder if the conversational "warm-up" activity used as part of the official assessment could also be used for instructional purposes. Back in the classroom, I explained to my learners that the conversation piece is another pre-writing activity that can help them to generate ideas for writing. As an intervention, I pointed out that they should ask themselves, "What did I learn from my partner?" not "What did I learn about my partner?"

Keeping within the REEP's ten-minute time limit for this activity, they practiced with the following questions keyed
to the new prompt:

1. What is your name?
2. Where are you from?
3. When did you immigrate to America?

4. What was one challenge that you
confronted when you first arrived
in America?
5. How did you meet this challenge?

Considering the time limit, I suggested that they focus on the last two questions, because these will provide them with ideas for writing. After ten minutes, I asked: "What did you learn from your partner? Did she or he give you any good ideas for writing?" I reminded the learners that this exercise does not require that they tell "the truth" about their experience; they can create circumstances and embellish to make their essays more complex and interesting. Therefore, if they listen carefully as the teacher gathers comments from other learners, they might just get a few ideas to use in their essays.

Step 5: Understanding the prompt

To successfully address any writing task, the learners must decide who their audience is, what the topic is, and what questions have been asked. I divided them into small groups and asked them to consider three variations on the same theme (see Figure 1 below).

In their groups they discussed the differences among the prompts and answered the following questions: "Whom am I writing this for?"; "Whom/what am I writing about?"; and "How many questions do I have to answer?"'

Afterwards, they returned to the whole group for a discussion of the differences among the prompts. I asked the learners how they could use the pre-writing activities to their advantage in responding to each prompt. All of sudden it made sense to them! The comments were overwhelming. "My transition was so easy, but I could use what Mayra said about her life to help me answer the third prompt." By referring back to the rubric, we could point to how these pre-writing activities might lead to higher levels of writing quality.

Step 6: Administering the prompt

Finally, the learners took the practice REEP, under much the same circumstances as an official program assessment. All in all, these first six steps took about three hours, or one class session. But the work had just begun!

Step 7: Scoring

I scored the essays myself and, using a simple form, highlighted the strengths of each and offered concrete suggestions for improvement based on skills that I knew the learner had been working on and had demonstrated some ability with. (See Figure 2 & Figure 3.) For example, if the learner had previously demonstrated the ability to write complex sentences but didn't do so throughout the essay, I suggested places where simple sentences could have easily and effectively been combined. During learner-teacher writing conferences, we discussed the scores and comments. I also gave the learners the opportunity to rewrite their essays,
a standard option in our class.

Conclusions

I felt that I had achieved my primary objectives for the lesson: The learners gain increased awareness of the standards by which their writing is judged, and they had some initial ideas about how they can improve their writing. But how could I use this experience to inform my writing instruction? I'm still working on this, but I did spend some time looking at the scores of the class overall, and found a few surprises. Some of my work had been effective. For example, the majority of learners demonstrated good control over the verb tenses we had been studying! However, many learners were having difficulty with paragraph formation and needed a review of the punctuation rules. (Try again, Joanne!) I felt like I had a new place to begin, both with each learner individually and with the class in general.

I have also noted a few unexpected results, including reduced test anxiety and improved REEP scores. But best of all, my learners feel like they have more control over meeting their mandatory "improve writing skills"' goal. Teaching from the REEP has been very empowering.

Using the REEP for ABE Instruction: Carey Reid

I have been co-teaching a pre-GED writing class at the Jamaica Plain
(Boston) Adult Learning Program one evening a week with the class's primary teacher, David Stearns. David and I have been experimenting with an authentic materials curriculum, using Boston Globe articles, Net-based research, and even research reports from Focus on Basics as starting points for activities that require our learners to summarize, analyze, and respond in writing to these sorts of challenging materials. We've been delighted at how willing our learners are to plunge into this stuff and make sense of it, even when a good understanding will most often require at least three readings.

When I learned that Joanne Pinsonneault was using the REEP in her mid-level ESOL class, it occurred to me that the learners in David's class might be able to handle it. David was game to try it, and so were the learners. We decided to use the REEP materials as instruments for periodic formative assessment. Here's how we did it.

Stage One: Unpacking the rubric

We gave each learner a copy of the scoring rubric and spent about 90 minutes going through it in a whole group discussion. David and I explained that the rubric taken as a whole represented most of the elements that make up writing. We explained further that the rubric as an assessment tool provided good indicators for levels of quality and sophistication.

At that point, we started to just cruise through it, pointing out indicators at random and giving oral examples of how these indicators might align with a particular piece of writing (e.g., "If the writer provides a lot of supportive detail and can use complex sentences, then she's writing at such-and-such a level") After this exploratory phase, we asked the learners if they were willing to look at some sample essays and try their hand at scoring them. I am not exaggerating when I report that the learners were very intrigued by the rubric and equally eager to try applying it to actual writing.

Stage Two: Scoring the anchor essays in small groups

David and I really kept our fingers crossed with the next stage. Could the learners apply the rubric to actual writing with some accuracy, or would they become frustrated by such a task? We reminded ourselves that one of the reasons we decided to use challenging materials and exercises in our class was our determination to put into actual practice ideas and theories we professed to believe in-e.g., that adults learn best by doing, that they know what they want, and that they have acquired tons of knowledge through their life experiences. We did not want to appear to be the harbingers of special knowledge ("We know all the grammar rules, but you don't"), or superior in position ("The REEP is for teachers, but not for learners"), or unconsciously condescending by coddling or "protecting' our learners ("Concepts and materials from standardized testing are much too complicated for you to understand"). In fact, in our class-planning sessions, David and I would call each other on perceived instances of these kinds of presumptions.

For this stage, we decided to use the essays currently used in the training of Massachusetts teachers for REEP scoring. As in that training, we decided to use the six "anchor essays," each of which represents the same score across all levels-for example, a "3" all the way across. Specifically, the six anchor essays represent scores from 1 through 6. We told the learners that they would be looking for that consistent score level for each essay, explaining that this would simplify the exercise a bit, adding that this was the same process used to train Massachusetts teachers to understand the REEP.

David and I then organized the class into three workgroups of five learners each. We gave each learner a packet of anchor essays and a blank scoring sheet and asked that they read a particular essay together, score it together using the rubric, and try to reach agreement on a score. For some time, we'd been using collaborative learning approaches, so the learners stepped right into this task. One by one, the workgroups read and scored a particular essay, and then when all three groups were finished David and I asked the facilitators to report the scores. Then we discussed the scores and the reasons behind them.

Frankly, David and I were surprised by the results of this exercise. In nearly every case, the workgroups gave the same score to a given essay as the REEP "experts" assigned to it. In those cases where their and the experts' scores were different, the variance was only a single point. In addition, the three workgroups nearly always agreed on the same score, and when they varied it was only by a single point. In full class discussions, the reasons for giving scores were discussed, and where scores differed the workgroups defended their choices. Occasionally, an individual learner would disagree with the other members of her group; in the full class discussions, she would get a chance to defend her choice. (A great anecdote to share: One time a very shy learner revealed that she did not agree with the score the other members of her group had given. At that point, the other members asked her to speak up and defend her score. As it happened, her score matched that of the experts. Needless to say, the shy member's standing in her group went up a notch
as a result of that episode.)

Stage Three: Using the rubric to score learner essays

The class explorations of the REEP really intrigued the learners. When David and I said we hoped to apply the rubric to their own writing, they were completely supportive of the idea. At this point, David and I began to realize that because the REEP is not used as an official learning gains assessment in Massachusetts for non-ESOL learners, such as those in our class, we were free to adjust the materials and approaches as we felt we needed to. For example, we could alter the rubric if we wished, to make it more appropriate for our class. We were also free to design prompts that we felt suited our class better than those that are more appropriate for ESOL learners, which are based, for example, on personal letters or narratives. It struck us that it might be better for our learners to use prompts that resembled those used for the GED Writing Test.

The first prompt we developed basically centered on the question, What is the most important profession in modern society and why? To get the ball rolling, we started with an open discussion and some brainstorming, as Joanne Pinsonneault had done. Then we asked learners to write original essays for 30-45 minutes in response to a prompt based on the brainstorm question. (See Figure 5.)

David and I then read over these essays and, again as Joanne had done, scored them and noted those scores on a single-learner score sheet. We decided to list for each learner only a few Notable Strengths and one Area for Improvement. Our idea was to provide the learners with a Next Step for improving the draft, based on ideas about process writing we'd been applying all along. At the next class, David or I sat down with each learner and went over the scoring and the notes we'd made together. (We've provided an example of an essay and score sheet as Figures 4 & 5.)

Stage Four: Using the essays as "authentic materials"

At this point, everything we were doing with the REEP and with our overall authentic materials approach came together in a nice neat package: David and I decided to use the learners' essays as source materials, as the stuff that future lessons could be based on. For example, we gleaned fifteen sentences with errors from the various essays, listed them on a single sheet, and then asked the learners to discuss and edit them together in workgroups. The learners were much more motivated to edit samples of their own work than materials from workbooks.

Next steps: Using the rubric more widely

David and I are just beginning to experiment with using the rubric as a self-assessment and peer-assessment tool. For example, the class is now knee-deep in a several-week project in which we're applying the NCSALL research on learner persistence to the learners' own lives. We began by reading and discussing together the research findings (Focus on Basics, Volume 4, Issue A). The learners are now writing essays composed of an initial summary of the research, a commentary on how the findings apply or do not apply to their own lives, and a final section on ways in which our own writing class might be altered to better support their persistence(!). Now that most of the learners have a first draft, we will be asking them to use the rubric to self-assess the piece and, perhaps in conference with one of us, plan out the second draft.

David and I are also just beginning to try to integrate the rubric with a peer-editing process that the class is developing together. We've come up with rules and guidelines around constructive criticism; now we're wondering if the rubric would be a good tool for promoting objective and constructive critiquing. We shall see!

A final note...

David, Joanne, and I have presented these ideas in conferences and workshops over the past year. A common concern among participants is that the REEP scoring rubric might be set too high for new readers and beginning ESOL learners. Gradually, however, we've all come to realize that teachers are free to adapt these materials for their own learners. They can simplify the indicators, make them more positive sounding, reduce the number of levels, and so forth.

We would like to offer one caveat, however. Using the REEP rubric has, in our opinion, substantiated the claim among many theorists and experienced practitioners that adult learners know a lot more about writing than might be readily discernible. The learners often attach, constructivist-fashion, their working knowledge of writing elements
to rubric terminology: in a sense, they are enabled to give voice to what they already know. So, before simplifying or adapting the rubric, you might first want to find out if you really need to.

Joanne Pinsonneault is an ESOL teacher with the UMass Dartmouth Workers Education Program in New Bedford, Massachusetts. Carey Reid is Staff Development Specialist for Licensure and Assessment with the System for Adult Basic Education Support in Massachusetts.

Originally published in Adventures in Assessment, Volume 16 (Spring 2004),
SABES/World Education, Boston, MA, Copyright 2004.

Funding support for the publication of this document on the Web provided in part by the Ohio State Literacy Resource Center as part of the LINCS Assessment Special Collection.