Involving Learners in Assessment Research

Kermit Dunkelberg

" It is very difficult to begin learning foreign language, coming to an unknown country, especially if you are not as young as you used to be.

When I entered into ILI, I was enrolled into the intermediate level. I didn't understand everything at first, but could catch the gist of speech. We had a very nice teacher who spoke to us coherently. We read the articles from a local newspaper and discussed them, wrote small essays, so that way learning new words and inproving our spoken language.

Of course, we had practice after class, by watching TV, reading books, etc. In our class, we had students speaking differtent languages, it was an original practice too, speaking with them, and understanding them.

One day our teacher offered me and my pal to participate in a project. She briefly told us about it, and we agreed to participate. In this project, several students also joined in. The main idea was to help teachers to define the knowledge level of students. Each of us told our own opinions that foucused on what could improve our spoke language and grammar. To express your feelings in an unfamiliar language is very hard, epecially if you have lived here for only six month. Nevertheless, our teachers listened to us very patiently, and we made a little reference for each level of English study as as second language. Participating in this project, I improved my English, that was very helpful when I took exam for college."

- Dina Bakousseva, student participant on ILI's
Curriculum Frameworks Assessment Team

From January to July, 2001, International Language Institute of Massachusetts' Curriculum Frameworks Assessment Team worked to develop an approach to assessment which would be consistent with our teaching philosophy as well as the emerging assessment criteria of the MA Department of Education. Since one of ILI's core values is learner-centered instruction, we were interested from the beginning in learner-centered methods of assessment. After some deliberation, we decided to involve learners directly in our process of researching viable assessment methods.

What follows is an account of our six-month process involving learners in assessment. I have built this account on the minutes I took of our meetings, in order to give a picture of how the project unfolded over time. As often as possible, I have included the comments of students and other teachers as recorded in the minutes. While my account will inevitably be colored by my own perceptions, I want to emphasize that this was a team effort — involving students, teachers, and administrators. Our Curriculum Frameworks Assessment Team (CFAT, for short) included five teachers (Yvonne Telep, Jennifer Rafferty, Sarah Miller, Kermit Dunkelberg, and ILI's Director of Programs, Caroline Gear), and a diverse group of six ESOL students. The learner participants were: Dina Bakousseva and Elena Sidorova from Russia, Gulsen Kosem from Turkey, Eric Lin Qu from China, Yumiko Millette from Japan, and Amos Esekwen from Congo. They ranged in Level from ESOL 5-8. I was particularly pleased that students outnumbered teachers on this project.

Laying the Groundwork

Several months before our project formally began, teachers at ILI began to meet around issues of assessment. On September 7, 2000, three teachers (Cindy Mahoney, Yvonne Telep, and Kermit Dunkelberg) discussed the pros and cons of standardized vs. alternative assessments:

We discussed the pros and cons of using a standardized test, such as the STEL [Standard Test of English — a grammar test used by ILI's Intensive English Program as part of intake assessment] for ongoing assessment. On the one hand, teachers felt that grammar is a good indicator of level. On the other hand, we questioned the validity of measuring via a standardized test. Some students have expressed a desire for more testing, others have strong opposition to traditional tests. Standardized testing seems, to us, to run counter to the philosophy of both ILI and the Curriculum Frameworks. But DOE's increased demands for accountability, and the fact that they have provided us with a long list of possible standardized tests (in SMARTT, Massachusetts' data collection system) suggests that there may be a place for standardized testing [in future assessment systems].

October 4-5, I attended the DOE Director's Meeting in Falmouth, and wrote the following notes to ILI's teaching staff: I attended sessions on Performance Accountability and NRS (National Reporting Systems). Of all big topics, this is the biggest:

—"What motivates students to come
to us is what defines our accountability."
(Bob Bickerton, MA State Director
of Adult Education, 10/04/2000)

I believe DOE is really committed to learner-centered-ness, and does not want to mandate, from the top down, what we teach. But at every level (us to DOE, DOE to Feds or State Legislature), there are the twin burdens of 1) determining what students need; 2) verifying that through data collection. We could cynically try to give DOE "what they want," or we can really listen to our students.

In the rest of my notes from the Director's Meeting, I raised some of the questions our Assessment Team would grapple with for the next months: Is a standardized procedure the same as a standardized test? What is a rubric? How can assessments be valid, reliable, and learner-centered?

Laying Out the Issues

On January 26, 2001, four teachers (Caroline Gear, Yvonne Telep, Jennifer Rafferty, and Kermit Dunkelberg) met to discuss these issues. From the beginning, we chose to view our work positively:

As we move forward, we will want to think of how to establish a time line and realistic benchmarks for what we can accomplish. We will also want to figure out practical ways to divide up the work, and develop mutually productive scenarios for learner involvement.

We move forward in the conviction that we have the potential to positively impact the ways in which we will be "asked" (required) to assess learners and program effectiveness in the future. We also have the chance with this project to strengthen our programs...

We began with each person present brainstorming (on whiteboard) on what s/he associated with the "universe of assessment" we were being asked to address. Among the issues raised were:

• What is the best timing of assessments?
• Will assessment be different in our classroom and Distance Learning programs? Is what is "valid" in one instructional setting equally valid in the other?
• How can competencies beyond the four language skill areas be assessed? (For instance, computer literacy, Intercultural Knowledge and Skills, or Navigating Systems)
• How can our assessments match what we're doing in the classroom?
• Learners need to be able to interpret the tools!
• Let's not reinvent the wheel! Look to models that are out there!
• How can our assessment procedures get around teachers' inevitably subjective responses to individual students?
• Will standardized tests have a place in our assessment procedure?
• What kinds of assessment tools are our learners most comfortable with?
• Accountability:
  — To our Learner's needs and goals (our primary accountability)
  — To Federal and State guidelines (WIA, NRS, DOE): our imperative to stay funded!
  — Accrediting organization (ACCET)
  — ILI's institutional culture: (Learner-centered, flexible, authentic)
• How can learners be meaningfully involved in this process?

Top of page

Beginning the Research

Over the next two months, we explored a wide variety of existing assessment tools, from standardized tests to performance-based prompts and rubrics. We began to review a range of writing rubrics from various sources (including K-12 and English Language Arts, as well as ESOL), and to use them to score sample student writings to see if we agreed on the level the writing represented. (We didn't quite know it yet, but we were testing our "inter-rater reliability" with various rubrics!). We discussed the pros and cons of these rubrics, and began to work on developing our own. At the same time, we worked toward a common understanding of key vocabulary ("valid," "reliable," "authentic," "standardized," "holistic," "analytical"), and key debates in the field. Members of the Team participated in a workshop on standardized testing and another on EFF (Equipped for the Future), both at SABES West (the western region of the state). We also participated in the Western MA SABES Assessment Work Group, in which we joined colleagues from other programs in investigating various models of assessment. We read a host of articles on assessment, which we found in Field Notes, Focus on Basics, or on the web. In EFF terms, we were rapidly expanding our "knowledge base."

Still, we had not found a comfortable way to involve learners in the process. The learning curve was steep enough for us. How could we expect learners to master these issues? On the other hand, we reasoned, perhaps students would bring their own knowledge base to the project, offering a different kind of expertise? After all, they know far more about what it's like to be a student in our program than we do!

First Meeting with Learners

On April 26, 2001, we had our first meeting involving learners. We were still unsure how to involve them in a meaningful way, but we figured, "let's invite them and decide that together!" We noted that:

One value of involving learners in our meetings is to offer them a meta-perspective on the assessment issues, allowing them to advocate for what makes sense to them, and to report back to other students from a student perspective.

The six learner participants had been chosen from our 5-6 and 7-8 SPL level classes on the basis of their interest, communication skills, and ability to work closely with teachers. Prior to the meeting, student participants had been given a brief orientation, and were given a packet which included:

• A Position Description (See
  Appendix B)
• "Words and Phrases You Will Hear
  A Lot" (See Appendix C)
• CRESST Assessment Glossary
• NRS Guidelines
• ESOL Curriculum Frameworks Chart
• Timesheet

The Student Participant Position Description emphasized that:

Our main goal in involving student participants is to be sure that we are listening to student voices as we build assessment policy. We would like you to share in our discussions, to offer your opinions, to do some reading and research with us, and to try out some of the assessment methods we are working on. Your experience and perspective as learners in the classroom are important to us.

We hoped to develop assessment procedures which were consistent with our classroom culture. I decided that our process should reflect our classroom culture, too. From now on our meeting style would draw on our classroom style, emphasizing pair and small-group discussion. As in the classroom, this would give everyone a chance to speak, and make sure student voices were constantly heard. The first activity of our joint meeting was to pair up teachers and students to discuss:

Why did you say 'yes' to this project? What is interesting about it to you?

Teachers answered that they wanted:

... to know more about different types
of assessment.
... to know how to be able to tell
students if they are making progress.
... to help students learn how to assess
their own progress.
... a system of assessment.
... to give the students confidence.
... to learn about state and federal
guidelines.
... to keep funding for our program.

Students answered that they wanted:

... more conversation/listening practice.
... to be part of decision-making.
... a chance to be involved in how to
know...am making progress.
... to help us make the time in class
as useful and important as possible.
... I am against the MCAS—I don't want
to see adults take a test.
... Assessment should be useful.

We agreed that:

Both students' and teachers' motivations for participation in the project should be respected.

Student/teacher pairs then discussed an area of competence (not language learning), and answered the question:

"How do I know when this is being done well/successfully?"

Examples included: Waiting Tables, Teaching Dance, Movies, Finding and Hiring New Employees, and Dee Jaying in a Club. In groups, we discussed the correspondences between "assessing" these activities and assessing language. We discussed what is subjective, or objective, about these assessment processes. This raised the question of the difference between "assessment" and "evaluation." Evaluation, we felt, was more informal and subjective. For instance, the teacher who had been a Dee Jay stated that she continuously modified the music she played based on her subjective feeling of what was the "right music" for the "right time." Evaluation often employed a "rule of thumb" rather than a standard measure. Assessment relies on quantifiable data, and has at least the appearance of being more objective.

Would an "objective" assessment system even be desirable in some of these cases? (Are movies which have been vetted by numerous "focus groups" more satisfying than those which have not? Often the opposite is true). What is easy to count? Does it tell us what we want to know? For instance, when the student who had been a waiter was asked if a waiter's ability could be assessed by counting tips at the end of the night, he emphatically said "no." Ultimately, he felt the best measure of a waiter's ability was evaluation by an informed observer (the manager). And not on one night, but over time.

Evaluation and Assessment

Finally, we asked: How does this exercise inform our process of developing assessment strategies at ILI? Students and teachers agreed that evaluation is a strong part of our day-to-day teaching at ILI. Oral and written feedback are incorporated into all of our classes, so that instruction is in response to student's goals and needs. Talking about the non-language learning situations above clarified some of the strengths of constant evaluation. Somewhat like the Dee Jay in the club, the classroom teacher modifies instruction in response to constant feedback and observation. Like the restaurant manager, a classroom teacher observes students over time, and has a good sense of students' day-to-day (as opposed to one-time) performance.

While we agreed that ILI had long been strong in the area of evaluation, we also agreed that there was room for improvement in the area of assessment. Specifically, teachers and students both wanted to be able to say more clearly to what degree an individual student had improved. Students were not so much interested in an abstract number as they were in knowing what they needed to do next to improve. Similarly, while teachers recognized that reporting requirements (to the Feds and DOE) are a "fact of life," and arguably can lead to program improvement, our primary allegiance was to the growth of our students. Consequently, our most pressing concern was our ability to document learner progress to the learner: to be able to say "you used to be able to do these things, now you can do these more advanced things, and this is how I know." Again, we were less interested in a number (a BEST or TOEFL type score) than in being able to articulate to students what skills areas they had improved in, how we noted that change, and what they might need to focus on next.

Top of page

Level Descriptors

This discussion brought us back to the subject of level descriptors.
( 1 ) At our next meeting (May 3, 2001), we looked closely at the NRS SPL descriptors. Both students and teachers found that the NRS guidelines were general and inconsistent. Moreover, students couldn't understand them easily. We felt the NRS descriptors were not adequate to serve as the basis for a dialogue between a student and a teacher about what the student's level was, and what that meant for the student's progress. Why not? We came to two conclusions:

• The NRS guidelines define entry level requirements, but our students want a continuum. Students want to know all the steps for moving through a level to the next one. They want to be able to see how the levels "connect" to each other.
• The NRS guidelines are difficult for students to interpret. We want our students to be able to look at the descriptors and say, "Ah! That's my level!"

Yvonne Telep brought in level descriptors from Washington State.
( 2 )They provided a model for us of a graded SPL descriptor, in which each level was divided into three or four sub-levels, enabling students to see more precisely where they stand in relation to a level.

We determined we would write our own level descriptors, consistent with NRS yet more detailed, in accordance with the MA DOE Curriculum Frameworks and our classroom system of instruction. In order to make these descriptors accessible to students, we decided to write them in student/teacher teams. (One of the greatest days for me, as Project Coordinator, was the day our Director of Programs sat down with a Level 5 student to begin writing a draft level descriptor for Level 5 Reading).

This was only the second meeting involving students, but already they were taking their place beside us as equal partners in the work. We were amazed at how interested they were in the issues surrounding assessment (clearly, a lot was at stake for them), how quickly they were mastering complicated material, and how effectively they were working with us as team members. Students were getting a lot out of it, too. We did short written feedback at the end of each meeting (part of the ILI culture of continuous feedback and observation). Student responses from the April 26 meeting included these comments:

• Today, we focused on the level description. And we found something that we really like and don't like and even some questions after we deeply touched every sentence what/how they separate the different level. I thought it's very useful and make me more get the topic what we will do and discuss
• I really like being in this project.
• Today I have get a lot of things.
• We had conversation about students level and everybody from us decide that a grammar and vocabulary are important for studying English. It was very useful personally for me because I had a big practice speaking with native citizens. I had a fun!

Student-Centered Assessments

Even as we moved forward with drafting more useable level descriptors, we continued research into what kinds of assessment procedures would be used to determine the level. Students had told us clearly that their time was valuable, and that the assessment procedure should be a learning opportunity.

At the same time, teachers were wary of "teaching to the test," if a standardized test were implemented for state-wide assessment. Instead of "teaching to the test," could we "test to the teaching"? Could the dog wag the tail, instead of the tail wagging the dog? Could assessments model classroom instruction?

Examples from Colorado State

Caroline Gear brought in materials from the Colorado State Department of Education. The Colorado DOE materials provided examples of clearly-defined performance-based assessments modeled on familiar classroom pair and small-group activities like role plays, interviews, information gap, and "Listen, Repeat, Do."
( 3 )

Both teachers and students were very excited by the Colorado model, because the performance tasks fulfilled some of our emerging criteria for learner-centered assessment:

• The assessments presented authentic speaking and listening situations. Students were communicating with each other with a real need to communicate.
• The authentic situation provided an opportunity for including Standards from ESOL Curriculum Frameworks Strand 5, Developing Strategies and Resources for Learning, as part of the assessment.
• The tasks were modeled on familiar classroom activities which students had done before.
• Assessment time was also learning time (the assessment procedure included a volunteer to assist pairs/groups who were not being assessed).
  

On the other hand:

• The assessments were complex to administer, requiring the assistance of at least one trained volunteer in addition to the teacher.
• The pair and group model was very appropriate for our classroom program, but not as appropriate for our Distance Learning Program.

Moreover, despite repeated requests, we were unable to get copies of the scoring rubrics for these assessments from the Colorado Department of Education.

Equipped For the Future (EFF) Standards

Our interest in learner-centered, authentic assessments also led us
to examine the National Institute for Literacy's Equipped for the Future Standards.( 3 ) We were attracted to EFF in part because, in determining "What Adults Need to Know and Be Able to Do in the 21st Century," NIFL had begun by asking adult learners what they felt they needed to know.4 #Also, as a federally-funded program, EFF had "clout," and seemed to us to offer a different perspective on learning from the NRS. We were therefore astounded and pleased to learn that EFF is developing a comprehensive, performance-based assessment system which will be linked to the NRS! ( 5 )

Unfortunately, the EFF assessment project had just begun in 2001, so none of their assessment materials were available to us.

Both students and teachers found the EFF Standards to be consistent with the ESOL Curriculum Frameworks. EFF seemed to us to provide new ways of thinking about aspects of the Frameworks which were important to us, but which we weren't sure how to incorporate into assessment. These included Strand 5 (Developing Strategies and Resources for Learning) and the Seven Guiding Principles. We saw a strong correspondence between these aspects of the ESOL Curriculum Frameworks, the EFF Standards for "Life-Long Learning Skills" and "Interpersonal Skills," and learner observations such as "What is hard for me is not pronunciation, but the fear of (mis)pronunciation!" and "Our level can change everyday." ( 6 )

Finally, we responded warmly to EFF's emphasis on communication as a shared, negotiated act (something not reflected in standardized pencil and paper assessments, but vital to our students' motivations for coming to our classes). This was reflected even in the titles of such EFF Standards as "Listen Actively" and "Speak So Others Can Understand."

Top of page

Level Descriptors, Again

At our May 10 meeting, we returned to the task of writing better SPL descriptors. We asked ourselves what sources we should consider in writing level descriptors, and agreed on the following short list:

• NRS Guidelines
• Massachusetts Curriculum
  Frameworks
• ILI in-house materials
• EFF
• Washington State rubrics

I would add that an additional, implicit resource was our students' own statements about their level. We asked students in the group to describe their "level" to us. What could they do easily, often, with confidence? What was difficult for them? In some cases, student members of the team moderated discussions with their classmates, eliciting responses to these same types of questions. Student's own descriptions of their "level" tended to be experiential rather than academic, with rich detail about such aspects of communication as confidence and cultural context, as well as "learning gains" nearly impossible to assess (i.e., "Now I can think in English!") Student descriptions of their challenges and successes informed our understanding of the "levels."

We divided into small groups to consider one skill area, at one level (Speaking Level 5), from each of these perspectives. We then met in large group to identify barriers to our process by discussing:

"What was hard about what we just did?"

• We identified the following difficulties, among others:
• Neither MA Curriculum Frameworks
  nor EFF are divided by level
• The NRS progression is not clear
• The levels in NRS and Washington State are not organized the way DOE classes in MA are organized.
• Our in-house materials are not clear or accurate enough.
• Washington State describes only 6 levels.
• It is difficult to make the descriptors precise without being prescriptive

We brainstormed about what "aspects" we should look for when assessing Speaking. Some that we agreed on were:

• Pronunciation/Intelligibility
• Grammar
• Usage
• Risk-taking, for instance use
  of new vocabulary
• Fluency, Rate
• Confidence

Another group of aspects was more controversial. These tended to reflect socio-cultural dimensions of communication. We all agreed they were important, but could/should they be part of assessment of levels?

• Social conventions of oral
  communication
• Context
• Register

I was reminded of Juliet Merrifield's comments on shifting definitions of literacy in "Contested Ground: Performance Accountability in Adult Basic Education":

"Over time, views of what literacy means have shifted from academic skills... to functional skills...Literacy is now described as multiple 'literacies' rooted in particular social contexts...When literacy meant what is taught in schools, performance was testable... The research on literacy in its social context has been carried out through careful observations of literacy events and activities which shed light on prevailing literacy practices....While it shifts the focus to performance in life, not in test situations, this new research has not yet been incorporated into practice, assessment, or policy." ( 7 )

If we believe, as Merrifield suggests, that literacy is "rooted in particular social contexts," then considerations such as social conventions, context, and register need to be taken account of in our assessment procedures.

Responses to the May 10 meeting included these comments, from students and teachers:

• Yesterday I had no idea how we would write a level description, today I have some ideas.
• What we did felt like progress, like we are swimming forward instead of treading water.
• The last 2 meetings we only talked in general, today we made ourselves more aware and organized to focus on one part.
• The most thrilling part was to hear (my student) use the word "concise."
• I learned that communication is a big deal! I like the way EFF says "Speak So Others Can Understand."

After this meeting, we abandoned our "parallel track" of drafting better SPL descriptors, while, at the same time, trying to create performance tasks and scoring rubrics. We focused our remaining time on creating SPL descriptors for as many skills and levels as we had time for, working in student/teacher teams.

As a first step, we each (students and teachers) wrote a descriptor for ESOL Speaking Level 5. The richness, complexity, and sheer difficulty of our task is reflected in this list of "highlights" of the various descriptors from our May 17, 2001 meeting:

• All the teachers agree that the thoughtfulness and quality of the students' contributions is extremely high.
• Jennifer's is admirably concise and seems to fit with NRS.
• Eric's is easy to understand and includes a sense of communication as negotiation.
• Dina's suggests good things for students to keep in mind, including confidence.
• Caroline's identifies competencies to be addressed throughout the level.
• Elena, Yumiko, and Kermit based theirs on NRS mandates. Elena included non-verbal language (eye contact and body movement), and stressed the EFF "Speak So Others Can Understand."
• Kermit's had a list of functions, levels of performance, good summing up of grammar and vocabulary, and had Intelligibility as a category.
• Amos emphasized active, rather than passive role in Level 5 ("initiates conversation"). Had a real learner's perspective.
• Yvonne's charted a clear progression. Elena felt it would help students see where they were on the continuum.

We all agreed that Yvonne Telep's proposal—based on the Washington State example, with the addition of a range of "competencies" which could serve as the basis for performance-based assessment tasks, was a major breakthrough (see Appendix D). We agreed to follow the format she proposed. In the remaining weeks of the project, we developed SPL descriptors for Speaking 3-8, Reading 5, and Writing 5-6.

Top of page

Curriculum Frameworks Showcase and ALRI Workshop

Students in our project conducted research, debated the conceptual framework of what we were doing, and participated in drafting rubrics and level descriptors. Students also became involved in sharing the results of our work with the field.

Student participants undertook the design of our display poster for the 2001 DOE Curriculum Frameworks Showcase on June 8. Elena Sidorova, a Level 6 ESOL student, designed a computer graphic showing "Our Influences and Inspirations," which was the centerpiece of the poster. The chart included Curriculum Frameworks, NRS, EFF, MELT, SABES West Assessment Group, Colorado Performance Assessments, Washington State Rubrics, "Field Notes", "Focus on Basics", ILI curriculum, ILI crosswalk, Students' Voices, and Teachers' Experience.

On June 14, ILI conducted a day-long workshop on Learner-Centered Assessment at the Adult Literacy Research Institute (ALRI) in Boston. Several of our learners participated in presenting the workshop. Like our assessment meetings, much of the workshop was conducted in small groups, in which our learners mixed with the teachers and SABES support staff who were present. The unanimous feedback of the group was that having learners at the table, helping to conduct the workshop and offering their perspectives throughout, was the greatest value of all.

For our learners, too, this opportunity to synthesize the lessons learned from our project, and to convey them to a group of professionals in the field, was a turning point and a fitting culmination to our work. One student told me several months later that she had accepted a new job which would require her to give occasional presentations. She felt prepared to accept the job because of her work on our project, in particular the experience of presenting the workshop at ALRI.

Moving On

Of the six students who participated in the project, only one is still a student at ILI. Three have gone on to college, and two more moved on to better jobs. Their success certainly says a lot for the individual qualities which led us to select them for the project in the first place. But I believe the project also served as a catalyst for each of them to take the next steps toward their dreams.

Fortunately, they have left a lasting legacy. As we at ILI continue to move forward in developing assessment procedures which will serve both government reporting requirements and our students' needs, the lessons we have learned from this project stay with us. The voices of these students still ring in our ears, and so it is only fitting to close with a few of their comments about the project:

• It was a lot of toil. But I would do it again. All students should participate in this project.
• Everything was OK! What was done and what will be do it's very interesting and useful personaly for me...
• As a student, I'm very pleased to be part of this working group. It's wonderful to know what teachers think about learners.
• We found out that everybody have some good points and very useful. After deeply talking, make more impression in our mind. That's good.

Kermit Dunkelberg is an ESOL teacher and Program Coordinator at the International Language Institute of MA in Northampton, Massachusetts.

Notes

1. We began by examining some of the historical background to the NRS. (Our information came from the March 2001 Implementation Guidelines of the "Measures and Methods for the National Reporting System for Adult Education, " found on-line at
http://www.air-dc.orq/nrs). We were surprised to find that the NRS had arisen, in part, out of an effort to save adult education from being subsumed by "a general system of workforcedevelopment." In 1995, Congress had demanded "strong and convincing data" to demonstrate adult education's "effectiveness as a separate program." State directors of adult education had asked for "a national system for collecting information on adult education outcomes." So, while some of us in the group had regarded the NRS guidelines as narrowly workforce-centered, they were in part an attempt to stake a claim for a broader conception of adult education. We were also surprised to learn that: Among the sources used to develop the NRS were the National Institute for Literacy's EFF (Equipped for the Future) and the CASAS (California Adult Student Assessment System). The two types of assessment specifically mentioned in the NRS are "standardized test, or 4 performance-based assessment with a standardized scoring rubric."

2. The Washington state leveldescriptors are available, with some diligent searching, from http://www.sbdctc.edu/Board/Educ/ABE/assess.htm

3.The materials were obtained at a presentation by Kathleen Satopietro Weddel (Northern Colorado State Literacy Resource Center) at the 2001 TESOL conference in St. Louis. Colorado assessment materials are available from:

Marie Willoughby
ColoradoDepartment of Education
CARE/family Literacy and Adult Education,
201 East Colfax Avenue, Room 100,
Denver,CO 80203.

4. The Equipped for the Future Content Standards are available
from the

National Institute for Literacy (NlfL)
1775 1 StreetNW, Suite 730
Washington, DC20006.

5. See EFF Voice 2:1, Winter 2001, also available from NIFL, and
on-line at http://www.nifl.gov

6. Compare these student remarks with the language of the Frameworks: "Language learning requires risk-taking." (Guiding Principle 5). "Teachers and learners need to understand that progress may be inconsistent from day to day." (Guiding Principle 4).

7. Juliet Merrifield, "Contested Ground: Performance Accountability in Adult Basic Education," NCSALL Report #1, July 1998. Available on-line at
http://gseweb.harvard.edu/~ncsall/research/report1.htm

Top of page

Appendix A

Questions which came up in an SPL level 7/8 class
when it was proposed that we try one of the CASAS writing assessments

• You mean, like a "test"?
• What's wrong with tests?
• Do we want to spend half an hour of class time just writing?
• How often would we do this? (If not often, how can one measure
  accurately reflect what a learner is capable of? What if I"m tired that day?
  If too often, see previous question!)
• Is it fair to have a time limit? (Some students write faster than others,
  although not necessarily better).
• Couldn't we take it home to do? (If we did, would it still be valid?)
• Could we use a dictionary?
• Should we be assessed on a draft, or a revised piece of writing?
  (This class had been emphasizing rewriting).
• Could we write on any subject we wanted to?
• Would we write about something we had done in class?
• Why not just use some writing from our journals?
• You mean, like a "test"?

Appendix B

Student Position Description
International Language Institute of MA Curriculum Frameworks Assessment Team

Purpose: The main purpose of the Assessment Project is to improve our methods of evaluating student progress in language learning. In other words, how do teachers and students know when, and how much, a student has improved in the "four basic skill areas" of speaking, listening, reading, and writing? Closely connected to this is the question of which class a student should be in, and when and how students progress to another class. In addressing these questions, our program has to consider ILI's teaching philosophy as well as state and national guidelines for Department of Education programs.

Student Participant's Role: Our main goal in involving student participants is to be sure that we are listening to student voices as we build assessment policy. We would like you to share in our discussions, to offer your opinions, to do some reading and research with us, and to try out some of the assessment methods we are working on. Your experience and perspective as learners in the classroom are important to us.

As a student participant, you are expected to:

• Attend a Thursday afternoon meeting once a week or once every two weeks.
• Learn a little about the "big picture" of state and national guidelines which our school must fit in with.
• Learn some special words and concepts we use to talk about assessment.
• Participate in discussions with the teachers about how to improve this part of our teaching.
• Write an evaluation of your participation in the project to include in our final report.

Project Dates: Week of April 23 through June 30

Stipends ($): All student participants will receive a weekly stipend of $32 per week, or $8 per hour for four hours of work a week. This will include both meeting time and reading and research time. You will probably find that some weeks you work more than four hours, and other weeks less, but the weekly stipend will be the same. The stipend is to honor the value of your work on the project.
It is not a wage.

Thank you for your interest!

Top of page

Appendix C

"Words and Phrases You Will Hear A Lot"
Student Participants — ILI Curriculum Frameworks Assessment Team

ABE: Adult Basic Education

Assessment: A way of measuring progress. Assessments should be "valid" and "reliable." Assessments are also "countable." We must report our assessment results to the Department of Education (DOE)
to show that our program is successful. In this project, we are focusing on assessment of the four "skill areas" of language learning.

Curriculum: Plan of study. What is taught, and how it is taught.

Curriculum Frameworks: The state guidelines for DOE Adult Basic Education (ABE) classes in English for Speakers of Other Languages (ESOL). The Frameworks are supposed to be a guide for how to teach, not what to teach.

DOE: Department of Education. The money for our program comes from the Massachusetts DOE.

EFF: Equipped for the Future. A federal study of "what adults need to know and be able to do in the 21st century," as workers, learners, family members and citizens. EFF provides Content Standards with useful descriptions.

NRS: National Reporting System. The NRS describes national SPL's (Student Performance Levels). All programs should fit with NRS descriptions (but the descriptions are not clear).

Performance-based Assessment: Measuring how well a student does at a particular task (speaking, writing, etc.).

Reliable: Able to give the same results each time, no matter who is testing.

Rubric: A scoring grid to measure student performance.

Skill Areas: We identify four "skill areas" for language learning: speaking, listening, reading, and writing. Students may be at different levels for different skills.

Student Performance: Level of performance expected (0-10). At ILI, we have classes at the 3-4, 5-6, and 7-8 levels. These are multi-level classes (more than one level).

Valid: Measures what you want to measure. "Valid" also means that an assessment "fits" with a program's curriculum and philosophy of learning.

Appendix D

Level Descriptor
Speaking, SPL Level 5
Draft/ Yvonne Telep
International Language Institute of MA

Functions: (Also referred to as competencies.)

Student will participate in the following activities or exchanges:

• Reporting an event in the past, such as an accident, a previous job or any specific experience.
• Explaining the steps in a process.
• Using polite language/ accepted conventions to request clarification and repetition when needed.
• Making polite requests.
• Conversing in limited social situations in English.
• Expressing agreement or disagreement.
• Stating reasons or giving excuses.
• Expressing future plans or needs.

Appendix D

Level Descriptor
Speaking, SPL Level 5
Draft/ Yvonne Telep
International Language Institute of MA

Exceptional (Exit Standard)

Individual demonstrates control of the simple, progressive, and perfect present and past tenses. Errors are infrequent and don't interfere with meaning. Syntax and pronunciation are usually intelligible, although errors still occur. Syntax and pronunciation are not perfect, but competent enough so that meaning is clear. Participation in the exchanges listed above is clear and intelligible with only occasional hesitation or errors. Individual is aware of errors and is able to self-correct, or to restate and clarify when asked to do so.

Competent

Individual demonstrates control of the simple and progressive present and past tenses and errors with infrequent errors which don't interfere with meaning. Individual shows awareness and understanding of the perfect present and past tenses but errors and omissions are frequent. Syntax and pronunciation may be problematic but meaning is generally clear. Individual can participate in the exchanges listed above with some hesitation, but with increasing confidence. Individual can sometimes self-correct, or is able to restate or correct when asked to repeat or clarify.

Developing

Individual uses the simple present and past tenses with occasional errors that don't interfere with meaning. Individual also shows awareness and understanding of the progressive present and past, but makes frequent errors. Grammar and syntax errors are less frequent, but may still interfere with meaning. Pronunciation problems occasionally interfere with meaning. Individual can participate in the exchanges listed above with hesitation and some correction. Individual shows a willingness to participate in the exchanges listed and understands mistakes when made aware of them.

Beginning

Individual has understanding of the simple present and past tenses. Errors of grammar and syntax are frequent and sometimes interfere with meaning. Pronunciation problems occasionally interfere with understanding. Individual can participate in the exchanges listed above with hesitation and assistance.

Top of page

Originally published in Adventures in Assessment, Volume 14 (Spring 2002), SABES/World Education, Boston, MA, Copyright 2002.

Funding support for the publication of this document on the Web provided in part by the Ohio State Literacy Resource Center as part of the LINCS Assessment Special Collection.