So What IS a BROVI Anyway? And how it can change your (assessing) life?

Betty Stone and Vicki Halal

Why did we develop our own
assessments?

Through the years at SCALE (Somerville Center for Adult Learning Experiences), ESOL assessment had continually been a source of debate and concern. Frankly, the topic drove people crazy. What kinds of assessment were teachers using to support their "feelings" that students were ready to be promoted? Did the assessment results match teachers' intuitions? How similarly did different teachers rate the abilities of the same students? Why couldn't we agree? Staff often raised the issue of assessment in terms of documentation needed to support level change recommendations. We long sought easily administered and appropriate assessment tools that would more consistently measure all language areas (reading, writing, listening, speaking) across all program levels. More recently, this coincided with the state and national movement toward "reliable and valid" assessment, as well as the Massachusetts Department of Education (DOE) requirement to report learner progress according to Student Performance Levels (SPLs).

From instructor to instructor ongoing assessment style and content varied widely. While such a range of assessment strategies would not affect the appropriateness of particular assessments within a class, as a program we lacked consistency and the level recommendation process could sometimes become murky. Counselors were sometimes called upon to mediate lively testimonials between a teacher who wanted to promote "M " and a second teacher who refused to accept her. Without a program-wide assessment tool, we could not easily come to a consensus on when learners were prepared to move ahead. Since our primary objective is to help students realize their fullest potential, we felt that perhaps we would serve them better by having at least one method of assessment that all staff would utilize. We hoped that would help us more clearly identify students' strengths and weaknesses over time and, therefore, keep better track of their needs as they proceeded through the program. A consistent assessment protocol would also make clear to the students the expectations of the program at each level.

We had at times considered adopting published assessment materials as well as possibly instituting a formalized portfolio system. The popular ESOL assessment tools were ill-suited to our population. Maintaining an elaborate portfolio assessment system for over 300 learners at two sites was not realistic for a primarily part-time staff. A mini-grant from the Adult Literacy Resource Institute (ALRI) gave us an opportunity to create our own assessment package. As the project flourished, we realized that the tools not only served SCALE's program well, but could be replicated in other programs, and might even provide MA DOE with an example of effective alternative assessment for ESOL.

Who was involved in developing the assessments?

SCALE was fortunate to have received an ALRI mini-grant for two of its part-time ESOL instructors, Laura Brooks and Vicki Halal, to coordinate research, development, and implementation of the initial assessment package. Program Administrators, Betty Stone and Ngaio Schiff, also contributed time and expertise to the project. Additionally, this small team involved the entire staff by soliciting their ideas and feedback through surveys, staff meetings, a pilot round of assessment, and trainings. Some members of the staff chose to dedicate a portion of their staff development hours to the project as well. The team also surveyed groups of students (one from each class) at the outset about their ideas on assessment. This comprehensive collaboration created a great deal of intellectual and practical momentum during the process, and insured that everyone was invested in the project and all voices were heard from the very beginning.

What kind of assessment did we develop?

SCALE's current assessment package consists of two tools, the Writing Sample that measures writing skills, and the BROVI, which has two forms to measure oral/aural skills, an individual speech and a role-play. What, you ask, does BROVI mean? It is, as you may have suspected, an acronym made from the names of the developers that takes the place of "the listening/speaking assessment," a phrase that tripped up all our efforts to exchange ideas about that developing tool.

Assessments are routinely administered program-wide three times per academic year to track students' progress through SCALE's internal levels as well as to satisfy the reporting requirements of the Massachusetts (DOE). The program designates two-week periods in October, January, and May for assessment, and accommodates students who enter SCALE classes at other times with special assessment arrangements.

Our Writing Sample consists of the
following:

• "To the Student" Instructions
• Administration Instructions to teachers
• Master Writing Sample sheets for each topic: lined sheets headed by the topic or a picture (Teacher selects a single topic from the Master List for the class.)
• A scoring rubric for teachers

Typically, teachers set aside approximately 40 minutes of class time: 10 minutes for explanation of the purpose and instructions, and 30 minutes for the class to complete it. Once the Samples are collected, instructors score them outside of class according to the rubric and are compensated based on the number of samples per class.

The BROVI incorporates the option of an individual Speech or a paired Role Play activity. The teacher selects one of the options and administers it to the entire class on the designated assessment day(s). The components of the BROVI are

• To the Student' Instructions (see appendix)

• Administration Instructions to teachers

• Speech Topics (Master List includes choice of three per level; teacher selects one for all) (see appendix)

• Role Plays for literacy to high beginner levels (laminated photo cards with scenario descriptions printed on the reverse side)

• Role Plays for intermediate
  to advanced levels (laminated
  scenario cards)

• Audience listening activity worksheet (see appendix)

• A scoring rubric for teachers (see appendix)

The amount of class time needed to complete the BROVI will vary according to the option chosen as well as the class size. In both cases, the teacher reviews with the students the purpose of assessment and the instructions. The speech topic or role-play scenarios are distributed to learners who work for 10-15 minutes to prepare (speech topics are discussed in groups of 3-4; role-plays are prepared in pairs). As each person or pair bravely performs the speech or role-play without notes, the rest of the class are filling in their "Audience Listening Activity," preparing questions for those giving speeches or answering questions about the role plays, and the instructor is completing the rubric. All BROVI scoring is done during class time.

Both the Writing Sample and the BROVI are given raw scores from 0-76, which correlate to SPLs 0-8, the range of ability among SCALE ESOL students. The components of each rubric and their weights (Note the x2, x3, x4, x5) represent the relative importance of those aspects of language within our program.

What was the hardest part of the development process?

From the beginning, we were aware that we would face a number of challenges. First, we needed a set of user-friendly tools for students and instructors to use during class time. Most of our staff is part-time and limited financial resources for extra paid staff time mandated that the bulk of the assessment work take place within the framework of class hours. We succeeded in raising supplementary grant funds to provide the necessary training for all staff, but the core assessment responsibilities and ongoing feedback on our model fit within expected expenditures for teaching, meeting, and staff and program development time.

Next, we wanted to ensure that our scoring system would reflect meaningful progress through internal program levels and be correlated to the SPL system. As previously stated, we wanted to avoid the standardization that might limit the possibilities of student performance. We chose to develop our performance-based assessments so that we could offer students opportunities to use their English to their fullest abilities. The challenge in scoring was to be able to give credit (and points) for the complete range of proficiency levels that exist in our ESOL levels. In this way, we wanted the instruments to reflect the patterns of progress within our entire program. By creating a weighted system of scoring the various components of writing, speaking, and listening, the raw score range covers 0-76 and correlates with SPLs 0-8. Following the second full round of assessment (May 2001) we had a large enough number of raw scores to re-adjust the raw score/SPL correlation based on how actual students scored at each internal SCALE level.

All through the development process, we tried to keep our staff involved. With each step, we asked for feedback and suggestions for change. We needed their input to improve content and administration of the Writing Sample/BROVI. By being involved during the development process, we hoped the staff would feel more confident in utilizing the resultant tools. Without the participation of the entire staff in both development and implementation, this assessment package would be compromised. Of course, the more we asked, the more feedback we needed to incorporate. Initially, the pages of notes seemed daunting; however, once we began to sort through them and incorporate their suggestions, we found we appreciated the input even more. Staff involvement in the entire process made us feel more confident in the final product and helped avoid the feeling that the final tools would be an imposition on teachers or students.

Once the Writing Sample/BROVI was ready to be used, the issue arose
of training and compensating staff fairly. Even though staff were familiar with the package through its development, the implementation approaches still varied from teacher to teacher. Additionally, scoring could be rather subjective, so there needed to be consensus in order to have "reliable" and "valid" assessment. SCALE offered four sessions of paid program development dedicated to staff training that allowed instructors, counselors, and administrators to discuss and fine tune the administration and scoring procedures involved in the assessment package. After the initial pilot of the Writing Sample/BROVI, we were able to verify the number of hours generally needed to score the Writing Sample, and, we developed a pay scale accordingly. Instructors are allotted a certain number of hours based on their class size and are paid for them at their regular hourly rate. Facing and overcoming the many hurdles inherent in this project led us to develop what we feel is a user-friendly, meaningful, and fair assessment package.

Top of page

Self-assessment of our assessment

We have been pleased and encouraged that both the BROVI and Writing Sample assessments have gotten high marks from the ESOL teaching staff at SCALE, as well as from the ESOL program administrators. Practitioners particularly like the following features:

Strengths:

The assessment tasks are performanced-based and learner-centered. They are related to the learners' goals of communicating more effectively in English and/or improving writing skills. Teachers report that students have fun preparing and performing the role-plays and learners enjoy hearing each others' "speeches" and asking follow-up questions. The BROVI and writing sample topic selections offer a reasonable degree of choice and allow students to display their language ability, though we continue to refine the master lists in response to teacher feedback. While the exact topics for the "official" assessments are considered "secure," teachers are encouraged to practice role-plays and sustained speaking activities as part of their usual classroom routine. The bottom line is that the assessment tasks themselves are representative of activities in an interactive ESOL classroom. These are not strange, threatening, or irrelevant tasks that suddenly invade the classroom; rather they are natural language learning activities that are easily integrated into curriculum design. The "To the Student" handouts keep the "test stress level" among students in check. Teachers are listening or reading for what students know, not what they don't.

Materials are well "packaged" and easy to use. Administration guidelines and directions are standardized, clear, and easily accessible. Assessment protocol, pay for related work (scoring writing samples), and timelines are unambiguous. This is particularly significant at SCALE where the ESOL teaching staff is primarily part-time. Special student handouts make an effort to demystify the assessment process to the students. We want the learners to know what we are asking them to do, why we do it several times each year, and what we expect of them. Assessment drawers contain classroom packets of Writing Sample master sheets, BROVI and Writing Sample rubrics, laminated beginning level role-play photo scenario cards, intermediate/advanced level role-play scenario cards, BROVI Speech Topics, and "To the Student" handouts to assist learners with understanding the purpose and expectations for each of the three assessments (BROVI speech, BROVI role-plays, and Writing Sample). January 2002 marked the fourth and the smoothest administration round of these assessments at SCALE. Teachers and students are beginning to take the process in stride.

We have achieved a uniformity and consistency of assessment conditions with these tools that had never before existed across the range of classes in our program. Though we needed to invest in a second round of intensive training in January 2002, to orient new teachers and reinforce scoring practices of veteran teachers, consistent scoring of BROVIs and writing samples is improving. Inter-rater reliability among assessors is key in a program such as ours, where 19 instructors teach and assess five core ESOL levels and three ESOL literacy levels, representing the range from SPL - to SPL 8.

As a program, we are beginning to witness predictable patterns of progress as we track learners through various classes. Assessment results for a sample student who has repeated ESOL 1 two times and then is promoted to ESOL 2, come from three different teachers in the three distinct classes. (ESOL 1, ESOL 1, ESOL 2) Raw scores and correlated SPLs over the student's career at SCALE show little improvement or sometimes some slide-back initially. Ultimately, however, sufficient raw score (and SPL) increases indicate the student's readiness to advance to the next SCALE class level.

The rubrics are clear, specific, and easy to use. They have seen numerous iterations, always in response to teacher feedback, and always with the goal of facilitating the process of capturing learner performance in a fair, accurate, and streamlined fashion. Teachers have commented that using the rubrics has been helpful in sharpening their diagnostic skills in general. They are regularly reminded of the objective criteria the program uses to rate a learner's competence. Good attendance, cheerful attitude, and social connection to the class are not on the rubrics. While those may be characteristics of many of our successful learners, they are not the components the BROVI and the Writing Sample are designed to assess. On the reverse side of the rubrics, teachers have the opportunity to add anecdotal comments on an atypically poor or outstanding BROVI due to extenuating circumstances. Each rubric entry stands as documentation of a learner's performance on a specific task at a given moment. The rubrics are designed to capture the initial, ongoing and final assessment history for a student on a single page. Because each student has a BROVI and a Writing Sample rubric for the year, it is convenient to see, at a glance, how she is progressing over time.

The BROVI and the Writing Sample are significant, but not the only criteria for promotion. As the time for level change recommendations approaches, teachers consider BROVI and Writing Sample assessment results, classroom participation, homework, attendance, and other informal assessment they have made of each student, as they weigh a student's readiness for the next level. The official assessment record is just one bit of data, one piece of the puzzle to consider in the level recommendation process. It is an aid, not a straightjacket. Clear agreement on this point has freed staff up to complete the assessments as honestly and consistently as possible, and to continue to consider how to improve our assessment tools.

Top of page

Limitations:

No matter what, there is always some subjectivity in evaluating language proficiency. Efforts to quantify the components of effective oral and written communication are elusive. Describing fluency, richness of expression, and grammatical accuracy in a speaking activity with a numerical score will always be part art. Satisfactory levels of inter-rater reliability can be achieved, but intensive training, which is costly in time and money, is still required.

Role-plays are dependent on the strength of one's partner. As in sports, if you "play" with a partner who has equal or better skills than you, you will "play up." It is often difficult, though, to maintain your own level of skill when you "play down" with someone who is less skilled. For this reason, the BROVI guidelines require that initial assessments always use the speech option. By the time a class gets to the ongoing assessment, students know each other well enough and have improved their skills sufficiently to manage role-plays. Furthermore, the teacher is better able to pair students effectively for role-plays.

The listening comprehension aspect in the Speech format is limited. It is weighted less because it is dependent only on the 1-3 questions that the audience asks the speaker. The concept of the listening activity handouts was included thanks to the persistent enthusiasm of Tim Laux, a part-time member of the original assessment team.

The Writing Sample is useable, but not ideal for low literacy students. It tends to show what they cannot do, rather than what they can do.

Feedback to students is limited. The protocol now calls for teachers to encourage the class in general terms following assessment, but to avoid "reviewing the assessment with individual students." The rationale is based in preserving the official assessment as an assessment, not an instructional activity. This is an area where we are divided on how much more time we might give to one-on-one feedback.

Each teacher assesses her own class. Each task is assessed only once. Ideally, teams of teachers would assess a class batch of writing samples to guarantee accurate scoring. Teachers might also be more objective if they assessed another's class on the BROVI (though the students would likely perform worse for an unfamiliar teacher.) The cost of multiple assessors is prohibitive and the logistics of swapping classes for BROVIs is too unruly. We acknowledge, however, the energetic team spirit that surfaces during group trainings and the benefit of sharing scoring tips and rationales.

What's next?

One of the advantages of alternative assessment tools such as the BROVI and SCALE Writing Sample is that the program has full control to reflect on their usefulness, identify priority points for modification, and incorporate improvements. Not only do teachers regularly ask "what if..."questions about administration, but they also suggest new topics for both the oral/aural and written tools. We have refined the rubrics several times in minor ways to make them easier to use and clearer to read. We have tweaked the scoring correlation and likely will make one last adjustment at the end of the current year. We continue to train together to share strategies on how to listen to or read the same samples and hear or see similar strengths and weaknesses. Our aim is to become reliable within a few (raw score) points, such that the SPL correlation will generally be the same. We anticipate that training will be an annual event, to sharpen the skills of veteran teachers and to orient new staff to the fine points of our tools.

We would love to revisit our student focus groups, especially to collect ideas from learners who have been through several rounds of assessment. Follow-up focus groups would give us valuable information: Have we succeeded in making the assessment process "friendly" for the students? Do they see the connection between assessment, curriculum, and class activities? Does regular assessment help the learners understand their own learning curves? Are there changes the students would recommend? Are there essential elements in assessment from the student perspective that we have overlooked?

Like many adult education programs, we are struggling to abide by the regulations our funders have required, while we design and implement an assessment system that is integrated with our program. Teachers, counselors, administrators, and students are learning to understand the role of this type of assessment and to assess fairly and honestly, without fearing for the program if there are occasional backslides in learner scores, or outrageously low scores for students with performance anxiety on the day of assessment. Assessment contributes to level promotion criteria, informs curriculum design, and represents part of the picture of the success of our programs and of our field. We need to remember, however, that assessment is still only a snapshot of how a learner is doing at a particular moment on a particular day. We load the film, prepare the subjects, focus as best we can, shoot, and hope for the best.

Top of page

Betty Stone earned her M.A.T. in French and ESL from the School for International Training in Vermont. She has been ESOL Program Administrator at the Somerville Center for Adult Learning Experiences, SCALE, since 1978, and keeps her hand in teaching through subbing, occasional guest teacher appearances, and sharing teaching, learning, and assessment ideas with the great staff at SCALE and around the Boston area.

Vicki Halal has been an instructor in the adult education field since1993, after receiving an MA in ESL Studies from UMass/Boston. She has worked mainly at UMass and SCALE teaching an array of ESOL learners and levels. Currently, she is working at SCALE in the ESOL and GED departments.

TO THE STUDENT: The Speech

• Why? You do a speaking/listening activity 2 or 3 times a year to see what you know and how much you have learned. This activity shows us one way you can use the English you are learning.

• What are we looking for?

  We are looking for strong speaking:

  • speak for a few minutes about 1 subject

  • use good grammar

  • speak clearly with good pronunciation

Read the Speech instructions below. then turn this paper over and look at the "remember" hints.

SPEECH INSTRUCTIONS

• Your teacher will give you a topic to speak about for 1 or 2 minutes.

• In a small group, share iformation about the topic. Talk about what you know about it.

• After you prepare, tell the class about the topic.

• The other students are listeningh and writing on the paper ("Audience Listening Activity").

• Answer questions that your classmates ask.

Remember Hints

Remember, to speak well, you:

1. Speak for enough time to give the information

2. Stay on the topic

3. Use good grammar structure

4. Use all the vocabulary you know for that subject

5. Speak clearly

Remember, when you are listening, you:

1. Do not talk.

2. Try to understand the people who are talking.

3. Pay attention to the people who are talking.

4. Write anything you want to remember.

BROVI SPEECH TOPICS

ESOL 1/A/1Lit

1. Describe your job. Do you work in your home or outside? Do you work alone? What kind of work do you do?

2. Describe a special place in your country. Why is it important to you? Why do you go there?

3. Describe your favorite (living) relative. Who is she/he? What do you do together? Why do you like her/him?

ESOL 2/Basic Skills

1. Compare the weather in your country and in the United States. How are they the same or different? Which weather do you like better (prefer)?

2. Describe an ideal job you would like to have or a great job you had in the past. Describe the job and the working conditions. What is/was your favorite part of the job? Why?

3. Describe the life of older people in your country. Where do they live? Are older people in your country happy?

ESOL 3/B/ESOL Intermediate R/W

1. Describe your first trip away from your home in your birth country.

2. Describe when and why you miss a typical food from your country
so much.

3. Describe a valuable lesson you learned in life when you were younger.

ESOL 4/C

1. Explain how you get news about your country (now that you live in the United States.)

2. Describe the reaction you had the first time you ever used a computer.

3. Begin your speech with the words: "Let me tell a story about an
accomplishment that makes me feel very proud."

ESOL 5/D/ESOL R/W

1. Describe a piece of excellent advice you once gave a friend or family member.

2. Describe how you will continue to learn after you leave SCALE.

3. Describe the advantages and disadvantages of being an immigrant in the Massachusetts.

Audience Listening Form-Conversations

Name________________________ Class__________ Date______

Scoring Rubric for BROVI Assessment

Student___________________________ Program Year: 2001-2002

Circle R (role-play) or S (speech) below for each assessment. S only for INTIAL.

Scoring Key

Top of page

Originally published in Adventures in Assessment, Volume 14 (Spring 2002), SABES/World Education, Boston, MA, Copyright 2002.

Funding support for the publication of this document on the Web provided in part by the Ohio State Literacy Resource Center as part of the LINCS Assessment Special Collection.