Assessing Oral Communication at the Community Learning Center: Development of the OPT (Oral Proficiency Test)

Joanne Hartel and Mina Reddy

Why Create a New Assessment?

The impetus for developing a new form of oral assessment at the Community Learning Center (CLC), a large adult education center in Cambridge, Massachusetts, came from new federal and state requirements that began in the summer of 2000. Before then, we had been using a standard in-house procedure to assess speaking, listening, reading, and writing on intake. We also had curricula for each level and criteria for moving students up to the next level. We had developed a writing sample administered under standard conditions and scored using a rubric. At the end of each semester teachers held individual conferences with students to discuss their progress. However, there was no program-wide oral assessment. Teachers created their own in-class processes to assess speaking and listening or, more often, based their evaluations entirely on classroom observation. SPL (student performance levels) levels, required by the state for reporting purposes, were assigned based on the classes students were placed in.

Given the increased emphasis on accountability and the need for standardized assessment procedures, we realized that this would no longer be sufficient. We considered using the BEST test, the most common off-the-shelf, standardized oral assessment. We liked the idea of a picture-based test that could be administered in a conversational, informal way. However, the BEST test was not a good match with our ESOL core curriculum, which had recently been revised based on the Massachusetts Curriculum Frameworks. Since we did not find any existing tests that matched our curriculum well, we decided to develop our own assessment of students' oral communication. The assessment needed to match our curriculum, provide information for placement and advancement, yield an SPL level for accountability purposes, and work for beginning, intermediate, and advanced students. We also decided to create alternate forms of the assessment so that it could be given up to three times a year, and to design something that would be easy to administer and score. Finally, we wanted the actual administration of the assessment to take no more than 10 minutes because we planned to administer it individually and we did not have the resources to give a longer test to our entire ESOL population. We realized that satisfying all of these criteria in one assessment would be no easy task.

Description of the Assessment

Each form of the Oral Proficiency Test (OPT) consists of a line drawing with six questions. Three questions involve describing what is in the picture. The last three questions pertain to the student's own experience. One question is intended to prompt a past tense answer, and another a response with a modal. The test assesses comprehension of the questions, the use of certain grammar forms, vocabulary, syntax, fluency, and pronunciation. (See the sample picture and questions.)

The OPT is administered by a trained tester (a teacher or counselor in the program) who is not the student's own teacher. The tester begins by introducing him or herself and meeting the student. He/she then says, "This is a very short test. It's for listening and speaking. It's only one measure of your progress in learning English. There are many things you and your teacher will talk about. I'm going to show you a picture and I'm going to ask you six questions about the picture. Please give me big answers. I'm going to write down the things you tell me so that I can remember what you said." He/she then briefly introduces the picture and proceeds to the questions. The questions can be repeated once if the student wishes, but without changing the wording. As the student answers the questions, the tester writes down what the student says or takes some notes if the response is very fast and long. The tester also makes a symbol to indicate whether the question was repeated. Once the test is over, the tester says, "Thank you very much. It was a pleasure to talk to you," and adds some words of encouragement. The test is scored immediately based on the guidelines (see appendix). There is a range of scores for each answer, depending on the accuracy and completeness of the student's response. There are also holistic scores for pronunciation and fluency. The scores are totaled, and an SPL is assigned and entered on the Department of Education database for accountability purposes. The scored rubric with the tester's notes and/or transcription of the student's answers is given to the teacher to use when conferencing with the student. It is one among several factors to be considered when deciding whether to move a student into the next class.
The others include classroom performance, homework, attendance,
and the writing sample.

Designing the Assessment

We decided to base the assessment on a conversation about a picture with the aim of making the language as natural as possible. We started with six pictures drawn for us by Joann Wheeler, an artist and former Community Learning Center teacher under the direction of JoAnne Hartel, who also made up the first draft of the questions. Each picture was used for a different form of the test. JoAnne started with the CLC's ESOL curriculum, using topics and vocabulary from the beginning and intermediate levels. The questions were designed to elicit simple sentences.

During the summer of 2000, the new oral assessment was piloted with students in several CLC classes and with new students on intake. At the same time, the BEST test was given to students in two classes for comparison purposes. Beyond the beginning level, the BEST proved to be very unsatisfactory for our students. The scores did not seem to reflect oral ability, particularly with more educated students.

Our pilot worked well enough to reassure us that we were on the right track. We continued administering the test to incoming students in the fall, and the ESOL teachers and counselors gave feedback on it. JoAnne trained and worked with a team of eight teachers to administer the OPT to every student in ESOL levels 1 to 4 in January 2001, at the end of the semester. Training involved discussion of the scoring criteria and practice scoring to make sure the results were as reliable as possible.

After all students were tested in January, the testing team met again to revise the questions and scoring criteria. They chose the three pictures that worked best and asked for some modifications of the pictures (e.g. "Make the woman in the clinic look more pregnant").

JoAnne and Mina sat down with lists of all the students by class and looked at their OPT scores and their class levels based on the judgment of their teachers. We recalibrated the scoring so that these matched more closely and served to discriminate better between students at different levels. The original scoring seemed to work less well at the upper level, so we adjusted it accordingly.

Evaluation of the assessment

We feel confident that the results of the OPT are, in most cases, a true reflection of students' oral communication ability. The new assessment has a number of advantages. It is a standard procedure for all students, administered by a few trained testers, so the results are more comparable than those that would come from individual teachers each using their own methods. In a program with many part-time ESOL teachers who may not have had an opportunity to teach more than one level, as is the case in many ABE programs, making judgments can be difficult. This also helps students feel that there are clear criteria for advancement. The OPT is quick, and it yields a numerical score. Raw scores can indicate improvements within an SPL level. It is based on the grammar and content in the curriculum.

Although it is a test, it feels close to a natural conversation and does not cause students to feel intimidated. This is particularly true for those who have been given the test more than once and are familiar with the process. There are three forms of the test available.

According to one CLC ESOL teacher, "It's useful to see how and how much the students can express with someone other than the teacher. Sometimes they can do more. It reminds us that our students need to communicate with other people in a different context. It's more realistic than the classroom."

One of the ESOL counselors said that the OPT is another tool that combined with everything else we use, gives us
a clearer picture of the correct ESOL placement level. It gives a better idea of students' grammar skills and sentence structure. And after doing a second round of OPT testing with the same group, the counselor noticed an improvement in the area of conversation. Also, the intake process is more complete now. On some occasions, when it is difficult to make a correct class placement, the OPT has been a key factor in placing students in class.

However, like any point-in-time assessment, the results can vary depending on how the person is feeling that day. Some ESOL students made mistakes, not because of their English, but because they misunderstood the intent of the line drawing they were looking at. Photographs might help to solve this problem. Although it is short, it is time intensive because it has to be administered individually. It was not scientifically designed. We had some initial discussions about designing procedures for assessing the reliability of the OPT, e.g. administering two forms of the test to the same person and seeing how closely the scores matched (to see how comparable different forms were), taping the test and having two testers score it (to check inter-rater reliability), etc. However, before investing the time needed for these efforts, we have decided to wait for the Massachusetts Department of Education to make some decisions about assessment and accountability. We have also continued to make small revisions in the questions and are collecting data on student scores that will help us to evaluate the effectiveness of the procedure in the future.

The Community Learning Center is a large adult basic education center located in Cambridge, Massachusetts. It serves 1000-1200 students each year, over 60% of them in ESOL classes. Students come from between 60 and 80 countries. Most attend class 5 to 6 hours per week. The majority are working.

Funds for the development and initial administration of the OPT came from the City of Cambridge and the Massachusetts Department of Education.

JoAnne Hartel is a teacher and curriculum and staff development
coordinator at the Community Learning Center. Until recently, Mina Reddy was the director of the CLC.

CLC Oral Proficiency Test
Drawing | Questions | Scoring

Originally published in Adventures in Assessment, Volume 14 (Spring 2002), SABES/World Education, Boston, MA, Copyright 2002.

Funding support for the publication of this document on the Web provided in part by the Ohio State Literacy Resource Center as part of the LINCS Assessment Special Collection.