Assessing Oral Communication at the Community Learning Center:
Development of the OPT (Oral Proficiency Test)
Joanne Hartel and Mina Reddy
Why Create a New Assessment?
The impetus for developing a new
form of oral assessment at the Community Learning Center (CLC),
a large adult education center in Cambridge, Massachusetts, came
from new federal and state requirements that began in the summer
of 2000. Before then, we had been using a standard in-house procedure
to assess speaking, listening, reading, and writing on intake. We
also had curricula for each level and criteria for moving students
up to the next level. We had developed a writing sample administered
under standard conditions and scored using a rubric. At the end
of each semester teachers held individual conferences with students
to discuss their progress. However, there was no program-wide oral
assessment. Teachers created their own in-class processes to assess
speaking and listening or, more often, based their evaluations entirely
on classroom observation. SPL (student performance levels) levels,
required by the state for reporting purposes, were assigned based
on the classes students were placed in.
Given the increased emphasis on accountability and the need for
standardized assessment procedures, we realized that this would
no longer be sufficient. We considered using the BEST test, the
most common off-the-shelf, standardized oral assessment. We liked
the idea of a picture-based test that could be administered in a
conversational, informal way. However, the BEST test was not a good
match with our ESOL core curriculum, which had recently been revised
based on the Massachusetts Curriculum Frameworks. Since we did not
find any existing tests that matched our curriculum well, we decided
to develop our own assessment of students' oral communication. The
assessment needed to match our curriculum, provide information for
placement and advancement, yield an SPL level for accountability
purposes, and work for beginning, intermediate, and advanced students.
We also decided to create alternate forms of the assessment so that
it could be given up to three times a year, and to design something
that would be easy to administer and score. Finally, we wanted the
actual administration of the assessment to take no more than 10
minutes because we planned to administer it individually and we
did not have the resources to give a longer test to our entire ESOL
population. We realized that satisfying all of these criteria in
one assessment would be no easy task.
Description of the Assessment
Each form of the Oral Proficiency Test (OPT) consists of a line
drawing with six questions. Three
questions involve describing what is in the picture. The last three
questions pertain to the student's own experience. One question
is intended to prompt a past tense answer, and another a response
with a modal. The test assesses comprehension of the questions,
the use of certain grammar forms, vocabulary, syntax, fluency, and
pronunciation. (See the sample picture and questions.)
The OPT is administered by a trained tester (a teacher or counselor
in the program) who is not the student's own teacher. The tester
begins by introducing him or herself and meeting the student. He/she
then says, "This is a very short test. It's for listening and
speaking. It's only one measure of your progress in learning English.
There are many things you and your teacher will talk about. I'm
going to show you a picture and I'm going to ask you six questions
about the picture. Please give me big answers. I'm going to write
down the things you tell me so that I can remember what you said."
He/she then briefly introduces the picture and proceeds to the questions.
The questions can be repeated once if the student wishes, but without
changing the wording. As the student answers the questions, the
tester writes down what the student says or takes some notes if
the response is very fast and long. The tester also makes a symbol
to indicate whether the question was repeated. Once the test is
over, the tester says, "Thank you very much. It was a pleasure
to talk to you," and adds some words of encouragement. The
test is scored immediately based on the guidelines
(see appendix). There is a range of scores for each answer, depending
on the accuracy and completeness of the student's response. There
are also holistic scores for pronunciation and fluency. The scores
are totaled, and an SPL is assigned and entered on the Department
of Education database for accountability purposes. The scored rubric
with the tester's notes and/or transcription of the student's answers
is given to the teacher to use when conferencing with the student.
It is one among several factors to be considered when deciding whether
to move a student into the next class.
The others include classroom performance, homework, attendance,
and the writing sample.
Designing the Assessment
We decided to base the assessment on a conversation about a picture
with the aim of making the language as natural as possible. We started
with six pictures drawn for us by Joann Wheeler, an artist and former
Community Learning Center teacher under the direction of JoAnne
Hartel, who also made up the first draft of the questions. Each
picture was used for a different form of the test. JoAnne started
with the CLC's ESOL curriculum, using topics and vocabulary from
the beginning and intermediate levels. The questions were designed
to elicit simple sentences.
During the summer of 2000, the new oral assessment was piloted
with students in several CLC classes and with new students on intake.
At the same time, the BEST test was given to students in two classes
for comparison purposes. Beyond the beginning level, the BEST proved
to be very unsatisfactory for our students. The scores did not seem
to reflect oral ability, particularly with more educated students.
Our pilot worked well enough to reassure us that we were on the
right track. We continued administering the test to incoming students
in the fall, and the ESOL teachers and counselors gave feedback
on it. JoAnne trained and worked with a team of eight teachers to
administer the OPT to every student in ESOL levels 1 to 4 in January
2001, at the end of the semester. Training involved discussion of
the scoring criteria and practice scoring to make sure the results
were as reliable as possible.
After all students were tested in January, the testing team met
again to revise the questions and scoring criteria. They chose the
three pictures that worked best and asked for some modifications
of the pictures (e.g. "Make the woman in the clinic look more
JoAnne and Mina sat down with lists of all the students by class
and looked at their OPT scores and their class levels based on the
judgment of their teachers. We recalibrated the scoring so that
these matched more closely and served to discriminate better between
students at different levels. The original scoring seemed to work
less well at the upper level, so we adjusted it accordingly.
Evaluation of the assessment
We feel confident that the results of the OPT are, in most cases,
a true reflection of students' oral communication ability. The new
assessment has a number of advantages. It is a standard procedure
for all students, administered by a few trained testers, so the
results are more comparable than those that would come from individual
teachers each using their own methods. In a program with many part-time
ESOL teachers who may not have had an opportunity to teach more
than one level, as is the case in many ABE programs, making judgments
can be difficult. This also helps students feel that there are clear
criteria for advancement. The OPT is quick, and it yields a numerical
score. Raw scores can indicate improvements within an SPL level.
It is based on the grammar and content in the curriculum.
Although it is a test, it feels close to a natural conversation
and does not cause students to feel intimidated. This is particularly
true for those who have been given the test more than once and are
familiar with the process. There are three forms of the test available.
According to one CLC ESOL teacher, "It's useful to see how
and how much the students can express with someone other than the
teacher. Sometimes they can do more. It reminds us that our students
need to communicate with other people in a different context. It's
more realistic than the classroom."
One of the ESOL counselors said that the OPT is another tool that
combined with everything else we use, gives us
a clearer picture of the correct ESOL placement level. It gives
a better idea of students' grammar skills and sentence structure.
And after doing a second round of OPT testing with the same group,
the counselor noticed an improvement in the area of conversation.
Also, the intake process is more complete now. On some occasions,
when it is difficult to make a correct class placement, the OPT
has been a key factor in placing students in class.
However, like any point-in-time assessment, the results can vary
depending on how the person is feeling that day. Some ESOL students
made mistakes, not because of their English, but because they misunderstood
the intent of the line drawing they were looking at. Photographs
might help to solve this problem. Although it is short, it is time
intensive because it has to be administered individually. It was
not scientifically designed. We had some initial discussions about
designing procedures for assessing the reliability of the OPT, e.g.
administering two forms of the test to the same person and seeing
how closely the scores matched (to see how comparable different
forms were), taping the test and having two testers score it (to
check inter-rater reliability), etc. However, before investing the
time needed for these efforts, we have decided to wait for the Massachusetts
Department of Education to make some decisions about assessment
and accountability. We have also continued to make small revisions
in the questions and are collecting data on student scores that
will help us to evaluate the effectiveness of the procedure in the
The Community Learning Center is a large adult basic education
center located in Cambridge, Massachusetts. It serves 1000-1200
students each year, over 60% of them in ESOL classes. Students come
from between 60 and 80 countries. Most attend class 5 to 6 hours
per week. The majority are working.
Funds for the development and initial administration of the
OPT came from the City of Cambridge and the Massachusetts Department
JoAnne Hartel is a teacher and curriculum and staff development
coordinator at the Community Learning Center. Until recently, Mina
Reddy was the director of the CLC.
CLC Oral Proficiency Test
Drawing | Questions
Originally published in Adventures in Assessment,
Volume 14 (Spring 2002), SABES/World Education, Boston, MA, Copyright
Funding support for the publication of this document
on the Web provided in part by the Ohio State Literacy Resource
Center as part of the LINCS
Assessment Special Collection.