Working with Industry
Authentic Assessment in the Workplace

Debbie Tuler
Jewish Vocational Service
Boston, MA

C&K Components is a switch manufacturing company based in Newton, MA. The company has had an ESL program for approximately five years. For an 18 month period from October, 1992 through December, 1993 (three 20-week cycles) the program was funded by a federal grant. The program always included two classes running simultaneously, although for a 10-week period in the fall of 1993, four classes ran simultaneously. I was the Site Coordinator/Instructor at C&K during the 18 month grant period, and for an additional 20-week cycle after the grant period ended, teaching two classes each cycle. Another instructor (Elinor Pitkin) taught the additional two classes in 1993; these classes were added because there was money left in the budget. It was discovered though that production could not handle having that many workers off the line at the same time, so after that we returned to offering only two classes. Most of the participants in the program work as assemblers and machine operators, although there have also been some stockroom clerks and line and unit leads.

We (the instructors and Workplace Education Coordinator at JVS) used an initial assessment tool for the purpose of placing students in one class or the other based on their abilities and levels. For the first cycle of classes, the levels were predetermined based on the needs and wants of management: one class was for those who held semi-supervisory positions (line and unit leads), assumed to be advanced. The other was for beginners. Management wanted to help those in semi-supervisory positions, in particular, those individuals they felt would be able to offer something to the company. In particular management hoped that by helping leads learn work-related topics, the leads in turn would be able to train their line workers. For example, management wanted the instructor to teach leads how to read and understand work orders so that they could turn around and teach this skill to others.

Management had the same goal for assemblers in the lower level class: that they would learn work-related skills and then teach them to their co-workers on the line. They also wanted to groom those beginners they felt had a chance of advancing to lead positions. They therefore hand-picked “strong” employees for us to assess and place in class.

The original assessment tool we used at the start of the grant period consisted of two components: an oral interview and a writing sample. The oral interview consisted of ten questions which attempted to elicit both factual and inference information, and the ability to use different tenses. The interview was scored by checking off whether the interviewee answered at all, answered appropriately or not, and whether the answer was complete and grammatical. Some interviewers also attempted to write down the interviewees’ responses, but others did not do so. The inference questions addressed a photograph, and then shifted from the photograph to the students’ own life dreams. The photograph was not a picture that had any connection to the lives of the employees; one of the first changes we made in the assessment was to use a picture of an assembly worker. Employees suddenly had much more to say to the interviewer.

The writing component gave employees a choice of two topics to write about, and asked them to write a paragraph. They were also asked (in writing) why they wanted to study English.

At the start of the program, we perceived the assessment test in very narrow terms, considering only the interview and writing components once potential students arrived in the room for testing. Over the course of several months (2-3 cycles), a number of issues arose which caused us to reconsider the initial assessment. These issues included the following:

1. Multi-level classes. Since we only tested a few more students than could be placed in classes, we were in the position of filling classes with whoever we tested, regardless of actual level of ability. While trying to keep the classes as homogeneous as possible, we still ended up with a wide range, especially at the lower levels, where we would have literacy and beginner/high beginner students in the same class. Part of the problem was that the test did not accurately reveal skills, particularly at the literacy and beginner levels. Both teachers and students were concerned about the level differences within classes. It is important to emphasize the students’ role in prompting us to review the assessment tool; they voiced strong concerns over the mixed level classes and over low motivation of some students. Responding to these concerns involved re-thinking the assessment. The other teacher also played an important role; coming in as an outsider for a short period of time, she was able to pinpoint issues and concerns which I had felt but not yet articulated.

2. Initial selection process. We began to question the company’s recruitment policy for a couple of reasons. We tried offering more classes (adding two in the fall of 1993), but management found that when we ran four classes, there were too many people off the line. Production could not handle this need in terms of numbers. We also wanted to test many more people than we could serve as a way to resolve the problem of the multi-level classes. However, management opposed this idea because they did not want to raise false expectations by testing employees when they could not serve them; management felt that open testing would result in too many people being tested and too much disappointment. We also felt that supervisors and managers were pre-selecting those employees they felt would further benefit the company through classes, and ignoring other, potentially successful, employees. It seemed that they were selecting those who were bound to do well. They were also pre-selecting based on their idea of who was ‘advanced’ and who was ‘beginner’, but their perceptions did not necessarily match skill levels reflected on an assessment test. Second, and related to the pre-selection process, was confusion and resentment on the part of employees over who was chosen to be in the program; some employees were able to participate more than once while others never even got tested even though they wanted to study.

3. Curriculum and materials development. We found that the original initial assessment did not provide instructors with the information they needed for determining what to teach, and at what level. The assessment did not provide students an opportunity to provide input into shaping the curriculum. Nor were instructors sure of the best way to approach the resulting multi-level classes.

As a result of the concerns of instructors and students, and with the intention of improving service to the company and students, during the third cycle of classes we (the instructors and coordinator) began to re-look at the assessment tool and ultimately re-design it. The first concern we needed to address was what the assessment tool was for in the first place: what information we wanted to get from the assessment, what skills we wanted to assess, who we were assessing for, and how that information would be used.

We decided that there were a number of reasons for assessment:
1. determine level of motivation and interest in classes
2. place in appropriate and homogeneous classes
3. establish and standardize proficiency levels
4. measure learner progress over time
5. track program effectiveness (by showing learner progress) (We have since re-thought this point because learner progress is not the only - and probably not the most important - measure of program effectiveness.)
6. ascertain and clarify needs and expectations of both participants and teacher.

We saw a need for some mechanism that went beyond a pen and paper test, that would ensure an ability to shape classroom instruction, and that would provide potential participants with enough knowledge to choose whether or not they wanted to take a class. The new assessment tool, intended to address these reasons or issues, consists of five components: an orientation for potential students, an interview, a reading section, a writing section, and a literacy assessment.

Top of page

The purpose of the orientation is to provide information to potential students about the ESL program, what is covered in classes, and what the expectations are for participants, and how people are selected. It offers potential students the opportunity to ask questions and express their expectations, needs, interest, and commitment. This latter point was particularly important to students who were committed. The orientation gives them a chance to choose whether to participate. In retrospect, we see that many people who were chosen by their supervisors may have felt obligated to attend the class, even if they did not want to. This lack of interest and motivation is reflected in lack of completion of the course, poor attendance, and/or poor attitude.

The interview was redesigned in two ways. First, we re-did the way responses are scored, to provide more accurate information. Second, we changed some of the questions so that they would provide more useful information regarding abilities and needs. Questions are familiar, in that they ask for personal factual information and about the interviewees own life goals.

Originally, the assessment did not include a reading component, but we decided to add one because we felt we needed information on general reading fluency, decoding skills, and reading comprehension. We decided to offer students a wide choice of reading materials, at different levels, and let them choose for themselves which one to read. In this way, they could show us what they found interesting or relevant. We found that students read better when they read something familiar, either in a familiar format (e.g. want ads, work memos), or a familiar topic (e.g. immigrant experiences, work stories). Reading selections included work-related materials, real-life materials, and selections from readers. We rated the readings for level, based on difficulty of the grammar, length of sentences, and vocabulary. Comprehension questions include yes/no, information, and inference questions; we also note students’ answers, pronunciation problems, and whether the selection seems too easy or difficult. The idea for organizing a reading component in this way came from an assessment developed at Quinsigamond Community College in Worcester, Mass.

The writing section also became more detailed. We found that with the original test, students often did not write because they could not think of anything to say, not because they lacked writing ability. We instruct the student to choose a topic, and if it appears that they are having difficulty, we provide them with a card with specific questions about the topic.

If it appears that the student cannot write at all (or if the students say that they cannot write), we use the literacy assessment. This was an important addition to the assessment tool because initially we had no way to evaluate the skills of someone who claimed to be unable to write, and many of the students at C&K are at literacy or very low levels. The literacy assessment provides information on letter and sound identification, numeracy, sight word recognition, and writing of personal information. If the student is able to decode the sight words well, we have them attempt the simplest level one reading selection.

The detailed information which this expanded assessment tool provides enables an instructor to look back at the test and evaluate skill levels. Guidelines for the assessment result in greater consistency in scoring among different testers. In this way, if the teacher is not the same person who gave the test, he or she is still able to compare tests and more accurately group people by level, as well as glean significant information for pre-planning and developing appropriate materials for classroom instruction.

Parts of the revised assessment tool were piloted for cycle 3 in the spring of 1994; specifically, we had the orientation, and we used the literacy and reading components. The interview and writing components had not been completely revised in time and were not piloted. We found that management was still reluctant to have open testing for all interested employees, but we were able to continue testing until we had filled two classes with fairly homogeneous skill levels. While the revised assessment takes longer, it provides more accurate information.

We found that the orientation clarified goals and expectations so that students were aware of what the program would offer. It brought to our attention what workers wanted the program to offer. Some workers who came to the orientation decided not to take the class. It also raised other issues, such as possibly needing to revise the program to address particular needs, offering specialized courses for students. This issue has not yet been addressed.

The literacy component provided much needed information at the lower levels and enabled us to better place students in classes and use appropriate materials.

At this time, we have neither used the complete test for initial assessment of students, nor have we used it to measure student progress or program effectiveness. We plan to pilot the complete assessment tool and see if it adequately addresses all five of our goals during the next cycle at C&K Components. The original assessment did not address student concerns or provide us with the results and information we wanted and needed. So far, it seems that the revised version makes more sense in terms of assessing abilities and informing instruction. However, we are still left with the apparent contradiction of needing to test sufficient number of workers so as to create homogeneous classes, but not testing so many that we raise expectations of people we cannot serve. Therefore, a question which remains for us is how to find a middle ground that will satisfy all parties.

Top of page

Adapted from literacy assessment by Jon Phillips, International Institute of Boston

This article was published in Adventures in Assessment, Volume 7 (December 1994), SABES/World Education, Boston, MA, Copyright 1994.

Funding support for the publication of this document on the Web provided in part by the Ohio State Literacy Resource Center as part of the LINCS Assessment Special Collection.