Partners In Evaluation: Evaluating the South Cove Manor Nursing Home Workplace Education Program with Participants

Johan Uvin

For me, one of the exciting aspects of being a Workplace ESL teacher at the South Cove Manor Nursing is that my job description is defined more broadly than in more trdditional educational settings. As a teacher, I am not only responsible for instruction but for coordination and curriculum development as well, with assessment and evaluation being an integral part of the latter. In this article, I take the excitement that goes hand in hand with the planning and implementation of assessment and evaluation for granted and focus mainly on the challenges that the selection and design of appropriate assessment and evaluation procedures and activities created for me and how I resolved them. It is my intent that teachers with similar role definitions and in similar contexts will find my experiences useful in their own assessment and evaluation practice.

Context

The South Cove Manor Nursing Home Workplace Education Program was established in 1988 with funds from the National Workplace Liter- acy Partnership Program. The goal is to assure the quality of patient care by upgrading the communication skills of nursing assistants and entry-level workers in other departments of this bilingual nursing home on the edge of Boston's Chinatown. To achieve this goal, the nursing home administration and management collaborate with the Chinese American Civic Association who act as the learning provider. An Advisory Board consisting of representatives of each of the workplace, ESL classes, supervisors, licensed staff, teachers and managers meets regularly to discuss the planning, evaluation, and monitoring of the program as a whole. The meetings of the Advisory Board provided the context for the evaluation work that I describe in the remainder of this article.

Challenges

The first challenge for me was to strike a balance between the different evaluation needs and preferences expressed by all those who had a vested interest in the program: learners, teachers, supervisors, administrators, managers, sponsors, and funders. It was also important not to ignore my own beliefs about assessment and evaluation. Other factors I needed take into account were my own skills in assessing and evaluating, the limited resources that could be allocated to assessment and evaluation, and the evaluation guidelines as specified in the federal grant.

Getting Started

As with other planning tasks, I accomplished the planning and implementation of assessment and evaluation as a collaborative effort of all those affected by the program. I chose the meetings of the Advisory Board as the forum to facilitate direct communication on the issue between learners, teachers, supervisors, department heads, administors, managers, board members, and funders. The members of the Advisory Board engaged in a two month dialogue early on in the prognun to put assessment and evaluation objectives and procedures into place.

Discussions evolved around these questions:

• What do we want to evaluate?
• What qualitative and quantitative information do we need?
• What information is already available at the nursing home?
• How are we going to collect, analyze, and interpret this information?
• Who will provide the information?
• Who will collect the inforrnation?
• When and how often will the information be collected?
• Which resources can be allocated to the evaluation?
• How will the findings be used and reported?
• What should the findings enable us to do?

In preparation for these meetings, I listed the program goals and brainstonned possible objectives and procedures to evaluate them. Learners prepared as well. They shared their assessment preferences as part of the intake process. Participants filled out questionnaires about how they liked to learn and how they liked to be assessed. The questionnaire was adapted from Nunan2 and asked these assess- ment related questions:

How do you like tofind out that your English is improving? By

--written tests ?

--teacher evaluation ?

--statements by your supervisor?

--seeing i/you can use what you learned on the job ?

--checking your progress by corn paring your work (eg. tapes)?

Do you get a sense of satisfaction from

-teacher grades?
--being told that you have made progress?
--keeping a learning diary?
--feeling more confident in situations at work and elsewhere?

Clarifying Assumptions

As the members of the Advisory Board shared their evaluation needs and preferences, the need arose to clarify the beliefs about evaluation that surfaced in the discussion and to reach a consensus on what evaluation should entail. These statements reconcile the viewpoints of those involved.

Evaluation entails more than identifying the individual and group achievements of participants. A variety of procedures are needed rather than one single measure of outcome. Depending on the level and purpose of evaluation, activities are needed that show how learners and the program are doing as the program is growing (ie. , formative evaluation) as well as activities that create opportunities to reflect on the achievements of learners and the program at set times (ie. , summative evaluation). Evaluation activities provide qualitative as well as quantitative information and both kinds of information are considered equally valuable. Activities and procedures are compatible with the assessment and evaluation pref - erences of those involved. Evaluation encompasses three levels: assessment of participants, evaluation of the program, and evaluation of organizational change.

Evaluation activities facilitate the participation of the following people:

When assessing participants

-participants themselves

-teachers

-nursing home residents

-supervisory and licensed staff

The assessment of participants aims at identifying changes in learning and changes in the transfer of learning to thejob.

Changes in these areas are documented:

-personal

-affective

-social
(within as well as outside the classroom)

-oral language use

-written language use

-functional uses of English in the delivery of patient care

-functional uses of English outside the nursing home

-metacognitive

-other

When evaluating the program

-participants

-teachers

-supervisors and licensed staff

-administrators -managers

-members of the Board of Directors

-funders

-representatives from CACA, the learning provider

-members of the Advisory Committee

Here the focus of the evaluation is on the program itself, that is, all the processes and resources used to bring about intended changes (ego , program design, the content and method of instruction, materials)

When evaluating organizational changes

-supervisors and licensed staff

-program coordinator

-administrators

-representatives from management

This level of evaluation aims at identifying organizational changes (eg., changes in the quality of care) and at clarifying the link between the education program and these changes.

Planning

Inspired by Feuerstein' s participatory approach to evaluating community development programs, I then facilitated the design of an evaluation plan.

This is the grid I used to represent the different planning decisions that needed to be made:

To determine which procedures were feasible or not, certain criteria simplified the decisionmaking process. These were key: the time requirement needed to be low; cost needed to be moderate; the active involvement of workers and supervisors needed to be assured; procedures needed to be able to produce qualitative as well as quantitative information; program staff and learners needed to feel comfortable with the procedure; and program staff needed to have the skills to implement them.

Formative Evaluation

These are the formative evaluation and assessment procedures and activities the Advisory Board chose to find out how learners, teachers, and the program were doing along the way and that allowed for midstream monitoring: feedback sessions at the end or during each class; class observations by supervisory, licensed, and administrative staff; peer observation by teachers; observation and mentor coaching by the Program Coordinator in weekly meetings with the on-site staff; meetings with the administrator on a regular basis (ie. , biweekly and when the need occurred); meetings with department heads, supervisors, and licensed staff; progress notes by teachers based on observations within as well as outside the classroom; learning diaries and individual and group journals; individual conferencing with learners at the cycles' beginning and end and at several time throughout the cycle; meetings with the Advisory Committee ( monthly); portfolios of learners' work; and learner writings and publications.

Summative Evaluation

These summative procedures and activities were selected and implemented with the completion of each 22-week cycle: learner self-assessments using checklists of functional uses of English as part of their individual education plan; teacher assessments of each learner using various reference lists to describe observed or reported changes and to analyze the learner's work; teacher assessments of the achievements of participants as a group (ie. , percentage of objectives achieved over cycle); learner evaluations of the program in the form of discussions in the learners' native language; supervisory and licensed staff assessments of learners (ie., transfer of learning to the job) and evaluations of the program in the form of a questionnaire; resident assessments of learners and evaluations of the program in the form of a conversation with program staff at a coffee and donut party; coordinator and administrator reviews of nursing home records (eg. , comparison of retention among participants and non-participants); and Advisory Board review of the program using key points drawn from the activities above.

A Sample: Learner Self-Assessment

Once a week in class participants filled out their learning diaries. By completing a number of sentences, learners documented on an on-going basis what they learned and studied and where they used English and with whom. They also specified what was difficult for them and what their learning and practicing plans were for the coming week.

The idea for the learning diaries came from David Nunan. I adapted Nunan's diary sheets, translated them into Chinese, made photocopies, and stapled the copies as a booklet with a cover sheet including the participant's name and cycle.

Several times during the cycle and at the cycle's end, I asked participants to review their learning diaries in class. I also usel the diaries in the individual assessment of learners and generated and compared inventories of purposes learners used English for. This allowed the learners and me to identify gains made over time.

These were the statements I included in the diaries:

This week I studied. ..

This week I learned. ..

This week I used my English in these places

This week I spoke English with these people

This week my difficulties were. ..

I would like to know. ..

My plans for next week are. ..

Evaluating Assessment and Evaluation

The implementation of the initial evaluation procedures and activities revealed the benefits and shortcomings of the evaluation plan and necessitated its ongoing evaluation. Also, learners,teachers, su- pervisors, administrators, and managers became more proficient in articulating their evaluation needs and preferences and priorities changed over time.

To accommodate these changes, the Advisory Board did not only review the outcomes of the evaluation but devoted time and energy to the evaluation of the processes that generated the information as well. As the language skills of learners deve1op, for example, the learning diaries were replaced by less structured dialogue journals.

The on-going communication about assessment and evaluation between learners, teachers, supervisors, administrators, and managers, however, does not mean that the evaluation plan cannot improve. On the contrary , the Advisory Board had committed itself to continue to look for ways to assess and evaluate that specific to the program's goals, tailored to the resources of the nursing home, conform with the grant guidelines, and are compatible with the individual and culture evaluation needs and preferences of those involved. This is, in my opinion, where the strengths of a participatory approach to evaluation lies.

References

Marie-Therese Feuerstein, Partners in Evaluation:Evaluating Development and Community Programmes with Participants. (London: McMillam, 1986), p. 1.

David Nunan, The Leamer-Centered Curriculum. (Cambridge: Cambridge University Press, 1988), p. 190.

Feuerstein, p. 20. 4 Nunan, p. 134.

Originally published in Adventures in Assessment, Volume 1 (November 1991),
SABES/World Education, Boston, MA, Copyright 2003.

Funding support for the publication of this document on the Web provided in part by the Ohio State Literacy Resource Center as part of the LINCS Assessment Special Collection.