Program Evaluation at the Community Learning Center

Mina Reddy

The Community Learning Center is the adult basic education center of the City of Cambridge. We have classes in English as a second language, basic reading, writing, and math, G ED preparation, and an Adult Diploma Program. Over 1000 students a year come to the program, 60% of them for ESL. The Learning Center has 18 salaried staff members, most of them full time, and 29 part time staff. Salaried staff (teachers, counselors, and administrative staff) meet weekly in staff or depanmental meetings. The CLC operates with a group decision-making process. All major decisions must be made by majority vote of the entire salaried staff. Each depanment (administration, ESL, and ABE/high school) selects a coordinator. The three department coordinators and the director meet every other week. They can make some limited decisions but are primarily responsible for maintaining good communication between and within departments and structuring center-wide activities like evaluations. Two student advisory committees, one for morning and one for evening students, and made up of one representative from each class, meet monthly to make suggestions to the staff and help organize activities like parties and fundraisers. Discussions led in class by these members help give students an overall picture of the school and make them feel better able to evaluate it. Informal program evaluation occurs in staff, department, coordinator, and student advisory council meetings all year. A more formal program evaluation process occurs yearly. Questionnaires are distributed to staff and students and meetings are held to discuss the results and plan program development for the following year. (see Fig. 1)

FIGURE 1

Development of the Staff Evaluation Form

In previous years, we had used a brief open-ended form for staff program evaluation This year, in part because we had made a commitment, for the first time, to develop a three-year plan, we decided to do a more extensive process. Although we had not joined the SABES program development pilot, we asked for a copy of the materials distributed at the training for the pilot. We based our form on the clndicators of Program Quality Checklist developed by SABES and spent two coordinators meetings modifying it to fit our program.

Administration of the Staff Evaluation Form

The form was explained in a staff meeting and was distributed to all full and part time staff. They were told they could skip any questions about which they did not have enough information, and they were asked to star their top five priorities among needs. Part time teachers were given an additional hour of paid time to fill out the form. People took it very seriously and, in some cases, wrote extensive comments; they appreciated being asked their opinions.

The results were tallied separately for full and part time staff and then combined and all comments (15 pages of them) were typed up. Tallies and comments were distributed to all staff.

Development of the Student Evaluation Form

In February 1991, in morning and evening student advisory committee meetings, students were asked, "What do you want to learn in school?" and "Does the school help you with anything you do outside?" Students came up with a lot of specific answers, including some that staff might not have thought of putting on a questionnaire like, "Help me enjoy reading," and "Be able to make friends." These ideas, along with a few additional ones added by teachers, were turned into questions on a check- list for students to fill out in a program evaluation questionnaire. Two problems arose, however, when the questionnaire was administered in 1991. The checklist was a long list of items with directions at the top asking students to check what the school had helped them with. In some cases, students forgot or misunderstood the directions and checked everything they were able to do even if the school had nothing to do with it. Also, students who did not have a job were checking that the school had helped them find a job and students who did not take math responded to some of the questions specifically related to activities in the math classes. The form was therefore revised last spring by breaking it up into sections, making the questions clearer, and allowing students to skip sections that were not relevant. The revision was based on feedback from teachers and a meeting between the director and two representatives of the student advisory committee. (see Fig. 2 )

FIGURE 2

Administering the Student Program Evaluation Form

Copies of the form were given to all teachers to distribute and explain to students in late May. Teachers of beginning ESL students could choose not to use it if it would be too difficult for students to understand. We received 32 responses. We were fortunate to have a VISTA volunteer to tally the responses, turn them into percentages, and type the comments—32 pages of them!

Comments from the student questionnaires

"Please give other examples of what you learned in school and how the school has helped you."

"Before I came here I didn't speak English. I remember once I went to the hospital. I couldn't find an interpreter. I came back without an examination. Now I can explain everything I want. "

"I really learned in school, to lose the timidity, to communicate, to read a little."

"This school gave me a second chance to succeed in life. I made a mistake before in not finishing school and this school gave me hope for a better future. "

" I learn to have an open mind by coming to school. I can think better."

"The school has given me the understanding of other people and their cultures, of how diversified we all are, and that we have one common interest: to better ourselves."

"Since I came back to the Learning Center it has helped me a lot. My English comes better every day. I had a promotion in my job."

Analyzing the Data

Coordinators met to summarize the staff program evaluations and decide on the areas of greatest need for development. In order to determine the highest priorities, we considered 1) the number of people who checked an item as a need, 2) the number of stars put next to an item, and 3) comments. (Ideally, student responses would have been part of the discussion at this stage, but tallying of the forms had not yet been completed. They were available by the time action groups met.) The needs were grouped into six categories which were revised in the next staff meeting. The final categories were curriculum, counseling and follow-up, physical plant, part timer needs, staff development, funding, and assessment.

Action Groups

Two weeks had been set aside at the end of June in order to have four full days for planning and still leave enough time for our graduation and several staff development workshops. In a staff meeting the Friday before the first planning week, 1) the coordinators reported on the proposed action groups and specific issues or needs to be worked on in each one according to staff and student responses, 2) staff modified and combined the groups, and 3) each person chose one action group to participate in. Specific needs were spelled out within each category , e.g., under curriculum: math, Adult Diploma Program history, intermediate reading, and work-place education curriculum; student input into curriculum; use of textbooks; use of technology; number of hours of class per week. The groups were to meet Monday and Tuesday of the first planning week to draw up three-year plans for that area. Action plan meetings would be alternated with staff and departmental meetings. The following week, there would be two more all-day staff meetings to finalize the plan, with action groups meeting separately as needed. Each action group was given a form (see Appendix 4), lifted from a request for proposals, to write their objectives, aCtivities, persoris responsible, and timeline. Staff sometimes had difficulty seeing the difference between making a plan to solve a problem and solving the problem on the spot. Controversy arose when the assessment group wrote into the plan specific tests to be given when the staff had not agreed on the appropriateness of those tests for large numbers of students. The plan was revised to be more general and allow time to examine and select tests and set policy.

It was also difficult for staff to think in terms of a three-year plan. In part because of the nature of our funding, we have been used to thinking only one year at a time. We tended to set September 1992 as the time to accomplish an unrealistic proponion of the tasks and had to revise the timelines when we looked at all the plans together. Before ending the meetings, we set up times to monitor our progress towards our objectives over the next year. June was a difficult time to hold a series of day-long meetings because everyone was tired from the semester of teaching that had just ended. However, at the end of the process, we felt we had accomplished something significant.

Next Steps

Once the meetings were finished, the task remained of typing up the separate action plans into one document and prefacing it with a statement of strengths that were identified in the process. Most responses to both evaluations were very positive, and this was not acknowledged anywhere in this process which focused on needs and areas for development. This work will be done during the summer. When we meet again as a whole staff in September, we can make any revisions that seem appropriate at that time.

What Will We Do Next Year?

Will we use the same form and process next year? Probably not. However, we hope that this description of what we did is useful to other programs in developing their own system of program evaluation.

Originally published in Adventures in Assessment, Volume 3 (April 1992),
SABES/World Education, Boston, MA, Copyright 2003.

Funding support for the publication of this document on the Web provided in part by the Ohio State Literacy Resource Center as part of the LINCS Assessment Special Collection.