Planning and Evaluation Teams: A Model for Workplace Education

by Olivia Steele, Deborah Tuler, Jane Shea, John Antonellis and Kathe Kirkman
Massachusetts Workplace Literacy Program

Evaluating the quality and effectiveness of workplace education programs creates both opportunities and challenges. To meet the challenges, the Massachusetts Workplace Education Initiative promotes a team-based approach to program planning, implementation, evaluation, and institutionalization. The approach uses a "Planning and Evaluation Team" model. Since its inception in 1991, the model has been implemented in over 50 workplace education programs. This article describes the issues that have surfaced and includes a response to them from four practitioners.

Evaluating the effectiveness of workplace education programs creates exciting opportunities. If done well, evaluation results can demonstrate concrete and tangible benefits of the program to the students, the employer, the union, and in some cases the community. Evaluation results--including, but not limited to, assessment results--can show what was learned, how that learning transferred to, and got used at, the workplace and in the community, and to what extent participation in the program impacts on the performance of the business. Evaluation results can also show to what extent the improvement of basic or language skills contributes to increasing the quality of goods and services and improving production or service delivery processes. It can inform program improvement, can promote institutionalization of the program, and can help determine whether to continue funding.

Evaluating the effectiveness of workplace education programs also creates challenges. Identifying gains in student learning, describing the transfer of that learning, documenting the impact of the program on the workplace, and demonstrating the program's overall quality and effectiveness have been daunting tasks for many practitioners. Several factors make evaluation challenging. One important factor is that workplace education programs are not yet part of the workplace's infrastructure and culture when they are initiated. Establishing this infrastructure and culture, as well as building systems to take care of planning, implementation, and evaluation functions need to take place simultaneously and require a considerable amount of resources and energy.

Another factor is the involvement of many different players, including students, teachers, administrators, employers, and union representatives, all of whom bring their own ideas and expectations regarding what constitutes success. Striking an appropriate balance between these different expectations is not an easy task.

The varying degree of involvement in the program by those who have a vested interest in it or have decisionmaking authority over it, is yet another important challenge to evaluating a program. When key decision makers such as Chief Executive Officers are involved directly, planning, implementation, evaluation, and institutionalization decisions appear to be facilitated more easily. High levels of managerial involvement, however, have in some cases created an imbalance of power. When key decisionmakers are not involved directly in the planning and evaluation of a program, those who are need to invest considerable time in keeping decisionmakers fully apprised. If not, the program may end up being decided upon from a distance by someone based only on his/her expectations and perceptions as to how responsive the program is. This perception may not be an accurate reflection of the program's performance.

Evaluating workplace education programs is most definitely challenging because of the expertise required to develop and implement an evaluation system. Ensuring the confidentiality of records, avoiding legal issues related to testing, and developing instruments that are reliable and valid are not easy tasks.

The most important challenge, however, is one of facilitation and is faced by the provider who in many instances has the primary responsibility for overseeing the evaluation. Often the teacher or the coordinator of the program become responsible for managing the whole planning, imple- evaluation, and institutionalization of the program. These are not small tasks and require a substantial amount of skill and experience in programming and working with different constituent groups. Even when teachers and coordinators have the know how and resources to do this work well, they still encounter numerous obstacles on the road to demonstrating program effectiveness. Limited access to information and key individuals at the company or union have made the management of the planning and evaluation process a tough hurdle to jump.

From Practice the Theory and Policy: The Planning and Evaluation Team Model

For a decade, a variety of federal and state funding sources have supported workplace education programs in Massachusetts. Under the umbrella of the Massachusetts Workplace Education Initiative--a statewide partnership among business, labor, education, and government (represented by agencies that fund workplace education programs)--extensive evaluations are conducted annually to identify best practices. A 1991 evaluation summarized the challenges practitioners faced in implementing workplace education programs and articulated policy recommendations to address them.

When key decisionmakers are not involved directly in the planning and evaluation of a program, those who are need to invest considerable time in keeping decisionmakers fully apprised.

One of the main recommendations was to put in place a governance structure, rather than an advisory structure, for local business, labor and education collaborations. The recommendation acknowledged the need to systemically address the significance of performing, planning, implementation, evaluation, and institutionalization functions through a teambased approach. This approach would ensure that clear and realistic goals for the program were set, agreed upon by all partners, and eventually met by enabling all those who could benefit from the program to participate in shaping it. The proposed model was the "Planning and Evaluation Team." The Massachusetts Department of Education has since required all recipients of its workplace education awards to implement this team-based approach.

When debating goals, it often becomes clear that team members may hold different views. To strike a balance, realism and solid data help the team get clearer on realistic goals and how to prioritize them.

The initial results of piloting the Planning and Evaluation Team (PET) model were favorable and currently variations of the model are being used by all workplace education programs funded by the Massachusetts Department of Education with National Workplace Literacy Program and 8% Job Training Partnership Act funds. Today there are 27 PETs at sites funded through the National Workplace Literacy Program and an additional five at sites funded through the 8% Job Training Partnership Act. PETs consist of representatives from upper management, supervisors, employees (who are enrolled in the program, have an interest in enrolling, or have graduated from the program), teachers and administrators representing the education provider, and labor representatives. Teams meet regularly throughout the life of the program. Initially, teams may meet very frequently. As the program solidifies, teams meet less often.

Top of Page

Functions and Tasks Performed by Planning and Evaluation Teams

The purposes of the PET are multi-fold: to plan the program, to oversee implementation, to design and implement evaluation, to promote improvement, and to institutionalize and integrate the program in the long-term strategy of the business and/or union. The Massachusetts Department of Education views its funding as seed funds or an investment in building the capacity of the businesses or unions so they are equipped to continue the program without public funding. The return on this investment has been promising. Almost 65% of all partnerships that received funding under the National Workplace Literacy Program through four funding cycles, continued when public funding ended.

PETs handle specific tasks and go through several stages. First, the commitment from labor and management to a team-based approach is sought. Subsequently, a team is established by identifying key players in the company, labor and the learning provider. In this discussion, the role of the team, the time and energy required, and ways to orient team members are usually clarified. The first tasks are usually accomplished in pre-program discussions between the education provider, management, and the union.

Once the team is established, a start-up meeting is set. The team begins identifying goals for the workplace education program by answering questions such as, "What is our ultimate reason for having the program? What is the purpose of this program? What goals do we hope to accomplish by the end of this program?" When debating goals, it often becomes clear that team members may hold different views. To strike a balance, realism and solid data help the team get clearer on realistic goals and how to prioritize them. At this point, it is critical to spell out how program goals relate to business and labor goals and not to view them as fully equal. Workplace education programs can help address business needs and goals but in few instances are the sole incentive needed.

The next task involves identifying appropriate indicators that show the program is making progress towards its goals. Following this step, the PET identifies which components of the program (e.g., three ESOL classes) and what kind and level of business and labor support (e.g., release time) is needed. This is also the time when resources are allocated.

Deciding what to evaluate, how, when, and why is next. The PET selects or develops ways to gather the data the program needs; collects and analyzes the data; and utilizes the information to improve the program. In considering data-gathering instruments, the PET considers resources (e.g., time and dollars required); levels of participation required; staff expertise in evaluating; and other factors.

Improving the program based on evaluation results may lead to modifying goals. This is the last task the PET undertakes after going through the planning and evaluation cycle once. PETs, however, do not necessarily wait until this stage to make revisions to their goals. Changes in the industry (e.g., the recent restructuring and refinancing of health care) may cause the team to modify the course of the program prior to completing its planning and evaluation cycle. Another instance may be that the team discovers that they are collecting important data that are not captured by its current goal statements. These instances demonstrate the need to implement stages and tasks for PETs on an ongoing basis that allows for flexibility and ensures that the process is non-linear.

From Theory to Practice: Implementation Issues

PETs have been successful to varying degrees. More than five years of working with the PET model has taught practitioners that a number of issues surface year after year and in both state-funded programs and other publicly- and privatelyfunded programs that have attempted to replicate the PET model. The issues are:

• How do we start up a PET in a program that is not funded by a public funding source that requires team-based approaches to programming?

• How do we get supervisors and other company representatives involved in looking at program impact on the workplace?

• How does the PET as a group develop and use a set of tools to gather data?

• What do PET members need to be aware of when developing tools? What happens when we get data we cannot use?

• How do teachers and coordinators on PETs facilitate identifying goals that do not hold the program accountable for the broader company goals, which may require more initiatives or interventions than the program?

Program coordinators from workplace education programs at business sites across the Commonwealth responded to these issues and identified lessons learned.

1. How do we convince an employer of the importance of a governing body responsible for developing and implementing a workplace education program when it is not required by a grant? How do we pull all the necessary players together? Debbie Tuler talks about her experience in a Hospital Workplace Education program:

I was raised on Massachusetts' team-based approach to planning and evaluating workplace education programs. The data collected over the years in the Massachusetts Department of Education programs demonstrated the value of this team-based approach. So, when I took over a private contract with a hospital (which had offered ESOL/ABE classes for five years without a team or advisory board), I expected that I would institute the PET model in order to bring the program in line with what I believe are indicators for effective programs. I also believe that a collaborative model like the PET approach is important because it keeps players connected, committed, and informed about what is involved in a long-term educational process, the workplace education program. I did realize, however, that I could not force it.

By asking questions and engaging supervisors and managers in conversations we encourage them to recognize the benefits.

First, I had to convince managers/supervisors that it was worth their time and effort to be involved in the program at some level. Managers and supervisors were unfamiliar with the whole idea of having a coordinator/instructor speak with them about areas of concern and evidence of progress and to request documents and information from their departments. They viewed the program as a benefit that the hospital offered to workers and in some abstract way was a benefit to them too, but they had never dearly understood this benefit.

I pursued conversation with managers and supervisors. There were a lot of changes taking place in the hospital. This meant that employees would have an opportunity to upgrade their position if their language skills improved and if supervisors and managers noticed that their language skills had improved. It was important to get supervisors and managers to see the benefits of the program to them and to the workplace in general. By asking questions and engaging supervisors and managers in conversations we encourage them to recognize the benefits.

Top of Page

By encouraging managers and supervisors from Environmental Services, Transportation and Nutritional Services at the hospital to offer input into the program, I did in fact open the door for expressing concerns. Until now, they had mainly voiced frustrations that more workers from the day shift do not take advantage of the benefit of classes (classes met from 3-5 pm and their shifts end at 3:30). At the same time, supervisors can only let so many people attend class at the same time during work hours because of staffing constraints. Students pointed out to me that other day shift workers cannot attend class because they had other jobs or child care concerns. Issues of concern then related to scheduling, recruitment and policies. I pointed out to each party, supervisors and students, that it would be easier to resolve these issue if everyone sat at the table together and discussed these issues rather than having me be an intermediary.

Last spring (1996), after six months of working at the hospital and getting to know the various players, I was able to get commitments on the part of managers from Environmental Services, Nutritional Services, and Transportation to participate in an "Education Committee" (a PET by another name) which would include workers in the program. Their task would be to address some of the issues mentioned above. We would start there and move to other areas of the program where this collective input is needed, including curriculum design and goal setting.

We had our first meeting in October to air and clarify concerns and to identify next steps. After discussing the reasons people cannot come to class and the issues surrounding releasing workers to attend class, we came to some decisions. These changes amounted to: change the time of day of classes in order to encourage more participation from day shift workers, have the instructor attend departmental staff meetings to promote the program with assistance from current and past students, and set up a buddy system where workers who attend the program could bring another worker/friend to class to try it out.

What impressed me about the meeting was that people were talking to each other, not just me. I also got a commitment from the committee to meet monthly to resolve issues that arise. I felt that communication was opening. With this, though, came the realization that the managers believed that the programs solely benefited the workers who participated in it and not themselves. The meeting did little to change this attitude: "They should take advantage of this.... They should make time to improve their language skills." For the program to continue and develop, this perception needs to change. This can be done in the committee or with individual managers. It is important to keep asking questions and engaging in dialogue. They need to buy into the program so that they argue for its continuation amidst budget cuts and layoffs.

My experience at the hospital during the past nine months has made it clear to me that PETs (or their equivalents) in different workplaces may play different roles and define their purposes in different ways. Defining these roles and purposes is a real challenge.

2. How do we get supervisors and other company representatives involved in looking at program impact in the workplace? And why is this important? Jane Shea from Quinsigamond Community College's workplace education programs talks about taking involvement of supervisors and other company personnel to another level:

It really all starts with the initial goal setting process that we do with our planning and evaluation teams. This is our springboard. We ask the PET to help identify people within their workplace to solicit feedback about the progress of the program. We have them identify people who they think are not that knowledgeable about the program and its benefits, as well as those who are directly involved through the PET, supervisors of workers, etc.

Most people tend to respond to questions about program impact and progress by talking about individual students. We make it clear that we design our questions to help people look more broadly at the program's impact and less on individual students. This process takes time; it also involves asking the questions over and over and engaging in dialogue. For example, we ask, "Have you seen any changes in the workplace /workers? What do you hope to see in this workplace after the program has ended?"

Of course, getting people to answer questions can be a difficult task. People are generally very busy and hard to track down so we need to be flexible in soliciting this information. As the coordinator, I will go to staff meetings and use the questions to gather information collectively. I will ask people when I happen to see them in the hall. I will send questions out to individuals in a survey form and then arrange an appointment to meet them and discuss the questions. I have also made appointments without giving out the questions in advance.

It is worth finding a variety of methods to gather this information in order to note the many different perspectives within the company. The ideas and comments we get from supervisors and workers who have direct or indirect impact on the program is very valuable. They deepen our understanding of the workplace. These comments inform us about the employees' perception of the program.

Of course there is always the question of confidentiality for students as well as supervisors. We are very clear up front before we start working with programs, that we have a clear policy protecting the confidentiality of individual students in the program and those that have offered to answer the evaluation questions. We realize that it is very hard when you ask someone how they think the program is going for them not to think about an individual student in particular. The questions, as I mentioned before are designed to look at the general and not specific.

But here is the dilemma: we really like hearing and sharing about the successes of students or the positive feedback of supervisors and managers and really have to try to hold to the same principles and give that worker the opportunity to decide whether to disclose the success, such as getting a GED or not.

Although people within the program (learners, teachers, PET members) aid in the data collection process, PET has the responsibility to report that information to others in a way that recognizes and validates the supervisor's voice, maintains a level of confidentiality for the workers participating in the program, and gives enough information to help inform the planning process.

Top of Page

3. How does the PET as a group develop and use a set of tools to gather data? John Antonellis, coordinator of Workplace Education programs at Jewish Vocational Programs in Boston, talks about where tools may come from:

The starting point for developing evaluation tools is the program's goal statement. Usually, the PET will examine the goals for the program and discuss who is in the best position to evaluate each goal. At this point we'll have a brainstorm about the kinds of tools that could be used to gather the information we need. Next, I'll draft a set of tools and bring these back to the PET at the next meeting. Having something concrete to work with is important. I've found that this makes it very clear if we're all talking on the same wavelength. The education provider is the perceived expert regarding program evaluation and does in fact have access to a wealth of resources based on past experience which can be used to develop evaluation tools that are tailored to each program. Further, as most PET representatives are under pretty serious time constraints, coming to the PET with a draft in hand is a more efficient use of time. The PET then looks over the drafts and any needed changes are made.

Embedded in every evaluation tool are assumptions about your program's goals. Ideally, the process described above occurs early on in the establishment of a workplace program. While programs usually develop a goals statement early on in the process, they have not always linked their evaluation tools explicitly to these goals. In some cases, the program may be using tools developed for a previous program or may have been developed independently of the goals statement. In cases like this, what I do is examine both the goals statement and the existing evaluation tools. Together with the PET we attempt to make the connection between the goals statement and evaluation tools explicit.. In some instances we have found that the goals embedded in an evaluation tool more accurately reflect the true program goals, and the goal statement is adapted accordingly. In other instances new tools are developed to account for any gaps found between the desired program goals and existing evaluation tools. My experience has been that programs are generally very good at designing tools which measure the impact of the program on the participant, but measuring the impact on the workplace and on the educational partnership is a greater challenge.

We are really trying to develop tools which provide both qualitative and quantitative data that is used to improve existing programs and aid them in their efforts toward institutionalization. One of our most successful tools is the survey because it can be designed to provide both quantitative and qualitative information, it is relatively easy to administer, and requires only a minimum of time to complete. The surveys I've developed are explicitly tied to the goal statement, are short (one side of one page), and include a combination of open and closed ended questions. Open ended questions provide qualitative information while closed ended question provide quantitative results. Another advantage of using surveys as an evaluation tool is that the same survey can usually be adapted for use by both supervisors and program participants. So everyone is evaluating the program for the same goals.

4. What happens when we get data we cannot use?

I think I have heard this joke before, and the punch line is, " Manipulate the data." Wait! That was intended as humor and not a suggestion.

Most data can be used in "legitimate" descriptive ways even if that isn't really what had been planned. Any data collection plan should include a section on how you intend to use the data. Potential "use" problems often can be detected at the planning stage. However, if you forgot this step in the plan, relax and think creatively (and legitimately).

For instance, you may have hoped to discover how many students progressed during their time in your course, but you only have aggregate data indicating class pre-test scores of 10, 10, 20, 20, 30, and 30 and post-test scores of 10, 20, 20, 30, 30, .and 30. In this example, one 10 could have moved to 30 (one person progressed); one 10 could have moved to 20 and one 20 moved to 30 (two people progressed); or two 10's could have moved to 20, one 20 moved to 30, and one 20 regressed to 10 (three people progressed); etc. You can't "use this data" to indicate how many students progressed, but you can use this data to state that the "class average post-test" (23.3) is 3.3 points (or 17%) higher than the "class average pre-test" (20). This is a classic "unit of analysis" problem: you wanted student (unit) information, but because you didn't track individual student progress, you got class (unit) information. Actually, I think a 17% overall increase sounds better than saying one or two students progressed.

Most data that is collected for loftier purposes (e.g., unearthing correlations) can be very useful as descriptive if your data really doesn't meet statistical assumptions. Most readers understand and enjoy pictures (descriptions) as much as statistics anyway.

A word of caution when thinking creatively about your data: in some cases your lemons cannot be turned into lemonade. If you really have garbage data (check list below for suggestions on getting garbage-proof data), don't try to recycle it... JUST TOSS IT OUT.

Hints for effective evaluations. These are suggestions that I find helpful in guiding my evaluation work:

1. QUERY. Start with a specific, simplyphrased research question, such as: Do students in the Math classes make fewer computational errors on sample work orders after the course than before? The question identifies WHO (students in the Math Class), WHAT (computation), HOW (sample work orders), and WHEN (before- participation) measurement will occur.

2. CLARIFY. Try to identify factors that might impact the answer to your question, other than the factor you would expect. For the question above, the style of instruction may make a difference in outcomes if there are more than one class/ instructor. The number of times (or time of day) students attend the class could impact outcomes. Each of these factors may require additional clarifications to your question, such as: "Do students attend 90% of the math classes...," etc. These clarifications will guide your analysis and interpretation of the findings.

3. SIMPLIFY. Simple data collections, such as a test or survey, may not be as soul-satisfying as extensive interviews or simulations, but they are usually easier to do and yield easily-interpreted data. If you choose a more complex route to data collection, be sure you plan for the extra time that will probably be needed to analyze the results. For example, openended interviews often yield very rich information, but analyzing this data can require several readings, coding and reliability checks.

4. QUANTIFY. Anecdotal evidence enhances evaluations, but most businesses and funders need quantified results. Even data that is collected qualitatively, through interviews or open ended questions, should be coded into categories and quantified.

5. UTILIZE. Even if you are doing evaluation to satisfy a requirement of your employer or funder, make sure it is useful to your program and that you USE the information you get. Evaluation takes time and energy, and GOOD evaluation is an excellent tool for program improvement.
   

5. How do teachers and coordinators on a PET facilitate identifying goals that do not hold the program accountable for the broader company goals, which may require more initiatives or interventions than just the program?

It is crucial during the goal setting process to phrase questions in such a way as to help employers see the distinction between company goals and program goals. When an employer says that his goal is to improve productivity, ask how the program can help you achieve this goal. What is the link between the basic skills instruction which we offer and the company's broader goals of improving productivity? Are those who will participate in the program the ones who are responsible for this desired outcome? Who else is responsible but will not be participating in the program, and how will we get data from them?

Too often I have seen increased reading and writing skills as a desired outcome for participants when they are not in a position that requires any visible reading and writing or are often not offered opportunities to show gains in these areas. The result is that we as a team do not have the data needed to determine if the company goal is being reached. The answers to these questions above will force the team to define the connection between the program and the company, and articulate the parameters of the program. This questioning should be the starting point for setting goals and designing instruments.

Top of Page

Originally published in Adventures in Assessment, Volume 9 (December 1996),
SABES/World Education, Boston, MA, Copyright 1996.

Funding support for the publication of this document on the Web provided in part by the Ohio State Literacy Resource Center as part of the LINCS Assessment Special Collection.