Using Data for Program Improvement

Luanne Teller
Stoughton ABE Program

Adult literacy practitioners now collect and report various levels of data
to meet state and federal accountability requirements. There is no denying that these tasks are burdensome; however, many programs such as ours are moving beyond simple collection and reporting and using the data to strengthen our programs. In this article I hope to show that measuring
and understanding student performance help programs, partnerships, and
government to demonstrate and promote our true impact on our communities. Given the increased competition for funds, our field's best hope for maintaining our funding might be to provide hard evidence of our
students' success.

Our program staff have come to believe that data analysis has the potential to:

• help us identify areas of strength
  and areas for improvement,
• provide information that can help
  to deliver services in the most
  effective and efficient manner,
• provide us and our community
  partners with information about
  the value of our programs,
• help us to make informed policy
  decisions,
• enable us to focus on results,
• help us operate in a way that attracts professional staff,
• enable us and our community partners to take pride in our accomplishments and to enhance our roles in the community,
• ensure that formally reported information is accurate, and
• help us to retain and increase funding.

We all face limited time and resources. At our program we have decided to incorporate data into our decision-making process to insure that the strategies we adopt will be effective. At this point, I would like to illustrate how our staff used data analysis to support a program improvement goal. Here are the major phases of that effort, with special mention of where and how we used data to support it.

Our goal: To improve student attendance

Just over a year ago, I met with our staff, students, and members of our
community partnership to understand the issues that affect student attendance. Because research shows a direct correlation between the hours of instruction and learning gains, we decided to increase the intensity of learning. As a new, small ESOL program, we offered only six hours of instruction a week. More than 80% of our students work full-time, so they are unable to commit to additional structured hours of classroom instruction. We asked ourselves what strategies we could devise to support learning in and out of the classroom.

We checked the attendance data

The attendance issue was initially raised by instructors who expressed concerns about the disruption to class continuity caused by students who arrived late, left early, or had sporadic attendance. Discussions with students echoed these concerns. A review of our SMARTT data revealed some troubling trends; for one thing, many students were not using the six hours of class time we were providing for them. Students also reported frustration with empty seats, especially knowing how long it took for friends and family to get off the waiting list and into the program.

Further data analysis revealed that attendance problems fell into two specific categories. First, we had students who attended every night, but consistently arrived 20-30 minutes late. While 30 minutes might not seem like much,
at the end of the year it is the equivalent of missing almost six full weeks of classes. Second, we had students with excellent attendance who suddenly left for extended periods of time, often due to illness or the need to return to their native countries for family emergencies.

In our staff meetings, we decided to develop an attendance policy (like I said, we're a new program!). At this point, our community partners offered a wealth of experience and a range of policy options to consider. From all this information we developed a draft policy and distributed it to all the stakeholders for feedback, which resulted in our adding a provision for leaves of absence.

We then met with each class to review the new policy. Students in our program were already accustomed to signing a Student Learning Agreement, stating that they agree to follow the policies and procedures in the Student Handbook. When we added the new Attendance Policy to the handbook, we were careful to allow ample opportunity for student discussion and questions before asking them to sign the agreement.
We developed several other means of providing more "intensity" without adding class time:

• We implemented a sign-in/sign-out procedure for when students arrive late or leave early. (The number of students arriving late and leaving early immediately decreased).
• We developed a lending library so students could take books home for additional practice.
• The Stoughton Public Library, a partner, began offering ESOL Book Discussion and Conversation Groups. They also offered to house the lending library during the summer when
  classes are not in session, so students could continue to have access to materials.
• The LVA-Stoughton began to provide tutors for some of our lower level students. (Some of our advanced students volunteer as tutors for the LVA as well, giving them additional English practice.)

We looked at the data again

Another analysis of our attendance data revealed a huge falloff in attendance during the December-January holidays. In the face of this reality, we revised our program schedule to include a longer holiday break in December, which also provides more time for our staff to plan classes and regroup.

We monitored our results over time

After we implemented all the attendance policy changes, we began to monitor the results on a monthly basis. We used the SMARTT attendance reports to identify students whose attendance was not satisfactory. To address that problem, we developed a Monthly Attendance Report for the counselor, who now meets with these students to provide strategies and support where possible. Most students who receive a verbal warning improve their attendance, which is easily tracked by comparing monthly attendance reports.

After the first year, the data revealed that our attendance rate had increased by 12%! We also began to wonder if, in fact, the increased attendance had generated a proportionate increase in learning gains. Thanks to SMARTT, COGNOS, and other data reports, we were able to demonstrate that there is indeed a direct correlation between increased hours and increased student gains. It's much more rewarding to know that relationship exists than to suspect that it exists.

We were also hit with a big surprise. While our attendance had increased, our retention had decreased. Although the numbers were not significant enough to cause panic, they revealed a need to address student retention. Once again, we gathered exit interview data from students to understand what prevented them from staying in the program. We looked at trends to understand under what conditions students tended to leave. As a result, this year we implemented a new intake and orientation procedure. While our year-to-date retention is higher, the true impact can only be measured at the end of the year when we look at the "big picture" relationship among attendance, retention, learning gains, and student progress towards goals. We are also beginning to improve strategies for transitioning students out of the program. Sometimes, student retention is not a good thing! Some students never feel ready to leave, but now when we arrive at a point where we can no longer serve their needs, we have constructive ways to encourage them to take the next step.

We've gotten into the data habit

Data monitoring is now a regular part of our work life and informs virtually all of program decisions. For example, at staff meetings we distribute class attendance rates to our instructors, with comparisons to the prior fiscal year and to state averages. When we discovered that one of our instructors consistently maintains attendance at around 90%, we started to look at what we all can learn from her!

Basically, we look at our data in two ways: we compare our averages to state averages, and we compare our own data across fiscal years. While state averages are interesting to see, we tend to be more focused on continuous improvement. Looking at data from year to year helps us better understand the impact of our current plan and discover new areas to consider for improvement.

When gathering data, we often find it helpful to substitute the word "data" with the word "information." The data available in SMARTT and COGNOS is invaluable, but sometimes it is overwhelming anecdotal information that informs our planning. A recent example of this is related to our process for intaking students from the waiting list.

I asked my staff why it was taking so long to fill slots in our beginner ESOL class, since we have over 150 students at this level on our waiting list. I learned that when these students were called, they typically hung up because they didn't understand us or thought we were telemarketers! To buy some time, we asked current students to translate for us while we looked for a more permanent solution to this long-term problem.

After several meetings, we decided to create a "We would like you to begin class" post card with our logo and phone number on it. During initial registration/assessment, students are asked to fill in their mailing address on the post card. We explain that when we have an opening, we will mail them the post card with the date for them to begin. When the post card arrives, the student immediately recognizes it and makes the connection with our program. This has resulted in a much more efficient, equitable way to enroll students from the waiting list. While anecdotal data drove this process, the measure of its success will come from hard data. We will document the length of time it takes to enroll new students and determine the number of students enrolled per contact this year compared with last year.

We've learned that to use data consistently and effectively we've had
to "institutionalize" its use. To do that we've had to put the following steps
into practice:

• Plan for data analysis. We've learned that good data analysis cannot just happen episodically. We've had to set up meeting schedules, choose participants and include them in the process, find meeting space, and prepare copies of data reports.
• Identify data leaders. For most of us, understanding data is an acquired skill. We've identified people in our program who are skilled at putting data into context and understanding what it is trying to tell us - and let them take the lead!
• Celebrate success. If the data points to success, don't forget to take pride in it!

Luanne Teller is the Director of the Stoughton Adult Basic Education Program, a collaborative partnership among Massasoit Community College, the Stoughton Public Schools, and the Town of Stoughton, funded by a grant through the Massachusetts Department of Education.

Originally published in Adventures in Assessment, Volume 16 (Spring 2004),
SABES/World Education, Boston, MA, Copyright 2004.

Funding support for the publication of this document on the Web provided in part by the Ohio State Literacy Resource Center as part of the LINCS Assessment Special Collection.