Analyzing Your Organization's Data to Tell Your Story

Heidi L. Fisher, Carol L. Gabler
Literacy Volunteers of America-Chippewa Valley, Eau Claire, WI,

The Challenge

A program with a tendency to collect too much data needed to develop a clear and concise data plan that would allow meaningful articulation of its successes and challenges.

Who We Are

Literacy Volunteers of America-Chippewa Valley (LVA-CV) is a non-profit organization that began providing services in 1986. Our main office is located in Eau Claire, WI, but we serve a tri-county area. Because of the multiple rural areas served, it remains a constant challenge to meet the needs of adults and families while remaining a cohesive organization.

LVA-CV provides services through several programs: one-to-one tutoring, jail instruction, workplace education, citizenship, and comprehensive family literacy. During the 1999-2000 fiscal year, 246 adult students and 96 children were served through LVA-CV programs, with a total of 13,701 instructional hours. Ours is primarily a volunteer-based program, but direct teacher instruction takes place at our family literacy sites located in two counties.

Our Story

At LVA-CV we knew that everyone was working hard to collect the data necessary to satisfy a tong and varied list of partners and funders. At year's end, we found our selves floundering in tong, detailed reports from the ten individual programs spread out over three counties. What was worse, all were using slightly different recording systems to collect data. Each submitted a variety of reports to our executive director. This made it particularly challenging to compile program and organization-wide evaluations, analyze the data, to share with our funders and board of directors. The sheer quantity of data was obscuring the essential information and impeding our progress and ability to share successes and challenges of the students served in our programs.

Our challenge was to pull consistent pieces of information from all segments, record that standardized data accurately in a computerized collection system, consolidate the findings, and produce a report. Our involvement with the What Works Literacy Partnership (WWLP) led to improvements in our approach. By asking the right and same questions of every segment, we were able to determine what information we needed at the beginning, thus avoiding a lot of wasted time and energy.

Top

Recommendations

Developing an efficient data plan involves a cycle of collecting, analyzing, organizing, revising, and articulating. We recognize that our work has only just begun, but based on what we have learned, we can recommend these steps when developing a data plan. See also the flow chart at the end of this article.

1. Examine your organization's strategic plan
   

   • Clearly define your program goals through strategic planning. LVA-CV's strategic planning process involves both staff and the board of directors. The strategic plan incorporates a healthy cycle of planning, reviewing, and evaluating at all levels. Each staff member also develops an annual action plan as a focus for his or her individual staff goals.

    

2. Determine the questions you need to ask
   

   • Include your staff at all stages to ensure staff "buy-in" and thoroughness. The staff is in touch with information that can be easily gathered and has an awareness of what will be required for consistent data collection.

   • General questions guide the early stages, but evolve and become more sophisticated with time. Examples of general questions:

   • What do we need to show learner progress?

   • What do we need to accurately measure outcomes?

   • What do we need to guide program planning?

   • More specific questions help pinpoint strengths and weaknesses in individual programs. Examples of more advanced questions:

     • Are family literacy students making progress after 50 hours of instruction?

     By asking what we need to know to become more effective, we are better prepared to determine from our data such factors as the percentage of students who make gains on standardized tests, the percentage of tutors who have completed the competencies for training, and the percentage of students who have achieved one or more persona goals. By including the staff in creation of the questions all data can then be gathered in an efficient way.

   • Changing needs affect question selection. We must schedule time to review and assess what we have learned from collected data at the end of each semester. This analysis helps us ask better questions and then adapt our programming to best meet students' needs.

   • Questions need to support strategic plan. This cycle does not always flow in a step-by-step manner. For example, you may discover that data questions do not support your strategic plan. In this case it would be important to revise questions to ensure that the organizational needs are being addressed.

    

3. Develop/revise the data plan
   

   • Define roles.
     Determine who is in charge of data (e.g., the data person, the teacher, and/or coordinators). in our program it was decided that educators and technical personnel should share responsibility for data decisions. As a group they determine how they will collect, process, manage, and analyze data.

   • Establish a timeline for the assessment process.
     Determine when testing will need to take place. In our program it was decided that we need at a minimum to pre/post-test annually. The data questions we ask help to determine the timeline.

   • Standardize the data collection process.
     We incorporated a computerized data collection system to provide consistent data recording. All teachers receive training and are expected to follow the same collection procedures. We discovered that not all teachers were assessing in the same manner, so we reviewed time guidelines and appropriate assessment procedures.

   • Revise forms to reflect the questions.
     This streamlines data entry. Revised forms have helped us to ensure that we were collecting all information up front and we did not need to go back and "fill in blanks."

   • Define terms for consistent usage.
     We provide time in monthly staff meetings to ensure that terms such as "on hold" and "waiting to be placed," mean the same to all working with data and assessment. We also discovered that individuals from our three counties used different definitions for "full-time" employment, which resulted in inconsistent data.

   • Review and standardize testing practices.
     When we formalize how tests are given, we can more accurately measure the outcomes. At the start of every year we review our test practices to assure consistency in timing and administration of assessment tests. We make sure students receive the same pre-test as posttest. We revise inefficient strategies, such as our original decision to administer standardized tests after 50 hours of instruction, which proved to be too soon. We now do pre- and post-testing every year with approximately 80% of our students. We have also come to realize that not everyone who comes into our program is going to benefit from the standardized assessment process.

   • Strengthen staff communication.
     Monthly staff meetings designed to deal with issues of data collection provide an opportunity to share information and ask questions. They foster a supportive environment in the team effort to do things right, as do occasional staff lunches geared to staff interaction time. Bringing in experts who can help clarify the crucial questions and assist with technology now can save time and money later.

    

4. Aggregate data for reporting

   • We had to decide what information would be collected, and when. Should reports be made monthly, every six months, yearly, or a combination?

    

5. Analyze your data to tell your story
   

   • With continual program improvement being our focus, it is critical to take the time to interpret the data that has been collected. Without this crucial step, a data collection cycle is not maintained; rather, it is a beginning and an end with no connection to the following year. We need to have the courage to make changes in our program, curriculum, and/or strategic plan based on insights, trends, strengths, and weaknesses in the data in order to continue the cycle.

    

   Recognize that you are probably never going to achieve the perfect system, but efficient standardized data collection is essential to continuous program improvement. The answers are there if you ask the right questions. With a focus on the needed elements to collect, it has helped our director and staff to be able to analyze and clearly share our story with the board of directors, funders, and other organizations.

Originally published in Adventures in Assessment, Volume 13 (Spring 2001),
SABES/World Education, Boston, MA, Copyright 2001.

Funding support for the publication of this document on the Web provided in part by the Ohio State Literacy Resource Center as part of the LINCS Assessment Special Collection.