Assessment Challenges in Supported Distance Learning: How the ABE Distance Learning Project is Implementing the Massachusetts ABE Assessment Policies and Procedures

Roger Hooper
Massachusetts ABE Distance Learning Project / ALRI

The Massachusetts ABE Distance Learning Project is a statewide collaborative including three specially funded demonstration pilot sites, and statewide project coordination staff. Our vision of "supported distance learning" (SDL) includes delivery of content and instruction largely through technology and multimedia curriculum packages, but also includes learners in some face to face contact with teachers, counselors, tutors and other students for support and assessment. The integration of various supports into a distance learning delivery system reflects the commonly understood needs of all adult basic education learners.

These learners, who bring so much wisdom and life experience to the educational process, often cannot bring the tools we expect of the typical "student" such as time management, understanding of their own learning needs and styles, ability to prioritize and evaluate the material to be learned, and the skill to navigate the barriers that work and family often present to the adult learner. Adult basic education learners need supports to work with these academic and social issues. In classroom ABE programs, teachers and counselors assist the learner "on-site" with these issues when necessary. But learners in our SDL pilot programs, although similar in strengths and needs to all other ABE students, do not have the same level of immediate or frequent access to teachers and counselors to address these issues. Our Pilot Programs provide access to necessary supports for learners that can help ensure their retention and progress even while the major part of their learning takes place at a distance from the program site and the staff.

The mission of the Project is to research which supports best help adult learners to succeed in distance learning, to document models of effective SDL programming, and to recommend by the end of FY'04 a schedule of program quality indicators and cost factors for SDL. The Massachusetts Department of Education has committed to evaluating our recommendations prior to deciding whether to offer SDL among the program options fundable to all eligible programs under the next state multi-year ABE funding cycle.

Our Project agenda for FY'03 includes building effective strategies to assess learners, and evaluate program effectiveness in SDL. We are building these on the platform of services that our demonstration pilot sites have developed since the Project's inception four years ago.

The framework for this task is the Massachusetts Department of Education's Assessment Policy for ABE. The challenge is to create strategies that work within the specialized services developed in our pilot sites, and that also meet the overall requirements of that Policy.

The Agenda for ABE Distance Learning

The Massachusetts Adult Basic Education Distance Learning Project entered its fourth year of operations in March of 2003. The Project has made strides in a broad spectrum of areas of program development for SDL including:

• identifying and using technology appropriate and accessible to ABE learners who enroll in SDL programs; finding and implementing published multimedia curriculum packages that conform to the body of knowledge needed by ABE learners in SDL (and in traditional programs) -- in both ESOL and in ASE/Pre-GED instruction;

• developing regional "hubs" through collaboration with other agencies and programs, doing outreach and recruitment of learners, and creating training and orientation models to bring these learners into the SDL program;

• developing and establishing within an SDL context supports for learners that support retention and educational progress;

• sustaining SDL pilot program staff with ongoing technical in-service support, and sharing lessons learned about program and staff development for SDL with the larger community of ABE practitioners;

• and exploring strategies for ASE/Pre-GED and ESOL learner assessment in SDL contexts.

Overall, assessment is a larger concept than just testing learners.
In our broad SDL developmental project, it is an "umbrella" under which we include measuring participation, retention, progress and completion of goals. The Massachusetts ABE Assessment Policy focuses on the assessment of learners' level gain as established by standardized testing instruments.

Since our SDL pilot programs are included under this policy, the pilot sites are developing logistics to try and meet the state demands. Because the policy was designed to be implemented in a classroom-based learning program (students in classrooms with a teacher and a regular schedule of meetings over a specified cycle of instruction), it is a challenge to develop strategies that make it fit SDL programs in which our learners

• mostly (but not exclusively) work off-site and independently with multimedia curriculum packages;

• rarely work in groups;

• have a tendency toward "open entry/open exit" participation even if the services are designed in set instructional cycles;

• and only occasionally meet face to face with a trained, professional ABE practitioner.

A Variety of Assessment Issues

In addition to implementing the State policy for assessment of learner level and learning gain, our SDL development agenda includes other assessment issues:

• how to include SDL program planning needs in the State's Guidelines for Writing a Comprehensive Assessment of a Community's Needs and Assets;

• how to identify and assess practitioners' skills needed in SDL services in administration, instruction, counseling, technology "literacy," etc.;

• how to assess learners for readiness and pre-service training/orientation needs as they prepare to enroll in SDL instruction (including assessment of learning styles and technology skills);

• how to assess learner participation in the SDL services, when "attendance" cannot be measured as "seat time".

Implementing the 2002 State Assessment Policy

Initially, the Pilot Sites developed assessment strategies as follows:

ESOL Assessment

The ESOL pilot sites are structured around use of CROSSROADS CAFÉ (Intelecom and Heinle and Heinle Publishers, 1997; includes video, worktexts, photostory readers, partners' guide, assessment package, and teacher's guide). These sites have created tools including an intake/assessment instrument with specific references to CROSSROADS CAFÉ content for in-house placement.

For reporting progress and learning gain to meet state and national reporting requirements, the SDL programs initially administered the commercial assessments used in the parent agency's core classroom program (either the BEST Test or the ESLOA).

Of the two pilots using ESOL assessments, only one was using the BEST prior to FY'03, but still found it necessary to put staff through the statewide training and certification procedures needed to use BEST validly under the new state policy. The other pilot previously used the ESLOA, and thus needed to switch to the BEST Test and meet the required staff training. Also, both of these demonstration sites as well as their host agencies needed training on the REEP assessment.

ASE/Pre-GED

Initially, only one of the three pilot sites was focused on ASE/Pre-GED instruction instead of ESOL. Before the end of the second year of the project, however, both the other Pilot Sites began to include ASE/Pre-GED learners in their service plan, and all were implementing an appropriate multimedia curriculum package (WORKPLACE ESSENTIAL SKILLS; developed and published in 1999 by PBS and KET-TV, currently distributed entirely through KET-TV; includes video, print, web, internet, and teacher guides). Currently, the pilot programs are also using (to some degree) TV411 (ALMA) and GED Connection (KET-TV/PBS) each of which as a similar range of component media resources.

Prior to the 2002 assessment policies, the sites either used the AMES or the ABLE to measure learning gain. Upon publication of the Policy, all three Pilot Sites converted to TABE and sent staff to train on this instrument as required by DOE.

Intake/placement tools were developed for learners in the ASE/Pre-GED programs as well. These include important resources to assess the learner's readiness for computer-based distance learning in terms of both computer literacy and independent learning skills. An orientation for new distance learners following intake offers a chance for them to relate their technical skills to the specific learning program, or to develop skills which are assessed as weak or missing at intake. After this orientation, learners are transitioned into active instructional use of Workplace Essential Skills in the SDL program.

For Further Discussion

Below is a sample of questions, which might deserve further discussion and study.

1. How do we best determine when and how frequently assessments
are administered to SDL learners?

The state policy refers to hours of instruction as seat time or attendance in class. However, in our Pilot Programs, we are counting participation by our SDL learners in a variety of ways, including independent learning supported by video, print, web, and CD-ROM/Software materials. The SDL learners do spend some time in direct face-to-face contact with teachers, and that might be considered the practical equivalent to attendance or seat time in a classroom setting. However, that's only part of the picture. In addition, our SDL learners receive instructional support at a distance, through (for example) mail, e-mail, and telephone. And even beyond that, these learners spend at least (on average) four times as many hours in independent learning "at a distance" supported by the materials mentioned above, than they spend receiving support from an instructor either face to face or at a distance. We are establishing a unique protocol for assessing participation and intensity of services based on this specialized approach to counting instructional intensity in SDL1. Policy for timing of administering assessments should reflect the unique criteria used to measure participation and intensity of services in SDL.

2. What it the best strategy for making the initial assessment a low anxiety, non-threatening event for a newly enrolling SDL learner?

The expectation in classroom-based programs is that the learner may need to become familiarized with the teacher, the class, and the expectations and procedures of the course before taking a test that will directly impact their education as an adult. For classroom programs meeting regularly several days a week, this is a readily manageable timing issue. But SDL offers a different challenge. Currently in the SDL programs, the learner may meet the teacher in an intake/enrollment session. The second meeting may be the orientation to the SDL program, which can involve from 2 to 15 hours of orientation and training, depending on the program and the learner. The next subsequent meeting may be several days or even weeks later, when the learner has completed an initial phase of independent 'supported distance learning.' At this meeting, the learner meets with his/her instructor to get support, orientation, and clarification of any problems. It may be difficult to identify any underlying process of familiarization and environmental comfort-building for the learner in this context, which is dramatically different than the experience in the classroom. At what point in this extended time frame does it make sense to "mark" a point in time when the learner will be familiar enough with the teacher and the services to be comfortable taking the initial assessment?

3. How well does the assessment instrument used for supported distance learners "match" the contents and scope of their instructional program?

In Massachusetts, the statewide adoption of a uniform, single set of assessment tools for ABE and ESOL learners overall required a team of practitioners and assessment specialists to "match" the standardized tests being evaluated to the Massachusetts ABE Curriculum Frameworks. The goal was to locate, or if necessary, develop a standardized assessment tool that actually tests the knowledge and skills outlined in the state Curriculum Frameworks. However, in our SDL pilot programs, the overall scope and content of the instructional program is defined not simply by the state Frameworks but also by the scope and sequence of the commercially published multimedia curriculum packages that our supported distance learner work with. Since as much as 80% of the supported distance learners' participation in the educational program occurs with the use of these commercial multimedia materials (print, video, software, web, internet), should there be a specific effort to assess the "content match" between the assessment and the multimedia curriculum used in SDL programs, as well as with the content and scope of the state Frameworks used overall in Massachusetts ABE?

4. Can a strategy for "distance assessment" for SDL programs be developed that will accommodate the standards and policy of the MA DOE Assessment Guidelines?

Each of our three pilot sites in MA is doing assessment "by the book" according to the state policy. That means that each learner enrolling in SDL services must take the REEP, the BEST or the TABE assessment on site, with directed supervision of a trained practitioner. However, there are already some indications that some of the very factors that "qualify" an adult learner for SDL services versus classroom services may make this on-site, direct assessment a challenge. Those factors include a lack of transportation or child care, unsuitable work schedules, even physical handicaps that make it equally as difficult for a learner to come for an assessment as it would for them to come to class. The current practice is to find ways to overcome those barriers and bring the leaner into the program site for an assessment. But it is reasonable to
look ahead and project a scenario where distance assessment (in some format yet undeveloped) will be as important to the leaner as the SDL program on instruction and learner support itself. We have no doubt that this is a high hurdle to reach, and that it is something for the long range rather than the current or emerging status of SDL, but it remains on the
horizon for further exploration and perhaps developmental effort.

Roger Hooper manages the Massachusetts ABE Distance Learning Project / Adult Literacy Resource Institute in the Graduate College of Education, University of Massachusetts/Boston. He coordinates the Project Partnership that includes SABES, Massachusetts DOE, and three Demonstration Pilot Sites based in the Northern Berkshires (Massachusetts College of Liberal Arts), greater Springfield (International Language Institute with Corporation for Public Management) and Cape Cod (Cape Cod Community College ACCCESS Program).

Originally published in Adventures in Assessment, Volume 15 (Spring 2003),
SABES/World Education, Boston, MA, Copyright 2003.

Funding support for the publication of this document on the Web provided in part by the Ohio State Literacy Resource Center as part of the LINCS Assessment Special Collection.