WMass Assessment Group -Tackling the Sticky Issues

Patricia Mew and Paul Hyry

• WIA and NRS are facts of our lives. We need to learn how to live with them and to develop inventive ways of handling the accountability and assessment issues they create.
• The Massachusetts ABE system affords us the opportunity to be proactive in the assessment dilemma, to develop some assessment practices that satisfy our funders, our practitioners and our students.
• Both standardized and alternative forms of assessment are useful and necessary.
• We can approach assessment as a way to link SPLs (Student Performance Levels) or GLEs (Grade Level Equivalents), classroom curriculum and the standards in the Massachusetts Curriculum Frameworks documents we are addressing.
• We should try to educate our students in the issues and challenges of assessment to make them partners in the assessment journey.

These are the assumptions with which the Western Mass. Assessment Study Group began its venture in January 2001. The idea was to bring together practitioners from our region to learn about a variety of assessment methodologies, decide what we wanted to investigate further, and develop processes and tools that would be useful for our programs. We also thought the time was ripe for better understanding, and perhaps ultimately influencing, the direction of assessment and accountability in Massachusetts. The PAWG (Performance Accountability Working Group), a group convened by the state director of Adult Basic Education to decide how Massachusetts would meet its accountability obligations, was just beginning its own process. We hoped our parallel process would help us keep abreast of the work the PAWG was doing at the same time as we educated ourselves about assessment based on our own interests.

The programs represented in the study group, as well as Western Massachusetts as a region, have their own challenges. We quickly learned upon advertising the group that both ESOL and ABE practitioners were interested in the assessment problem. This may be due in part to the fact that all programs involved had received state Curriculum Frameworks (CF) grants requiring them to address assessment in their curriculum development work. Regardless, these practitioners brought a diversity of needs and skills to the group. Some had been grappling with assessment and curriculum development using the Massachusetts Frameworks for years. Others were looking at that connection for the first time. Some were experts in the Curriculum Frameworks grant process; others were just beginning that process. In addition, Western Massachusetts has many different types of programs — rural, urban, multi-sited, correctional, community-based, large, small — all of which have their issues to overcome regarding the relationship between assessment and program design.

Our work took us down many paths over six months. We began by learning more about the National Reporting System and the Workforce Investment
Act upon which the NRS is based. During the course of our process, we also hosted guest presentations on standardized testing issues from both ESOL and the ABE perspectives. But our most important and fruitful work was in the area of Performance-Based Assessment (PBA), and more specifically in the development of rubrics. We did this for several reasons. First, many of the practitioners involved were critical of the limitations of the standardized tests used most in our state. Second, we also all understood that the PAWG, whose work was to last about 18 months, might possibly head in the direction of Performance-Based Assessment, and the group wanted to travel the same route. Finally, we hoped that some of our findings might, in fact, be useful to the PAWG in the context of its lengthier, more intensive process.

As mentioned above, rubric development was a central and practical task in which we engaged ourselves. Our interest was in developing rubrics that would help make a crucial, and heretofore undefined, link between federally defined skills and levels (SPLs and GLEs) and the learning concepts, strands, and standards elaborated in the Massachusetts ABE Curriculum Frameworks documents. This work proved to be incredibly complex more complex, in fact, than we had originally imagined. First, there was the task of understanding rubrics and where they fit into the wider picture of Performance-Based Assessment. Then we examined a variety of rubrics created by other practitioners, programs, and states. Finally, since we found none that explicitly met our needs around the MA Curriculum Frameworks or similar standards, we decided to create our own rubrics, trying in the process to build links between these documents and the qualitative assessment work to which we were so committed. At the same time, we tried to address the NRS performance levels in the categories we included.

This work resulted, first of all, in group products, including several flow charts about the overall process of Performance-Based Assessment and two rubrics - one for ESOL and one for ABE — that link MA Curriculum Frameworks standards to a specific student learning task (development and delivery of an oral presentation). A second result was the development of increased capacity to share assessment theory and develop locally customized rubrics and related tools. Finally, we developed a series of questions and recommendations for our state Department of Education and the PAWG. The most crucial of these are:

How do we create valid and reliable Performance-Based Assessment strategies and tools that link student tasks to Curriculum Frameworks standards (which are defined qualitatively) and, simultaneously, to the quantitatively-defined Federal SPL and GLE levels required by the NRS?

How can a classroom teacher assess in the way the NRS dictates we assess? (Even with standardized testing, we can only assess a single performance at a given time.)

Assessment should be done in concert with curriculum development, but at this moment in the development of ABE in Massachusetts, these two things (assessment and curriculum development) are quite far apart. Once the PAWG reaches its conclusions about how ABE/ESOL assessment should be carried out in Massachusetts, there will need to be an intense training process (supported by funding streams) for teachers to help them learn how to assess students accurately using PAWG-generated processes. This training should include understanding how to move between the CF documents, the assessment process, and the assignment of federal levels to student performances.

When the Western Massachusetts Assessment Study Group finished its first year of work last June, members were appreciative of the chance to learn a great deal together. At the same time, we were clear that we had only just begun the process. We were discovering the possible relationships between assessment practices that are valid, reliable, and pragmatically feasible
in light of multiple demands on learners' and teachers' time. We were also
discovering the linkage of these practices to state and federal policies to which these processes need to be accountable. As our work progresses this year, we are excited about continuing together toward the development of solutions to the assessment puzzle.

Patricia Mew first worked as an ABE/GED and creative writing teacher at the Hampshire County House of Correction in Massachusetts. She is currently the Curriculum Development and Assessment Coordinator for SABES/West, where she has worked providing program and staff development since 1994. Pat co-facilitates the WMass Assessment Study Group with Paul Hyry.

Paul Hyry has worked in adult education in Holyoke, MA since 1994 as an ESOL teacher, curriculum developer, program director, and local ABE collaborative coordinator. Paul co-facilitates the WMass Assessment Study Group with Patricia Mew.

ESOL Presentation Rubric | ABE Oral Presentation Rubric

Originally published in Adventures in Assessment, Volume 14 (Spring 2002), SABES/World Education, Boston, MA, Copyright 2002.

Funding support for the publication of this document on the Web provided in part by the Ohio State Literacy Resource Center as part of the LINCS Assessment Special Collection.