Assessment, the SMARTT System, and the National Reporting System (NRS): How do these interact? Where do they intersect? Where is Massachusetts heading in terms of assessment? All of these issues are ones that are on the minds of ABE practi-tioners in Massachusetts. It was the focus of my conversation with Bob Bickerton, State Director of Adult Education, and Donna Cornellier, Project Manager for the SMARTT (the Management Information System for ABE programs) Team. Our discussion fell into three broad categories:

1. Current performance accountability (including assessment) policy in Massachusetts and the impact of the NRS on it
2. The impact of the policy on teachers
3. The impact of the policy on learners

Current Performance Accountability Policy

CC: Can you describe Massachusetts' current performance accountability policy?

BB: We believe that the primary purpose of performance accountability, including assessment, is program improvement. We believe that we are ultimately accountable to students and to the goals that they have set for themselves. Assessment is a way to measure progress towards goals, and in particular the goal of educational gain.

Massachusetts has not mandated any one assessment tool. What we have done is require that programs give an initial, a mid-year, and a final assessment for each enrolled student using any tool that the ABE program has chosen. In order to comply with new requirements under the federal "Workforce Investment Act" (Title II of which is "Adult Education and Family Literacy"), we have also asked that each program submit a "Crosswalk form" which documents how their current assessment tool aligns with grade level equivalents (GLEs) and student performance levels (SPLs) as defined in the National Reporting System.

CC: Can you say something about the crosswalk form? What is a crosswalk? What were programs asked to do and why?

BB: Currently we are allowing programs to explore their assessments and think about how to strengthen them. We're not telling them to change the tools they are using as long as they can provide us with a crosswalk that says we really do assess people in a thoughtful systematic manner and if someone scores this way on our assessment, they are presumed to be at a specific grade level equivalent or at a specific student performance level. The crosswalk is a form we asked programs to submit to document this.

DC: The assessment crosswalks we now have on file for programs document our state's current standardized assessment procedure. In January, we will be convening the task force to look at and decide on a process that will replace the crosswalks, starting with program year 2003.

CC: Some states are choosing to mandate one standardized test to use across the state. Why hasn't Massachusetts chosen to do that?

BB: Because any time we consider the pros and cons of any of the commercially available tests, the cons outweigh the pros. With any standardized test, we need to look at the questions of how does this fit with the learning/content standards included in the Massachusetts ABE Curriculum Frameworks and how does it fit with what we're teaching? This "alignment" between curriculum and assessment is the key ingredient in determining whether the assess-ment is "valid" -- a requirement of the NRS and just plain good educational practice. That is why we are convening a task force to look at these issues.

CC: Can you tell us more about the Performance Accountability Working Group?

BB: The Department will be issuing a broad invitation to practitioners to join with us in developing a consistent, standardized system of assessment for our state. We would like people who have done work in the area of performance accountability, includ-ing assessment to join us in this effort. The Performance Account-ability Working Group will begin in January with the goal of having a standardized assessment procedure in place by July 2002.

CC: What determines whether an assessment is "valid" and "reliable?"

BB: Valid means the assessment measures what it is supposed to measure. Reliability measures the consistency and stability of assessment scores. When we talk of a valid and reliable measure of whether someone achieved a countable goal (e.g., "got a job"), that is a bit different. Whether that measure of the goal is valid and reliable is usually about clear definitions and verification. Then the issue becomes how good are we at reporting what students tell us they've achieved.

CC: How is NRS and its requirements impacting Massachusetts Assessment Policy?

BB: We believe that NRS is a subset of our vision for the Massachusetts performance accountability system, that is, being held accountable to students and to the goals that they have for themselves. NRS is not the main motivator behind what we ask programs to document through SMARTT. NRS requires a standard-ized system of accountability and has sped up the timeline, but assessment and accountability are issues that we have been studying and working on for a long time.

A key question for me is who do we want to control and own accountability in adult ed? It is true that there is some information that we must collect because of the NRS, but in this state, we collect more than NRS requires and what is driving that is what's most important. It's not NRS. In fact, people often ask me "Why are we collecting all this stuff?" If we collect only what NRS requires, then what we are doing is saying that is what's most important in adult education-data on getting people diplomas, getting people into jobs, post secondary education, and/or training. This would de facto say that's what's most important because that is what we are measur-ing; that is what we're putting the spotlight on. For years, we have been saying we don't want our students to be reduced to one dimension. We want to support all the reasons that people come to us in adult ed. If we don't ask the questions and record what's happening in those domains, then we run the risk of those things dis- appearing. Things like, "becoming a citizen," "helping my kids with their homework," etc. Do we want to allow a publication from a funding source to say this is what's at the center of accountability or do we let what comes from our students determine what's at the center of accountability? Who is driving this bus? It is our belief that it is the students -- not the feds, not our office.

CC: There seems to be some confusion around why the state is requiring three assessments per year. NRS actually only requires two assessments per year. Can you clarify this?

DC: Since we know that about 60% of ABE students across the state leave before the end of a fiscal year (about 30% because they accom-plished their goal(s) and another 30% who drop-out before accom-plishing any goal), we know that for many students we would not be able to get a final assessment (the second assessment that is required by NRS and which we feel is important to document the success of our services). If we wait until the waning weeks of the fiscal year to conduct a second assessment, we will lose any mea-sure of their educational progress. So what we've done is require a mid-year assessment with the result that assessments need to be admini-stered about once every four months for each student.

CC: How does the NRS relate to our Curriculum Frameworks in Massachusetts?

BB: The NRS says we must measure educational gain. The Curriculum Frameworks answer the question "Gain in what?" The Curriculum Frameworks are our attempt to define the universe of content which is an issue the NRS is largely silent on. When you don't know the answer to what is the content and skills we are measuring gain in, it often becomes, by default, the content of what is contained on the GED test. What we have done in our state is say that there is life beyond the GED, that the table of contents of a GED textbook does not define our curriculum.

Top of Page

Impact on Teachers

CC: How do you think the National Reporting System will impact teachers in the adult education system?

DC: I think the key is that everyone understand that this is not about penalizing teachers or programs. It's about improving the system.

BB: I think this presents teachers with a choice. Will they be acted upon by the Massachusetts ABE performance accountability system or will they take some control of the process? There are at least two places where they can do that. The most accessible one is when teachers begin to use the information that they put into the account-ability (SMARTT) system to inform practice and to see how things are and aren't working with students. We are not suggesting that the accountability system is the whole universe of feedback. There are a lot of ways that teachers get feed-back, but we need to get to a place where teachers include this as one of the tools they use. The second way they can become involved is to become involved on the performance accountability working group we talked about before or in their own peer groups (e.g., during staff meetings) so that they have some ownership over the direction this goes in.

CC: Teachers of literacy level students and students with learning disabilities are concerned that NRS will have a negative impact on them and on their students. They are asking how they can show the subtle gains with literacy level students? Can you speak to that concern?

BB: The standards that we set must be appropriate to the level and the circumstances of the students we serve. We will be using real data of how the system is working to help set benchmarks for different population subgroups -- whenever there is a significant difference in the participation and performance characteristics of that group. This could turn out to be the case for students with certain disabilities, or for homeless adults, single parents, etc. This is why it is crucial to have data we can rely on. If people give us data that may be overly ambitious (e.g., "inflated") about how many grade level equivalents students gain in a certain amount of time, and that data isn't accurate, we could end up setting unrealistic benchmarks -- and then everyone suffers! But clearly we will take into account population subgroups when setting benchmarks.

DC: It is also important for people to understand that the performance system does NOT stipulate the classes people are placed in. In some places, people are setting up six classes to match the six levels of the NRS. The classes you set up should be based on the needs of your students and your programs; they should not be set up simply to mirror the reporting system. We don't expect students to move from one class to the next necessarily in one reporting cycle. What we do look at is their assessments and their progress on those.

Impact on Students

CC: How do you think the NRS will impact adult learners in Massachusetts?

BB: I think if we do a good job at putting students in the drivers' seat of accountability, this could have a very positive impact. When we talk to students about these issues, as Anne Serino of Operation Bootstrap in Lynn has done in her program, we find out what students really want to know. They say things like "I'd like to know more about how long this is going to take, and I'd like to know how I'm really doing... I like hearing that I'm doing well, but give me a measure that says where am I compared to where I need to get." There are some people who believe that students will get a real benefit out of this. It depends on whether we keep them in the driver's seat.

CC: Will students end up penalized if they are not making progress fast enough? Some program feel pressured to produce measurable gain, and, as a result, students who struggle academically or have social issues that interfere with attendance get dropped from the program. These students represent some of our neediest clients. How will we deal with this issue given the re-quirements of the accountability system?

BB: As long as students have the capacity to learn how to read and write and are working seriously toward meeting their goals, we should not set artificial time limits to their learning. We ALL learn at different rates! One issue that the performance standards taskforce addressed was the issue of retention, and we all agreed that the definition of retention should not be continuous attendance but should include what people refer to as "stopping out." What we had a harder time with was defining what stopping out was. Did it mean stopping out of the program for a couple of weeks, a few months, a year? It is clear that we need to acknowledge the issue of students who need to stop out of a program and don't define themselves as dropping out.

DC: One of the reports I do is on total attendance. If they came, dropped out, and came, we take their total hours. The SMARTT system is set up to answer questions like that.

CC: Older learners sometimes come into adult education without employ-ment goals or family literacy goals but simply to learn to read and write better for their own satisfaction. Will there be room for these students under the new accountability system?

DC: Our system in Massachusetts is broader than the NRS, and we think that what the NRS refers to as the secondary descriptors are just as important as the goals of employ-ment and family literacy. Any goal can be a person's primary goal, whether its on our list of goals or not.

BB: People (our students) set the education-related goal that is important to them and we need to be ready to respond. The goals that they set are the ones we're accountable to. The list from NRS is a subset of a larger list that the students set themselves (the list of goals in the SMARTT database reflect seven years of collecting student articulated goals for their learning). After they set a goal for themselves, they should be asked if they think they can achieve that within a year. Helping set shorter-term goals is an impor-tant part of the process so that the person can get a sense of progress and mobility. What are the things that we can do within a year that will let you know that you are making progress? That is the question we need to ask.

Final Questions

CC: What do you think is the biggest misconception about the National Reporting System?

BB: That it is driving accountability in Massachusetts. It isn't. Students are.

CC: What do you think is the biggest misconception about the SMARTT system?

BB: That it asks a lot of pointless questions. Actually we thought hard about what questions it asks. For example, we ask if a person is a single parent not because we want to intrude on people's lives, but because we believe that single parents might be a group whose performance profile might significantly differ from other groups -- which would result in setting different performance benchmark(s) for that group. It's important to ask it so that we don't "cream" the crop and simply take students who we think will make us look good. Creaming in our field would be disastrous and would set up what they call "perverse incentives," where we end up creating incentives to do the very things we know will hurt our students -- like enrolling only those we know will quickly succeed. There really are reasons behind what we ask for in the SMARTT system.

CC: Do you have anything else to add that we didn't cover today?

BB: Yes, that we think its great that Field Notes is doing this issue.

DC: That the most important thing is to get the information out so that we can clear up misconceptions. Robert Foreman and I are the NRS trainers for the state. I would be glad to answer questions from people on this. People are welcome to email me with questions at dcornellier@doe.mass.edu

Cathy Coleman is a Staff Development Specialist at SABES/World Education. She can be reached by e-mail at ccoleman@worlded.org