Creating Change or Creating Accessibility: A Dialogue

Lindy Whiton and Loren McGrail

In January, 1992, the Bureau of Adult Education (BAE) of the Massachusetts Department of Education initiated the GOALS Project, a field test of ten varied components designed to lead to the development of an Accountability System for adult basic education programs in Massachusetts. The purpose of the GOALS Project according to Sandy Brawders, Supervisor of Development, BAE, is to "define a set of indicators of program quality which are educationally sound, useful to students, teachers and funders, and not excessively time-consuming."

Component 3 of the GOALS Project is working with 11 programs using alternative forms of assessment. Lindy Whiton, Coordinator of Component 3, describes the theoretical and philosophical basis of Component 3 as follows:

"We all believe in learner's goals being not only important, but a driving force in our curriculum; thus, our assessment practices are based on the individual goals of the learners as well as the goals of the individual programs. Whether we are teaching ABE, GED, or ESL, we believe that literacy practices have to directly relate to not only the goals of learners, but their immediate lives." (Whiton, Component 3, First Draft, p.1) Based on these beliefs, each of the 11 programs in Component 3 "have developed assessment/evaluation tools that take into consideration who their learners are, who their teachers are, the goals of their learners, their interests, and how they use literacy in their daily lives." (ibid.)

The following conversation with Lindy Whiton was conducted at the beginning of the summer of 1992 with Loren McGrail, editor of Adventures in Assessment. Minimal changes have been made to the original transcript so that the flavor of the dialogue could be retained.

Component 1 of the GOALS Project uses standardized tools to measure student progress. Before we begin talking about Component 3, could you please talk about Component 1?

Component 1 is set up so that you collect profile information on an intake of a student, everything from race to past educational experience to work history .What you do in the very beginning is to give the CASAS* pre-test and you get a score from the CASAS test and then people go about their normal lives and their programs track only participation and atten- dance. Then-I think it's every six weeks, but I'm not positive-they do another CASAS to test whether there has been growth. Now that's one track. The idea is to see whether competency-based tests will determine whether or not a learner is meeting their goals [as defined by competencies]. This is only being tested for ABE learners right now. Then they will compare this information with student participation by looking at attendance scores. They'll be able to look at CASAS quantitative markings and see whether or not learners are reaching their goals, thus [find out if] programs are fulfilling their duties/ responsibilities.

Compared to Component 1, what is the difference in design or intent of Component 3?

Well, there was no way you could measure what happens in a Component 3 program with the tools that they were using to measure Component 1. What we decided was that the method in which we did Component 3 should reflect the theoretical foundations of those [participating] programs. If I was going to run a component that used alternative assessment and participatory approaches or whole language approaches, then I'd better run that component in the same methodological way.

I've completely allowed the programs to determine what we're doing and what we're collecting. When we first got together I led them through a couple of activities just to get them to talk about assessment and evaluation and what the difference is between assessment and evaluation. What do you mean when you talk about evaluation? What do you mean when you talk about assessment? There were a lot of us in the room that day, not only program people, so there were a lot of different perspectives. It was exciting. People fought. I mean there was just a sort of fighting back and forth and really getting into it. The idea was to create a kind of community quickly and trust in that fighting, that it was okay. What it did was layout where people were coming from, their theoretical foundations came out immediately. You can't evaluate, you can't use a test that is totally skills-based to measure a curriculum that is not skills-based. Those kind of issues came out immediately, so that we all knew where we stood in that realm. Then they decided as a group the thing to do next was to meet again and share with each other what each program did on intake, ongoing, and final evaluation of learners, teachers, classrooms, and programs. They shared them all and then they sent me away with the task of putting it together in some sort of order and writing up a paper that explains what their foundation is, what those three pieces are. I used Elsa [Auerbach's] and your stuff [framework for Adventures in Assessment: getting started, ongoing, and looking back] to organize the notebook.

You said there was kind of a discussion about differences in assessment and evaluation. Did you come to consensus about these differences?

No. And we allowed it not to come to consensus. We were all sort of split and in the end made the decision that we all agreed upon what the concepts were, but not the labels for those concepts.

And what were those concepts?

We looked at assessment as something you did [on an] ongoing [basis]-something you did daily like a learning log or a teaching log that you daily put your perceptions on what happened. That was assessment. Evaluation was at the end of the cycle looking at doing miscue analysis, doing a program evaluation of the whole class. What were your strengths? Looking back at larger things...Some people switch the two concepts around so that they were calling evaluation what you and I call assessment.

So you decided not to quibble over the terms, but just made sure those concepts were...

...agreed upon.

Wasn't it true that you decided not to have people other than the panicipants in your component come to the meetings?

The group made the decision to ask that other people not come and that we allow ourselves the privacy to discuss what does happen.... Issues needed to be talked about within the group without anybody overseeing.... Like some of these tools have been developed but not put into practice unfortunately. Programs struggle with how to fit this into part-time teacher status. I think some of the programs feel that they have very strong intake procedures—and they do—and that their curriculum is very good, and they sort of have a hard time teaching math, and they don't have any means of evaluating anything they do. So what they'd like to be able to do is say this is the help we need. This is where we think we are weak. We're definitely strong here. These are not reasons to not get refunded.

So your belief is that they are worried about giving out information and letting...

I think they are worried about it now and the idea is to find a way to open up that dialogue. The answer to that question will then become: "This is who you should call at SABES," or "Here is this resource to help you develop/strengthen this piece.. .

In Component 1, learner goals are matched with competencies from the CASAS competency list. How are these goals identified in Component3?

The majority—this is where the uniqueness of the programs come into play—ask the learners what their goals are. They may start with them at the beginning of a cycle or they may plug them into a certain part later in the cycle so the curriculum is determined by those goals.

So then there's more of a direct relationship between the curriculum and those goals?

The Log School is probably the most direct. When they get a group of people in, they take those goals...that are most in common across all learners [and that] is where they start their curriculum. So that's the theme or the focus that they start in and then it goes on from there. The panicipants direct where the curriculum goes. In the end the students do the evaluation: Did your goals get met? Where do you think the strengths of the program lay? Where do you think it could be stronger? They ask them directly instead of trying to determine some competency that they standardized.

How does a curriculum that is already set upfit in with an approach that tries to find out what the learner's goals are? Wouldn't that be the same problem or configuration as Component 1 where there is something pre-designed to meet something that is coming from the learners?

In Component 3 there are three programs or four where that is true. I think that the Quincy School would say that people's goals are to learn English and that they have a standard way of doing that. One of the reasons that they chose to participate in this particular component is that they're interested in changing their assessment policies.

And changing that attitude?

Yeah. There seem to be reasons to keep it and reasons to get rid of it and they, as a staff, are having that debate. So they asked to be part of that larger debate.

Then not everyone in Component 3 is already doing alternative assessment? There are people who are coming into it because they want to?

Well, Quincy School defines themselves as doing alternative assessment because they made up their own tools. They don't use anything published.

So there are two definitions of alternative assessment?

Right. And what's been real interesting to me is the dialogue that's been going on between the programs... You might be able to use Component 1 on the Quincy School as it stands now but not on Read/Write Now.

Are you saying that because there was no operative definition for alternative assessment to start with, people chose this component because they felt they were doing alternative assessment and because they wanted to make a change?

Yes.

...And that the Department of Education made their selection based on wanting some of those programs who wanted to change to be in the company of others who have already made the change?

Yes. And what I think may be possible—this isn't in writing yet—but I think that what we'll do is once we [complete this research], we'll get ten other programs that are in the same spot who want that change but need help developing their tools and adapting them to their own programs.

Are you saying you all would act as mentors?

Yes.

One of your discoveries is that program-based assessment does not necessarily mean alternative assessment if it's following the same theoretical principles as those of standardized tests which are primarily skills-based. I think this is the middle ground where most of the state is. It's almost as if there is a missing component in the GOALS project-program based assessment-and its showing up in Component 3. Program-based assessment does not necessarily mean alternative assessment. It means alternative to standardized testing. So maybe we should stop using the word "alternative" and use something like "authentic" or "learner centered" or "participatory" to make the distinction.

Yeah. I think that's a good point. I think it is true that when we chose programs to go into Component 3, we chose a few of one kind and a majority of learner-centered.

Do you see a pattern among people who have either been through some university training or influenced by universities in the development of the kind of tools that are developed?

I think the majority of the people in Component 3 have been influenced by either Boston University or the University of Massachusetts. I think that's part of the difference between Components 1 and 3.

Are there other themes or issues that have emerged?

There are two fears the group [has]. One is if you develop these things-tools-are they just going to become standardized? [ Another is: ] How do you teach part-time teachers how to use these types of tools which will [eventually] fit their particular populations? Are we just going to find Janet Kelly's stuff all over the place without any adjustment made to who we are serving?

...Who they're serving and what type of literacy they are practicing? It seems to me one of the other fears, which may or may not be yours, is that people will take tools and impose them upon teaching situations that are not...

...meant to be measured.

In fact, they would be better measured with more standardized tools because of the ways in which they are teaching.

Right, I think that's true.

Maybe that's not a concern of the group's but maybe more of yours?

Well, I think the group is looking at how to... literally, physically create that toolkit and create little abstracts on why this tool works best in this type of situation. You won't have an SRA ** box full of tools, but rather a way that helps people say, "I want to know whether my students are getting the connections between what they're writing and what they're read..."

It seems to me one of the real benefits that Component 3 could give all of the other components is a deeper and richer understanding of what goals are, who the learner is in relationship to their own goals and some really appropriate ways in which that gets measured for the learner's sake. Do you see that as part of your agenda or is that an agenda of anyone in your group? From my viewpoint, that's the piece of research which I think could be very helpful to the Bureau.

What I'm trying to do is create a system that allows people who think like Component 1 to understand the benefit of Component 3. If you change the language and the whole way in which you talk about it entirely and leave nothing for comparison, then you leave no access into your ideas. If what I can do is create this line of data that is comparable in the same way [as the other components] then it gives the Bureau—and everybody—access to what you and I think are the important issues. I've got to create comparable data in a research/quantitative way.

Even though you have a totally different theoretical basis, you still think you can compare data?

I can create access into the more important meat. Once you can get a hold of this under- standing then things like understanding that we are not tracking learners...

...but following their tracks...

...becomes more easily understood by somebody who thinks like this. Otherwise we'll get thrown out as hippies.

Well, it's the classic argument: who needs to change what according to whom? I guess I'm suggesting that your criteria and values and your ways of looking at goals should not change.

I'm not disagreeing with you.

Well you kind of are.

I'm saying that we need to go slower—that we can't change them until they understand us. Before [ we] create change, we have to create accessibility .

* The Comprehensive Adult Student Assessment System is "a comprehensive educational assessment system designed to measure competency-based curriculum for all levels of Adult Basic Education and English as a Second Language, including a pre-vocational curriculum." (CASAS Overview, p.1) CASAS measures functional basic skills in Reading, Math, and Listening Comprehension, utilizing an Item Bank of more than 5,000 multiple-choice test items. CASAS also includes authentic or applied performance measures on oral proficiency, writing skills, pre-employment and work maturity competence, and critical thinking skills." (CASAS, Training Workshop Handouts, 1990, p. 8d)

**Science Research Associates (1968-1986) graded reading passages with multiple choice comprehension questlons.

Originally published in Adventures in Assessment, Volume 3 (April 1992),
SABES/World Education, Boston, MA, Copyright 2003.

Funding support for the publication of this document on the Web provided in part by the Ohio State Literacy Resource Center as part of the LINCS Assessment Special Collection.