Final Reflections from Your Editor

Creating, Editing, and Producing the Dream Conference

Loren McGrail
Editor
SABES Central Resource Center / World Education

As a sometime adult literacy practitioner, particularly interested in alternative assessment, reading through all five issues of Adventures in Assessment felt akin to attending a dream conference.
Cathy Luna

It is spring and it is still snowing. The crocuses are shivering but Passover is here and Easter is around the corner. Getting this journal out the door and into the anonymous hands of practitioners around this state and all over the country feels a little more like a nightmare than a dream during this phase of the production cycle but the dream is still alive: the simple belief that a journal dedicated to writings on learner-centered or participatory approaches to assessment and evaluation written by and for practitioners will be of use.

This issue of Adventures in Assessment is the last issue that I will edit. The journal will continue to be published bi-annually by SABES. It is an issue primarily devoted to responses to the journal, all 5 volumes. The field is vast and wide. The authors in volume 6 include a doctoral candidate in a graduate school of education, an ESOL teacher in a community based program, a staff development facilitator, a tutor trainer, a state ABE director and the former assistant director of a national clearinghouse. I invite you to listen and drink in their thoughts and reflections, how and why they have used or adapted the tools they have or how they have used the journal’s participatory principles and philosophy to guide their assessment adventures.

Before introducing each author to you, I would like to ask for your indulgence for a moment while I sing my swan song— reflect and do my own critical assessment of the five volumes and where I hope and think the journal should go in the future.

As stated already Adventures in Assessment is a field-based journal dedicated to writings on learner-centered or participatory approaches to assessment and evaluation. In my job as Literacy Specialist for SABES with the charge to provide technical assistance to adult literacy practitioners and pro- grams, it was my belief that the field needed both a framework for investigating participatory approaches to assessment and a forum for expressing, sharing, and documenting ideas, tools, and questions. From the beginning these two strands were woven together into one cloth: “writing about it”. Both were equally important; the it—learner-centered approaches to assessment—and the writing—documenting inquiry.

“It”
Let me start first with “it’, learner-centered or participatory approaches to assessment. From the beginning the journal was a way to reinforce and sustain energy my workshops on assessment had created- a way to get practitioners to talk and listen to each other across the state. The journal was a beginning point for the development of the toolkit of alternative assessment approaches and tools based on the three phases of the assessment process. And finally, the journal was an attempt to make real some of Susan Lytle’s recommendations to the field of adult education (Lytle, 1988). Lytle stated that funders and legislators use standardized measurements to determine program accountability and effectiveness because they lack good information about the qualitative effects of programs on learner’s lives. She advocated two basic strategies to remedy this situation: the first was to invite a wider participation into the conversation, thus, who better than the literacy practitioners themselves; the second was to conduct program-based practitioner research simultaneously across the country to strengthen these new conceptual frameworks and to exchange and critique innovative practices.

To start this process we needed a framework with clearly articulated guiding principles so that the word “alternative” didn’t just mean anything other than standardized tests. As Susan Lytle (1989) said, “Constructing new images of adults- images built on assumptions of dignity and competence, of literacy as reflective and self-critical practice, and of learning as participatory — requires that we rethink or reconceptualize not only our notions of what counts as literacy but also our methods of inquiry- the processes we use to document and assess learning”. The following principles are a synthesis of the work of Lytle, Fingeret, and Auerbach, my mentors. The reader will note that they are also part of the Massachusetts Participatory Assessment Team’s mission statement:

Principles of Participatory Assessment
1. It must be program-based and learner-centered.
2. It should help the learners achieve their goals.
3. It must build on learner strengths, not deficits.
4. It should be part of the learning experience.
5. It should not be a single procedure but a variety of procedures.
6. It should provide feedback that will lead to better instruction.

In addition to these principles, to engage in truly alternative assessment, we need to include learners as active participants at the center of the process of measurements as “co-investigators in determining their own literacy practices, strengths and strategies” (Lytle, 1988). As I stated in Volume II in talking about portfolio assessment, “If all we do is substitute new multiple measures for old standardized measurements and monitor student progress for diagnostic purposes in terms of identifying strengths and weaknesses in language and content areas, we will not have created a new paradigm.”

Paradigm shifts are not easy. To support this shift we looked at the assessment process in three phases: 1.) Start Up or Intake Activities; 2.) Along the Way or On-going Activities; and 3.) Looking Back or End of Cycle Activities (Auerbach, 1992).

History of the Journal
The journal began in the Fall of 1991. The intention from the beginning was to publish three journals devoted to the three phases of the assessment process. These journals were intended as guides, resources by and for practitioners to select and adapt tools for their own contexts.

By volume II, in addition to introducing the authors, I asked the reader to think about the different “lenses” through which to view and analyze the writings. I posed questions for the reader to consider like “compare the evolution of the forms and tools Paul Trunnell developed for his ABE learners to the way Kathy Brucker developed her tools. How are the processes similar? How do they compare to the kind of anecdotal reporting Janet Isserlis does daily in her classroom? If progress is achieved, for whom is it achieved and by whom?” In retrospect, I see in these words the beginning of a shift in tone, a caution for people not to just adapt someone else’s tool but to remember that first came the toolmaker- the context, then the tool.

In Volume III, Looking Back, I added three new features: What Counts? (math assessment), Voices From the Field (interviews, dialogues, or writings from practitioners who are not classroom teachers, and Letters.

I don’t recall precisely when or who made the decision that the journal should continue beyond the original three. This fact leads me to believe the field had started to count on the journal as a sure thing so then we institutionalized it and said we could publish two a year- one in the fall and one in the spring. However, between Volumes III and IV the journal lost its production editor. This forced SABES to reconsider what kind of editorial support we needed. After much thought, we decided to hire an editor, Rick, with lots of editing, layout and graphic experience, but who was not an ABE practitioner. His task was to redesign the journal so its format was more consistent with and attractive and to do final edits with the authors. These decisions were made in part because the journal was gaining popularity outside the state and we were moving towards selling it. One of the design decisions was to do away with the appendices, which included all the forms and charts practitioners used. Though at first this seemed like a design decision, it really became a shift in what the journal was supposed to do for the field. By integrating the charts, forms, and graphs back into the author’s piece we were hopefully sending the message that you can’t just go out and take someone else’s tool; you have to make it your own. In addition to these graphic changes, we included two articles from practitioners from out of state and a publication review.

Writing

“...Loren asked me, reminded me, encouraged me, helped and revised with me.”
Don Robishaw

“...It formalized a process that had been
developing with much discussion, but little or no documentation... In some cases writing about the process we were using with a particular form codified something that didn’t necessarily make much sense in practice. In others, the opposite took place- something that didn’t make sense became obvious in the process of writing about it. I was able to see it, or dump it altogether.”
Janet Kelly

“We started with an initial chat, followed by a sharing of the forms and feedback and then a more extensive chat—‘the diner dialogue’ where Loren reviewed my work and sent back a framework for preparing and pairing it into an article.”
Paul Trunnell

In addition to providing a framework and a forum for alternative assessment, I saw the creation of a field-based journal as a golden opportunity to put into practice a process approach to writing, a way for practitioners to experience first hand the power of having their writing responded to with non-evaluative feedback. Many of the authors in Lindy Whiton’s research survey in this issue wrote about this experience.

In general I am not surprised by what these authors had to say about the journal and how it helped them reflect upon their practice. I am struck, however, by a few prevailing comments and sentiments. The first is embarassment over how important my asking, prodding, pleading was to get people to write. Comments like “She asked me” and “She asked me to submit” make me feel both a little uncomfortable with my persuasive abilities and at the same time assured me that without that initial push, many wouldn’t have done it. The second is the power of what happens when you write something down. All the authors commented on the self knowledge they gained by the act of writing itself. I had underestimated the power of writing to reflect back to us what we really think and believe. And third, I was surprised to hear, repeatedly, the desire many authors expressed to connect with others so they could get feedback on their own practice— “I also thought other practitioners might think of improvements, and I learn from them.”

I am also struck by some authors’ clarity about the need to get this information out: “I want presentation portfolios to be used everywhere. I think they are marvelous!” and “I wanted to share my belief that self-assessment is not beyond beginning ESL students; that it reduces me and elevates the student to co-creators of the learning environments. I wanted to support the cause of alternative assessment and remind people that there are other options to the TABE.”

All these comments reinforce that writing for the journal has been an effective means for doing inquiry-based staff development as well as developing the knowledge base of what participatory assessment looks like in practice.

So now that you have listened to my own reflections and critical commentary, I would like to invite you to read with pleasure what some of your colleagues across this country who have attended the dream conference have said about the journal.

We begin with “One Step of Inquiry: Documenting the Voices” by Lindy Whiton. Whiton conducted a research project last year to find out if the process of writing for the journal and the product itself was useful staff development and whether it increased the field’s knowledge base. Her documentation on how she conducted her inquiry and what she learned from both the reader and writer surveys confirmed many of her hypotheses.

The voices of teachers in Massachusetts have been heard in Maine. In “Hello, Massachusetts”, Brawders outlines what the vision and goals are for the new nationally-funded Horizon Project. According to Brawders, Maine has chosen portfolio assessment to replace standardized tests. Maine is doing what they call “contextualized portfolio assessment”. By contextual they mean “each adult learner, each teacher, each administrator of an adult education program will develop a progress portfolio of their work for the year.” The programs will then measure their qualitative and quantitative progress in relation to this baseline over a three-year period. What’s important and exciting here is that programs are not measuring against some specified norm or against another but against themselves. As Brawders says, “What counts is the articulation of action steps that can be taken at the end of each year for the purpose of program improvement.”

Program improvement is not the only goal for portfolio assessment as Richard Goldberg tells us in “Portfolios as Alternative Assessments in a Community-based ESL to College Transition Program.” Influenced by David Rosen’s article “The Progress Portfolio” in AiA, Volume 2, and by his own experience in broadcast journalism with students assembling portfolios of their work, Goldberg decided to implement them in the U.S. Department of Education’s Massachusetts English Literacy Demonstration project (MELD). The MELD program in Chinatown consists of three steps: an ESL class, an Adult Basic education class, and then enrollment at Bunker Hill Community College. Goldberg teaches the ABE class and began using portfolios to document writing progress. The turning point in the MELD program’s use of portfolio assessments came when Bunker Hill Community College agreed to use student portfolios as an alternative to its standardized Comprehensive English Language Test (CELT).

Like Goldberg, Byron Barahona also got ideas from the journal on how to adapt and create tools for his volunteer program in northern California. He adds that all the different ideas shown in the journals have influenced his vision of assessment — from diagnosing problems and needs, to measuring the process of learning itself. For Barahona both the articles and the tools pose important questions: “ First and foremost, they make us think and rethink the need and the process involved in evaluating progress. Second, they make us look back to analyze more critically what has been done. Third, they help us reflect upon what can be changed, adapted or implemented.” His article, “Implementing Alternative Assessment Tools” is a wonderful tribute to the authors he has read as well as to the thoughtful process for real adaptation.

In “Images of Participatory Assessment in Adult Education: An Analysis of Adventures in Assessment”, Cathy Luna echoes Barahona when she says that the reports of “experiments” with participatory assessment practices may help practitioners, learners and researchers better understand the nature of critical reflection in adult education. Luna’s careful and thoughtful analysis of the journal focuses attention on the way the authors write about how they involve adult learners in the development, use, and revision of the tools and processes used to assess learner progress and program evaluation. She asks “In what ways can practitioners and learners talk and work together to build frameworks for critical reflection and assessment?” Her research and analysis of the authors’ writings about learner involvement reveal that there is less involvement of learners in the design and revision of assessment practices than in their implementation. This critical feedback is helpful because it points out where our strengths are and gives us direction for where we need to spend more energy.

In addition to these critical reflections are writings that fall under our established departments: What Counts? (math assessment), From the Field, Book Review, and Letters. The reader will note that there are a few other writings as well. Beginning in this issue and hopefully to continue is the start of a new column called “Learning From Experience,” designed to encourage practitioners and learners to write about their own personal experiences being assessed or evaluated. Two other writings that complete this Responding to the Dream Conference journal are a survey completed by a practitioner from West Virginia and the mission statement from the “Transformers”, the Massachusetts Participatory Assessment Team.

In “Out of A Pickle: Setting the Stage For Math”, Martha Merson draws upon her experience with the ABE Math Team teachers, and writes about how to create a “class evaluation that doesn’t hurt self-esteem, where learners’ knowledge about the world gets counted and woven into the learning at hand, where work that is done in the class has a purpose.” She reminds us that we need to educate our students to the alternatives and that we can do this by laying out a strategy for “familiarizing students with a broad view of mathematics, for opening the dialogue about what topics should get covered during math class, for using assessment as an opportunity to build expectations for a new or continuing class.”

Marilyn Gillespie in From the Field reminds us to keep the dialogue going and to try to find better ways to include learners in our conversations. She also challenges us, those who write for the AiA, to reflect upon the conditions that affect our work at large and how they could be changed.
Anne Marie De Martino’s letter to Don Robishaw discusses the impact his article had on her thinking about goal setting. She thanks him for calling attention to the need to place goal setting strategies within a context of the greater philosophy of self-directed learning but disagrees with his statement that “It is very important that the facilitator has had similar life and schooling experiences as the learners to develop solidarity with them.”

In “From Minnow to Overachiever” in Learning From Experience, I write about the lasting effects of being evaluated at different points in my youth and adult life.

Don Robishaw reviews a new book on portfolio assessment called Portfolios in the Writing Classroom. Though all the articles are written by and for teachers working in K-12, he recommends the book to us in Adult Education who are interested in portfolio as both a tool and a pedagogical stance.

And finally, Responding to The Dream Conference ends with the Transformers mission statement and strategies for achieving their goals. The Transformers are a group of ABE and ESL practitioners who are “dedicated to concentrating and coordinating efforts towards broadening the understanding of adult educators, learners, administrators, and funders about participatory principles which are the basis for meaningful assessment and social change.”

Recommendations for the Future
As part of my final reflection, I would like to conclude with some recommendations for the future of the journal. My first recommendation is that we need to figure out a way to bring the learners into our conversation. We need to hear their voices. We need to know if our new conceptual frameworks have merit, if our new approaches to assessing skills are useful and if our attempts to capture their gains meaningful. “We need to risk finding out not just how our learners experience learning but how we help or fail them in that process.” Second, I would also like to figure out a way to document what people do as a group or community. I think we need tools and procedures that not only measure individuals but communities of individuals working towards social change. Third, I would like to see the journal take a political stand and support native language and biliteracy. I would like to see articles in the journal on how to assess native language literacy, not just for diagnostic purposes or as way to establish a baseline for comparing English language acquisition, but as a way to affirm and validate the belief that our learners have a right to be bilingual and bi-literate and that we believe English is a plus, not a substitution. Lastly, I would like us all to practice a little self-reflection on our own past experiences at being assessed or evaluated. Were these positive or negative experiences? Do we now as teachers impose what was done to us on others, and if not, what informs our choice?

In general, we need more critical reflection. As Stephen Brookfield says, we need to take a reflective stance on our practice (both past and present) to be clear about what we stand for or are trying to achieve. We need to pose questions not just seek solutions or create tools. We need to become skillful teachers in our reflective stance on practice and our dance of experimentation and risk-taking.

Sources

Auerbach, Elsa. Making Meaning, Making Change. Delta Systems, Inc., McHenry Illinois. 1992

Brookfield, Stephen. Recognizing Skillful Teaching,” unpublished manuscript, 1993.

This article was published in Adventures in Assessment, Volume 6 (Spring 1994), SABES/World Education, Boston, MA, Copyright 1994.

Funding support for the publication of this document on the Web provided in part by the Ohio State Literacy Resource Center as part of the LINCS Assessment Special Collection.