What Counts?

The Right Answer: There Is More Than One
Adult Diploma Program Math Research

Susan Barnard, Kenneth Tamarkin
SCALE
Somerville, MA

The Adult Diploma Math assessment project was a collaboration of Kenneth Tamarkin and Susan Barnard. Kenny is a member of the Math team and Susan is the Program Administrator for the Adult Diploma Program (ADP) at SCALE. We wanted to update SCALE’s ADP Math Assessment to reflect the NCTM standards, specifically math as problem solving.

The premise of the ADP is to earn a high school diploma by demonstrating proficiencies in life-skills based competencies that have components of reading, writing and math interwoven throughout the curriculum. In order to start the ADP process, students must have an eighth grade reading comprehension level (Form 4B of Nelson) if a native English speaker, and a 7th grade reading level if a non-native speaker; an eighth grade math level; and the ability to write a paragraph with 90% accuracy in structure, grammar and spelling.

Over the years, we have identified critical thinking as the weakest skill area. We found that we can teach to the ADP math entrance test and students can quickly pass. But given a similar calculation, or problem out of context or with a change in format, students are not transferring their skills. This project was an opportunity to see how students would respond to a questions that could have more than one answer. After all, isn’t that what life is all about?

In our plan to implement the project, we targeted students who attended ADP intake/orientation sessions (during which they completed an ADP math pre-test), so we could establish some indication of the person’s current mathematical abilities. As we became more involved in the project, however, our target group expanded to include an ABE class, and ADP preparation class, a class at Quinsigamond Community College, and a few staff members. In total we involved 36 participants in our project: 30 students; 6 staff. The profile of the students varied greatly in reading levels, nationalities, ages, and educational backgrounds. The common factors were that all students were over 18 years of age, all had the goal of achieving a high school credential, and all were motivated to attend classes without stipends.

To address our goal of including more problem-solving skills in the ADP math curriculum, we developed a set of six open-ended math questions, all of which had a range of correct answers. In order to focus on problem solving rather that computational skills, we wrote all the questions so that they could be correctly answered using whole numbers, though some participants chose to use decimals or fractions in answering some problems. The directions stated that a problem could have more than one answer, but only one answer was required to be written down. Our criteria for success were to have questions that both drew the participants in and revealed their thinking; A follow up interview sheet was developed to gather individual input from the participants—including their reaction to the test questions, the test format and the test purpose.

We hoped to find out whether they considered the new questions fair or unfair, easy or hard, and clear or confusing. We particularly wanted to find out if the participants recognized that the new questions were meant to challenge their thinking skills. Each participant was individually interviewed and the answers recorded on the interview sheet.

The chart on the last page provides an overview of our results, the score on the ADP math pre-test, and the new questions, along with a graph displaying the relationship of pre-test and new questions scores.

To understand the relevancy between this test and the six new questions we asked, it is helpful to understand the scoring. In the pre-test, the highest possible score is 28. If students score between 18-28, a quick 1-3 hour math review usually brings their math abilities to the level where they can pass the math entrance diagnostic; a score of 10-17 usually requires an ABE class from 3 months to a year.

On our six math questions, the highest score could be 6, with credit given for partially correct answers. As we began scoring the six questions, we found it necessary that one person do the scoring in order to achieve consistency. Otherwise, with more than one answer possible and partial credit given, each scorer’s bias could distort the scores. As a follow up task to this research, we need to standardize scoring criteria, especially for partial credit.

When asking people to participate, we said we were thinking about revising the ADP math curriculum and needed their help and input. None of the test sessions were times; students were assured there was no penalty for guessing. The participants’ willingness to work with us and give us honest and complete feedback was outstanding. Word of the project spread and we had teachers asking if they could try the questions with their students. ( Those results are not included in our findings.)

We decided to write up the results of our findings separately and then met to discuss our findings. To our surprise, our write ups were strikingly similar.

Question 1: You have $100 to share among three people. You don’t have to divide the money equally, but no person can have less than $25 . How much does each person have?

We intentionally picked numbers that couldn’t be divided evenly and stated that the three people did not have to get the same amount. Yet, we found that most participants wanted to be fair by dividing the money equally in three ways, giving each person $33, or even $33,33, often losing sight that the amounts, once divided, should add up to $100. One participant, while being interviewed, commented, “ of course we divided everything evenly. We’re parents.

Question 2: You go to the grocery store with $25 in our pocket to buy milk, orange juice, bread and shampoo. What other items would you buy so that you would go home with no less that $10.00 in your pocket? How much money would you have left?

Grocery items:
bread $1
cookies $2
spaghetti $1
tooth paste $2
cat food $1
cucumbers $1
cheese $4
milk $2
pancake mix $1
orange juice $2
fish $4
ice cream $3
lettuce $1
lemons $1
shampoo $4

People seemed to particularly enjoy this question, probably because people love to shop. If they bought only the required items, they would have $16 left, which was a correct answer. Yet, people seemed to focus on the $10, opting to buy more, going home with exactly that amount. Other people did not organize or label their work and became con-fused, sometimes thinking $10 more needed to be spent and forgetting they needed to go home with $10. Overall, this question had the highest rate of correct answers.

Question 3: We showed them two jars and said: There are 10 pieces of candy in one jar. Estimate the number of pieces of candy in the other jar.
The other jar contained 105 pieces of candy wrapped in cellophane. The idea was to see if people’s answers would be in the ballpark. We decided a range of 30 pieces in either direction would be an accepted answer (because Kenny guessed 130 and didn’t want to be wrong.) A surpris-ing number of people got this answer wrong; quite a few answered 70. The problem was the paper added extra volume and the candy could be tightly or loosely packed, making it difficult to estimate accurately. Participants who guessed 50 or below also had severe problems with computation and seemed to lack basic number sense. We would alter this question if we make it a permanent part of ADP assessment. We would use M&Ms, beans or some other unwrapped, regularly shaped, small object as the standard so that the volume estimation could be more consistent.

Question 4: Draw a rectangle whose perimeter is 8 inches.
In this question, participants had to know what a rectangle and perimeter are. The standard approach to this question is: What is the perimeter of a rectangle whose sides are 3, 1, 3, 1? The question also emerged, “Is a square a rectangle? Yes, it is. There were two whole number correct answers: A square with 2 inch sides and a 3 inch by 1 inch rectangle. Most people drew the 3 by 1 rectangle. We also accepted as correct a 2 1/2 by 1- 1/2 inch rectangle, since we did not require that the answer had to be in whole numbers. One-third credit was deducted if the lengths of the sides of the rectangle were not labeled. If we implemented this question, we would directly ask for the sides to be labeled, since this requirement was not entirely clear. A number of people did not even attempt this question, considering it too hard. Others drew 8 by 1 inch or 2 by 4 inch rectangles, mistaking area for perimeter.

Question 5: A room is 15' by 12'. You need to place a wood burning stove that is 3' x 2' in the room. The stove must be at least 1' from any wall. Draw a floor plan that shows where the stove can be located.
This question encouraged more creativity. Although there were many correct answers to this question, many participants did not even attempt it, again stating that the question was too hard. Some people did draw good floor plans, but did not label the dimensions, missing the importance of clearly communicating their conclusions. During the interview, participants said they wondered if the plan should be drawn to scale or if they should label things. We would alter the question to make clear that the dimensions and distances need to be written on the diagram.

Question 6: 250 - 74 + 120 = 296 . Using these numbers, write a word problem.
This question also required a bit more creativity and the answers were fun to read. Some were quite entertaining and creative, as well as being correct, but many people had difficulty writ-ing an accurate word problem. Partici-pants had the most difficulty phrasing a question that would lead someone to write an answer. Instead, many included the answer in their word problem. For example, one word problem said, “I have 250 $ in the Bank. I took 74 from it to pay my bill, and the next week I deposit 120 now my balance from the bank is 296 $.” We feel that clarity in the directions is needed. Something like: The answer should not be stated in your word problem.

On our math survey we asked participants to rate the problems from 1 to 5 in the following categories: Fair to Unfair, Easy to Hard and Clear to Confusing. Most students thought that the questions were Fair and Clear, responding to these questions with a 1 or a 2. The Easy to Hard question mostly received 3’s.

During the interviews, people who did best on the traditional test and very well on the new questions tended to be particularly enthusiastic. Some comments were “It makes you think”, “It was fun.” The few negative reactions came mostly from people who had done poorly on both the traditional test and the experimental questions. They had comments such as “it was confusing,” and a number of these poor math achievers were unsure exactly why we were trying to do this project. However, overall, we were pleased to find out that almost everybody was very cooperative and actually enjoyed being asked for their input before changes were implemented.

In analyzing our statistics, we found that of the 14 participants who got 1/2 or less of the ADP pretest correct (14 or below), only 4 got 1/2 (3 or more) correct on the new questions; only one got 4 out of 6 correct. Of those 16 participants who got more than 1/2 correct on the ADP pretest (15 to 28), only 2 scored below 1/2 correct (less than 3) on the new questions.

The results raise some interesting questions. Is there a significant correlation between computational skills and reasoning skills? Can someone who was unsuccessful learning math in a computation-oriented program learn more effectively in a problem-solving oriented program?

The open-ended questions were definitely more interesting to score. The variety of answers and approaches gave great insight into how the student was thinking. The wrong answers were particularly helpful in showing how an individual approached a problem, and also when the working of the problem was open to misinterpretation. All our questions met our criteria; we just need to refine and revise our directions.

Our conclusion is that we should proceed with altering the ADP tests and curriculum to include open-ended questions and problem-solving skills. We believe this will create a richer, more relevant experience that will translate well into improved functioning and critical thinking in everyday life. We also thought that it was important to give people options in the testing situation, since that more closely reflects real life than exclusively having questions with only one correct answer. The correlation that we observed between achievement on the computation-oriented exam and the problem solving oriented questions also suggests to us that there is a role for effective computational instruction and that progress in problem-solving skills can indeed be meaningfully measured and evaluated.

Reprinted from the Mass ABE Standards, with permission.

This article was published in Adventures in Assessment, Volume 7 (December 1994), SABES/World Education, Boston, MA, Copyright 1994.

Funding support for the publication of this document on the Web provided in part by the Ohio State Literacy Resource Center as part of the LINCS Assessment Special Collection.