Discussion of Findings
|
Three methods of assessment were used in this study. The first, which was administered with Form A before the study and Form B after the study, was the Group Reading Assessment and Diagnostic Evaluation (GRADE) (Williams, 2001). The GRADE is a norm-referenced comprehensive assessment administered in a group setting for 45-90 minutes as either one or multiple sessions. The untimed subtests may be administered as a battery or individual measures. Scores can be interpreted as percentile ranks, standard scores, grade equivalents, normal curve equivalents, stanines, and growth scale values. This study focused on the growth made in the subtests of sentence and passage comprehension and was measured using stanines. The sentence comprehension subtest measured the student’s ability to comprehend a sentence as a complete thought, by determining the missing word based on the context or meaning. The passage comprehension subtest measured comprehension of various selections through multiple choice questions, which incorporated the metacognitive strategies of clarifying, questioning, summarizing, and predicting. The second method of assessment the Developmental Reading Assessment 4-8 (DRA2) (Beaver & Carter, 2006) was designed to assess the reading achievement of on-level and struggling readers in grades 4 through 8 in the areas of reading engagement, oral reading fluency, and comprehension. Its formative design allows teachers to monitor and document changes in student achievement as often as needed. The assessment was administered both before and after the study with two different nonfiction selections. It was conducted in four steps: the student reading survey, a one-on-one student reading conference, an independent student work session, and an analysis of student performance. In the first step, each student completed a survey which determined levels of engagement through the documentation of personal reading habits within the last few months. Each student was also asked to self assess his/her strengths and current goals as a reader as well as describe a plan to implement those goals to become a better reader. During the second step, a one-to-one student reading conference, the evaluator documented the student’s oral reading fluency in terms of expression, phrasing, rate, and accuracy to determine if the selected text level matched the student’s independent reading level. Step three required the student to silently read the remainder of the selected text as well as complete a written response activity. This portion of the assessment evaluated independent reading skills and the ability to express understanding of that text in writing. In the last step, the evaluator analyzed student performance by examining oral reading behaviors and written responses, and determined individual scores and stages (intervention, instructional, independent, and advanced) by selecting descriptors on the DRA 2 continuum. Once again, for the purposes of this study, the focus was mainly upon the growth made in the comprehension component of this assessment.
As the third method of assessment, each of the 25 Questioning the Author (QtA) sessions were taped for the purpose of recording the number of student responses per session as well as a means for the researcher to take anecdotal notes in order to more precisely and effectively document student response. The review of these taped sessions added a unique and noteworthy perspective in that it allowed the researcher to focus solely and candidly on evaluating the interaction between the instructor and the students. In 1978, Vygotsky alleged an individual’s “zone of proximal development” to be the difference between a child’s actual level of development, which involved independent functioning, and the child’s potential level of development, which involved adult or peer guidance (Harris & Hodges, 1995). This implied that children would be able to reach a higher level of functioning with help than they would independently (Forman & Cazden, 2004), and concluded that acquired skills are primarily the result of thinking that is internalized after practice with another individual and are critical to a child’s cognitive development. (Pressley, 2006). Vygotsky’s work was seminal in that it inspired educators to teach cognitive skills, which were at the farthest tip of a child’s zone of proximal development. With adult support, even those cognitive skills not fully developed, like comprehension strategies, could be effectively addressed with adult support (Pressley, 2006). The type of questioning presented in the intervention QtA enhances comprehension by actively engaging students to collaborate with the teacher and each other, inspiring a deeper insight into author’s purpose and text meaning, thus improving comprehension. Assessment of this comprehension enhancement, however, cannot be measured solely by standardized testing. Many additional factors such as positive changes in student motivation, social interaction, confidence, and critical thinking are sometimes very subtle, at times not effectively measurable, but are undoubtedly crucial in developing a deeper understanding of the text.
On the GRADE, most of the students scored in the same stanine for both pre and posttests. Only one student decreased from a stanine of 9 to a stanine of 8, and 2 students increased, one from a stanine of 7 to a stanine of 9, and the other from a stanine of 8 to a stanine of 9. The rest remained the same. On the DRA 2, in the area of engagement, 1 student moved from the independent to advanced range, 1 student moved from the instructional to independent range, and the other 5 students remained in the independent range. There were no students who scored lower in the posttest than the pretest. All fluency scores indicated an increase. Four students moved from the independent to the advanced range, 2 from the instructional to the independent range, and 1 remained in the independent range. Participant comprehension scores varied. Four remained in the independent range, 2 remained in the instructional range, and 1 moved from the instructional to the independent range.
Summary of Findings by Subject
Subject A’s results from the GRADE indicated only a 1 point difference in raw scores between the pretest and the posttest. Her stanine score of 9 remained unchanged. The two subtests of the GRADE that were considered when examining the feature of comprehension were sentence comprehension and passage comprehension. Subject A showed improvement in sentence comprehension with a stanine of 7 on the pretesting and a stanine of 9 on posttesting. Her stanine score of 9 remained the same for both pretest and posttest of passage comprehension.
Subject A’s score in the area of comprehension on the DRA 2 also showed improvement, as she moved to a higher level in the independent performance area on the continuum.
Subject B’s results from the GRADE indicated a significant point increase in raw scores between the pretest and the posttest. His stanine score was 7 on the pretest and 9 on the posttest. The two subtests of the GRADE that were considered when examining the feature of comprehension were sentence comprehension and passage comprehension. Subject B showed significant improvement in sentence comprehension with a stanine of 6 on the pretest and a stanine of 9 on posttest. His stanine score of 7 on the passage comprehension pretest increased to a 9 on the posttest.
Subject B’s score in the area of comprehension on the DRA 2 did not show improvement, as he moved to a lower level in the instructional performance area on the continuum.
Subject C’s results from the GRADE indicated no change in raw score between the pretest and the posttest. His stanine score of 7 remained constant as well. The two subtests of the GRADE that were considered when examining the feature of comprehension were sentence comprehension and passage comprehension. Subject C showed improvement in sentence comprehension with a stanine of 5 on the pretest and a stanine of 6 on posttest. His stanine score of 7 on the passage comprehension pretest remained constant on the posttest.
Subject C’s score in the area of comprehension on the DRA 2 indicated slight improvement, and he remained in the independent performance area on the continuum.
Subject D’s results from the GRADE indicated no change in raw score between the pretest and the posttest. His stanine score of 9 remained constant as well. The two subtests of the GRADE that were considered when examining the feature of comprehension were sentence comprehension and passage comprehension. Subject D’s raw scores as well as stanines remained the same for both the pretests and posttests. He received a stanine score of 7 on the sentence comprehension and a stanine of 9 on the passage comprehension.
Subject D’s score in the area of comprehension on the DRA 2 increased 1 point keeping him in the independent performance area on the continuum.
Subject E’s results from the GRADE indicated only a 1 point increase in raw scores between the pretest and the posttest. Her stanine score of 6 remained unchanged. The two subtests of the GRADE that were considered when examining the feature of comprehension were sentence comprehension and passage comprehension. Subject E’s scores of a stanine of 5 for sentence comprehension and a stanine of 6 for passage comprehension remained constant for both pretests and posttests.
Subject E’s score in the area of comprehension on the DRA 2 showed some improvement, but not enough to move her from the instructional to the independent performance area on the continuum.
Subject F’s results from the GRADE indicated a 2 point difference in raw scores between the pretest and the posttest which changed her stanine score of from an 8 to a 9. The two subtests of the GRADE that were considered when examining the feature of comprehension were sentence comprehension and passage comprehension. Subject F showed significant improvement in sentence comprehension with a stanine of 7 on the pretest and a stanine of 9 on posttest. Her stanine score of 7 on the passage comprehension pretest increased to an 8 on the posttest, and her total comprehension test score was a stanine of 9.
Subject F’s score in the area of comprehension on the DRA 2 indicated significant improvement as well. Although she remained in the independent range, on the pretest, 4 out of the 12 items scored were in the instructional range, and on the posttest, scores for all items had increased to either the independent or advanced stage.
Subject G’s results from the GRADE indicated a change of 3 points in raw score between the pretest and the posttest. This lowered his stanine score of 9 on the pretest to an 8 on the posttest. The two subtests of the GRADE that were considered when examining the feature of comprehension were sentence comprehension and passage comprehension. Subject G showed a significant decline in sentence comprehension with a stanine of 9 on the pretest and a stanine of 5 on posttest. His stanine score of 9 on the passage comprehension pretest remained constant on the posttest.
Subject G’s score in the area of comprehension on the DRA 2 indicated no improvement in the comprehension subtest, and he remained in the independent performance area on the continuum.
Formal testing with both the GRADE and the DRA 2 did not indicate significant changes in all students. Some students did not change at all; some changed very slightly. Only one student demonstrated noteworthy growth in all subtests of both tests. There were also 2 students who scored lower on one subtest. Most of the overall student change was documented by the researcher as she reviewed the audio tapes of each session. Many students became more verbal as the sessions progressed. Students asked for fewer clarifications, and their responses became less literal and more inferential, indicating a better understanding of the selection.
Pre and Post Study Change by Subject
Subject A’s pretest scores on both the GRADE and the DRA 2 were both above average. There was only a sight possibility for improvement on the GRADE, which she did achieve. Many of her scores on the DRA 2 moved from the category of independent to that of advanced. It was, however, the growth that occurred which was observed and documented, rather than which was scored by a number, that was most significant for her. First and foremost were the observations that were made during the intervention sessions. Throughout the study Subject A was consistent in using the strategies that good readers use as she reread the text, identified important information, made inferences about the text and author, constructed mental images and identified the main idea. She became adept at using the strategies of questioning the fallibility of the author very early on in the study. As early as the fourth session, the first time a biographical nonfiction selection was introduced, she questioned how the author presented the information, the information that the author chose to leave out or leave the readers to figure out on their own, and the sequencing of the selection. Subject A also made significant gains in group participation. Throughout the first half of the study, she was very verbal, the first to respond to almost every question. Her responses were very appropriate, but dominating. During the last half of the study, she did not respond as often, especially literally. Many of her responses were inferences or text-to-text and text-to-world connections as she began to think about the text at a deeper level. After the study had been completed, the students were all administered post assessments. One of the questions on the reading survey portion of the DRA 2 asked for the student’s plan to become a better reader. Subject A responded, “I plan to kind of see how it feels to be an author, because I think it would be easier to read when you know what an author does.”
Subject B made significant gains in all comprehension subtests the GRADE, with posttest scores all in the highest stanine. Pretest scores of the comprehension subtest of the DRA, however, were higher than the posttest, placing him in the instructional category for that assessment. In comparison to the GRADE, which has a multiple choice format, the DRA requires written responses. Subject B put very little effort into completing this assessment. His writing included only general examples, contained ideas and facts which included incorrect information, and contained little or no detail which is indicative of only a partial understanding. Subject B participated in the study on a very literal level. His participation percentages were high, but that can not be attributed to an engagement with the text and subsequent queries. He consistently asked for clarifications on vocabulary during every session throughout the first half of the study’s duration. His responses were usually in reaction to the comments of the other participants in the group and did not necessarily question the author’s fallibility or purpose in the written text. Many times he focused on the illustrations, using them instead of the text to try to construct meaning. On the seventh session of the study, his participation percentage dipped suddenly and drastically. This continued for 8 more sessions. Not only was he asking for less clarifications, he was also responding very infrequently to queries and was not really engaged in any of the selections, even though during that time a variety of genres were explored. His responses were limited to a few text-to-self and text-to-text connections, a small number of clarifications, and limited reactions to the comments of others. On the sixteenth session as suddenly as his responses had ceased, they began again in earnest. It is quite possible it took him that long to feel comfortable with the discussion format. He posed much fewer requests for clarifications and some very thoughtful predictions and insightful, well thought out responses like, “I think Dona Josefa thinks she is the richest woman in the world because she is able to help people.” Another statement in which he directly commented on author’s purpose was, “Some people know different things, so the author had to start that way because she really didn’t know what we each knew.” Although Subject B did not consistently respond to queries and question the author even at the end of the study, he did begin to think before responding. His responses were not always literal and even sometimes reflected hints of a much deeper level of understanding.
Subject C made insignificant gains or remained constant on all pre and posttesting in both the GRADE and DRA 2. On the GRADE, his overall comprehension stanine of 7 did not change between pre and posttests. He remained in the independent level of the DRA 2, with most of the individual scores remaining the same. In his reading survey, he indicated that one of the things he does well as a reader is to read slowly so that he understands more. Although his written responses indicated that he was attending to the text, most of his responses were literal rather than inferential or critically thought out. Subject C’s daily participation in the study hovered between 5% and 15%. His responses were few in number and were either literal or reactions to another student’s thoughts. In his reading survey, Subject C stated, “My favorite genre is nonfiction. I like to read dragon and animal books.” So, as we read one our first informational text, he was one of the first students to be critical of the author’s organization as he observed, “You have to go back into the selection to get the information about the picture and that makes it more difficult.” His responses to subsequent selections, however, continued to be very literal. The next time that he questioned the author was many sessions later, toward the end of the study, when we read another informational text about rescue dogs. He stated that the author was “kind of confusing because she kept switching back and forth between topics and dogs.” Subject C did not respond to the modeling of critical thinking strategies as did the other participants nor did he actively engage himself in conversations as did many of the others. For this reason it was very eye-opening to read in his reading survey that, “When my teacher asks me a question about what I read, I have trouble thinking of an answer.” It appeared that Subject C needed a little more time to think before he was able to respond, even at a literal level. Being part of a group was obviously difficult because the other participants reacted too quickly for him, unless he had a significant amount of background knowledge as he did when informational text was read. Interestingly, when asked to describe what he plans to do to become a better reader, he answered, “I plan to remember the strategies and use them as I read.”
Subject D had high pretest scores and did not make any gains on posttesting in the GRADE and a very minimal gain on the DRA 2, where all test scores in the comprehension feature fell into either the independent or advanced stage on the continuum. On the pretest, his summary had been scored as instructional. It is worthy to note that on his reading survey, when asked what he needed to do to become a better reader, he wrote, “I need to work on summarizing.” Interestingly, on the posttest, his summarizing score improved, placing him in the independent range. Those summaries were written in his own words, using appropriate vocabulary and supporting details. His written responses, especially on the posttest, were very thoughtful, and his questions and predictions were excellent as they went beyond the text, probing for a deeper meaning. Subject D was a very quiet, reserved individual. His responses in the classroom were usually limited, not necessarily because he didn’t know the answer, but most probably because he was a more thoughtful listener who carefully processed all information before he replied. For this reason, it was interesting to observe how he reacted in the study group. Initially, his percentage of response was very low, and he did not answer the queries directly, but rather in response to others. His responses, however, were rarely literal. He used his background knowledge and strong vocabulary to help others with clarifications and to interpret the author’s purpose in writing. From the sixth session on, his response percentage rose and remained between 15% and 30% for most of the following sessions. He replied directly to the queries, many times providing innovative insights like how the author brought us from fantasy to reality in The Great Kapok Tree, and how the boarding house in Mirette on the Highwire “must have been a famous place if people came to Paris from as far away as New York and Moscow.” Subject D was an individual who was very perceptive, highly motivated, and very interested in learning. In his reading survey he said, “Reading makes me feel happy inside (and)…is easier for me than it used to be.” Subject D aspires to be an author one day. He hopes that when his readers think critically about his selections that they would not have to say the things that he said about these authors. For example, “the author did not present the information clearly at the beginning because she gave us too much information at once”, and “after a while this story got a little boring because it was the same pattern all the time.”
Subject E’s pretest scores on both the GRADE and the DRA 2 were both in the low average range. Although test scores indicated no improvement on the GRADE, her scores on the DRA 2 increased significantly on the fluency subtest, moving her from the instructional to independent level, but only 2 points on the comprehension subtest, leaving her in the instructional category. It is worthy to note that on the DRA 2 pretest, 5 out of the 12 scored items fell in the intervention category, and on the posttest, there were none scored as intervention. Subject E entered the district as an ELL student and was mainstreamed only 2 years ago. Throughout this study, her effort and motivation were commendable. Most of Subject E’s responses during the study were on a very literal level. Her daily response percentages fluctuated between 5% and 15%. Many times those responses were a reiteration in her own words what some of the other participants had already said. Although her responses were few and for the most part unoriginal, she did make some comments during the last few sessions that not only indicated understanding at a much higher than literal level but were also reflective of her very sensitive personality. One example was, “The people are looking at the beauty of the bird and not what is inside her.” During the next session, she said “The servants care more about the palace than the emperor.” Whether her understanding came from just background knowledge, critical thinking, or a combination of both, it undoubtedly indicated a glimpse of change and some deeper thinking. When asked about the new strategies that she learned, and what they have taught her, she responded, “It taught me to share my thoughts. It helped me learn more. It’s going to help me understand the story better and better.”
Subject F made significant gains in both comprehension subtests the GRADE, with posttest scores for sentence comprehension a stanine of 9 and passage comprehension a stanine of 8. She also made significant gains in the comprehension subtest of the DRA. However, she remained in the independent range on the continuum. Subject F made very few contributions to the queries that were presented to the group during the study’s sessions. She had always been hesitant to respond in class and had often in the past said that she did not give answers unless she was sure that she was right because she does not want to give the wrong answer. Her daily percentage of response ranged from as low as 2% to as high as 21%. It was interesting to note that the 2 days which she scored the lowest were consecutive sessions at the beginning of the study in which the poem Sierra was being read and discussed. When she did respond to queries, Subject F’s answers were rarely literal. Many indicated critical thinking and a deeper understanding of the selection. About midway through the study, while reading a fable, she commented, “The author is using strong words like ‘cry’ and ‘shouting’ to let us know how angry the farm animals are.” Another clear indication that she was thinking on a much higher level was when she proclaimed, “The kidney stew must have been really good if they devoured it.” One of the last sessions of the study she made a text-to-self connection as she stated, “Parents always seem interested in church but the kids are confused and bored.” It became very clear that although Subject F was not participating verbally, she was thinking about the selections and the discussions at more than a literal level. When Subject F was asked to reflect upon the new strategies that she had learned, and what the study had taught her, she responded with, “It taught me to think about the story more. You ask yourself questions and you think more about it than just reading it and then starting a new book.” In order to become a better reader, she “plan(ed) to predict more because it will get me thinking more about what the story is about.” Subject F had definitely begun to think about thinking.
Subject G achieved stanines of 9 on both comprehension subtests of the GRADE pretest. Posttest scores indicated a decrease in the sentence comprehension to a stanine of 5, and the passage comprehension remained constant. This brought his overall stanine down to an 8. On the DRA 2, there was no change in his comprehension score, and he remained in the independent range for that subtest. Subject G was absent 6 of the 25 days of the study. On the days he was present, his percentage of response ranged from 2% to 14 %. The types of responses that he gave fluctuated as much as his percentage of response. At the first two sessions of the study, he gave one or two word answers that were in response to the answers of the other participants. During one of the sessions, when Subject G had given one very brief response during a 30 minute period, the last question that was asked that day was if the participants thought that the author had given us a lot of information in such a short selection. Subject G was the first to respond with a significant amount of information and various text references, indicating a very thorough understanding of the selection. The sixth session, when we read the poem Sierra, was his most verbal day. He used his prior knowledge about glaciers to explain how they cut through mountains, claiming that “they changed the land wildly." He also interpreted “the sisters” as being the other mountains. After the last selection was read and discussed, the participants were asked to think about the new strategies they had learned, and what QtA had taught them. Subject G stated, “It taught me to think about the authors, and what they did to make you like the story or not like the story.” When asked if he would be using these kinds of strategies on his own now that we were done with the study, he replied, “Yes, because this has taught me a lot to be in a group, but now I’ll try it by myself and I’ll keep doing it, keep trying.”
The intervention used in this study, Questioning the Author (QtA), was just one source of instruction that the students were receiving in the classroom to enhance comprehension skills. They were still involved in classroom activities which develop comprehension such as participation in reciprocal teaching groups, guided reading groups, and whole-class and small-group discussions. These instructional elements were, therefore, also contributing factors toward each individual’s comprehension growth. The findings of this study were limited to seven grade-4 students during a specific time frame and therefore not generalizable. This study also could have been more effective, and students might have made more significant progress if it could have lasted longer than 8 weeks, affording the subjects more strategy modeling and practice.
|
