Description of Assessment of Student Progress
For this study, the purpose of formal and informal assessments was to determine if improvements were made in reading comprehension and listening comprehension after students with language learning disabilities (LLD) received 8-weeks of explicit metacognitive strategies instruction. The data obtained from all assessments were graphed to allow for visual analysis of information. Data were triangulated to ensure strong support for the outcome of intervention for listening and reading comprehension. Pretest and posttest scores were obtained on all three measures for each area of comprehension, and ongoing assessments served to guide instruction and measure progress in reading comprehension throughout the study.
The goal of the reading assessments was to determine if the research subjects demonstrated improvements in reading comprehension skills.
- The Developmental Reading Assessment (DRA) (Beaver & Carter, 2003) was used to assess skills and growth in reading comprehension of fiction and nonfiction texts. The student read a portion of a leveled fiction or nonfiction passage silently. After the passage was finished, a written retelling of the story was completed and literal, inferential, evaluative, and reflective comprehension questions were answered. A comprehension skills and strategies score was provided, using a rubric, to determine reading comprehension skills and to monitor progress.
- The Informal Reading Inventory (B/RIRI) (Roe & Burns, 2007), which is an informal oral reading assessment, further explored reading comprehension skills (Harp, 2006). The subjects read a fiction or a nonfiction passage and provided an oral retelling for each passage. Closed-ended comprehension questions taken from the B/RIRI-7 were administered after the retelling. Due to the limited availability of expository texts, fifth-grade reading passages were selected for this text type to measure progress in reading and listening comprehension. This was in spite of the skills of each subject being lower than the fifth-grade level. A percentage was provided for the comprehension questions and was used to compare pre-intervention and post-intervention scores.
- Informal oral retelling rubrics taken from Harp (2006) were collected weekly, alternating between fiction and nonfiction reading tasks. These assessments were implemented to monitor progress, determine program effectiveness, guide instruction, and to compare progress to the other assessments. Each subject provided an oral retelling of the information reviewed during the lesson. The subjects were asked to pretend to retell the information to a person who had not yet heard the story.
Informal oral retellings were scored using a rubric that was a modified version of the retelling rubrics created by Harp (2006). The rubric scoring system ranged from a 1 to a 4. A score of 1 indicated that the student provided little critical information in the retelling even when given adult assistance. To achieve a score of 2, the student retold some of the critical elements of the fiction or nonfiction story with adult assistance; however, the retelling contained errors or minimal details were offered. Assistance included open-ended and closed-ended questions. A score of 3 revealed that the student was able to retell most of the story with minimal adult assistance. To obtain a score of 4, the student independently retold the fiction or nonfiction story and included all the key elements of the text.
The three assessments for reading comprehension were determined to be triangulated based on their purpose: to assess reading comprehension skills of fiction and nonfiction texts, and the differences among the tests in how they assess reading comprehension. The different types of reading included oral and silent reading. Question formats consisted of open-ended and closed-ended comprehension questions and oral retellings, which may have included prompted recall of information that was not explicitly stated in their oral retellings. Based on all these factors, it was concluded that the assessments achieved triangulation, providing strong support for the outcome of this research project. Each test was administered as a pretest and posttest while the informal retelling rubric was completed weekly, alternating between fiction and nonfiction, to determine the subjects’ reading comprehension skills using fiction or nonfiction texts that directly related to the curriculum.
The students also tracked their thoughts on Post-it notes, collected two times throughout the study, to measure progress in the application of comprehension monitoring strategies. The Post-it notes also allowed the students the opportunity to self-assess growth in comprehension monitoring and to identify strengths and weaknesses in strategy application. The information garnered from this assessment will be discussed in the Analysis section.
Finally, a daily journal was maintained to reflect on lessons and informally monitor progress in students’ responses and participation in tasks. These observations were used to corroborate the qualitative data obtained during guided reading lessons and will be discussed in the Analysis section of this study.
The listening comprehension assessments measured the subjects' ability to comprehend orally presented texts after receiving 8-weeks of explicit instruction in metacognitive strategies. Assessment of listening comprehension was accomplished through pre-intervention and post-intervention tasks. The subjects were required to listen to a fiction or nonfiction passage and to answer comprehension questions or provide oral retellings.
- The Clinical Evaluation of Language Fundamentals-Fourth Edition: Understanding Spoken Paragraphs subtest (Semel, Wiig, & Secord, 2003) is a standardized assessment of listening comprehension. The Understanding Spoken Paragraphs subtest measures listening comprehension skills. The subjects were required to listen to three separate stories that were either fiction or nonfiction. After listening to the story, the subjects were asked to answer five comprehension questions per story, a total of fifteen questions for the entire subtest. The closed-ended questions were both literal and inferential, which assessed skills such as the main idea, details, sequencing, inferencing, and predicting. A percentile rank was provided to determine skill level and compare pretest and posttest scores.
- The Informal Reading Inventory-7th Edition (B/RIRI-7)(Roes & Burns, 2007) was used to assess listening comprehension. This assessment further analyzed each subject’s ability to listen to paragraphs and answer comprehension questions. One problem that arose during this study was the lack of availability of nonfiction passages below the fifth-grade level. Therefore, expository passages were selected from the fifth-grade level for listening comprehension. The subjects were asked to listen to a fiction or nonfiction story and follow the same procedures used during the reading comprehension assessment described above. Please refer to the above paragraph for specific information regarding the B/RIRI-7.
- The final assessment consisted of informal oral retellings of both fiction and nonfiction stories presented prior to and following intervention. Fiction and nonfiction texts were selected based on the classroom curriculum and student interest. The story or section of a chapter was read aloud to the students and the students provided an oral retelling. The same testing procedure and scoring rubric used for the oral retellings of reading comprehension were administered to assess listening comprehension skills. For additional information regarding the procedures used to elicit an informal oral retelling, please refer to the procedures described above in the reading comprehension paragraph on informal oral retelling rubrics based on Harp (2006).
The measures used to assess listening comprehension were determined to be triangulated based on their purpose: to assess listening comprehension skills; the diversity of the types of assessments: norm-referenced, informal reading inventories, informal retelling rubrics; and the opportunities to demonstrate comprehension of information using different testing formats: question-answer response and oral retellings. The listening comprehension assessments were measured pre-intervention and post-intervention to establish skill level and to measure the effectiveness of intervention on listening comprehension skills. Based on the similarities and differences among assessments, it was determined that triangulation was achieved. This indicated that these tests offered strong support for the effectiveness of explicit instruction in metacognitive strategies on listening comprehension.
The Metacomprehension Strategy Index taken from Harp (2006) was used to identify what strategies the subjects were aware of before and after instruction. A percent correct score was tabulated to determine whether improvements were noted in this area. A Metacomprehension Strategies Checklist was completed by the students on a weekly basis, along with anecdotal notes, to assist in teaching metacognitive strategies before, during, and after reading. The checklists served to shift the role of responsibility from the teacher to the students and to allow the students greater responsibility in monitoring and applying the strategies necessary to enhance comprehension. Two samples of the checklist were collected during the study to establish knowledge and application of strategies before, during, and after reading. The results of the Metacognitive Strategies Checklist assessments will be presented in the Analysis section of this study. Finally, the metacognitive awareness subtest on the Developmental Reading Assessment (DRA) was used to track knowledge and application of metacognitive strategies pre-intervention and post-intervention. Although these measures did not formally assess reading or listening comprehension, they provided insight as to whether or not the students learned and applied strategies before, during, and after reading as a result of participating in explicit instruction in metacognitive strategies during guided reading activities.