Douglas, S., Taylor, J., Dexter, D., & McNaughton, D. (2021). Use of an online peer review tool to support feedback and collaborative skills in preservice teachers. Contemporary Issues in Technology and Teacher Education, 21(2). https://citejournal.org/volume-21/issue-2-21/general/use-of-an-online-peer-review-tool-to-support-feedback-and-collaborative-skills-in-preservice-teachers

Use of an Online Peer Review Tool to Support Feedback and Collaborative Skills in Preservice Teachers

by Sarah Douglas, Michigan State University; Jonte Taylor, The Pennsylvania State University; Douglas Dexter, Dallas Independent School District; & David McNaughton, The Pennsylvania State University

Abstract

Effective teachers require a variety of skills, including the ability to provide and incorporate feedback from others. Self-review and peer review are two methods that help preservice teachers develop feedback skills. Teacher educators face a number of challenges utilizing peer and self-review within their courses, especially in large university classes where preservice teachers of different majors are enrolled in the same course. Online peer review tools offer a promising approach to support peer review in preservice teacher education. Guided by adult learning theory, an experimental study was conducted to determine the effect of online peer review, using the peer review tool Peermark™. Preservice teachers in the experimental group used online peer review to provide feedback to peers about the creation of a content specific graphic organizer. Results show that online peer review resulted in higher quality graphic organizers when compared to self-review. Limitations, implications for practice, and future research suggestions are discussed.

Teachers require a variety of skills to be effective in the classroom, especially when supporting students with varying disabilities. They must be able to interact and support students with diverse needs (Hammond, 2014) and communicate and work collaboratively with other educators to develop, plan, and deliver individualized instruction using a team approach (Friend & Bursuck, 2012; Gulikers et al., 2009; Solis et al., 2012). Teachers must be able to provide high-quality feedback to students (Chan et al., 2014), paraeducators (Douglas et al., 2016), and other professionals (Nierengarten, 2013). Teachers must also be able to accept and incorporate feedback from administrators and other supervisors (Nolan & Hoover, 2011).

Preservice teachers have numerous characteristics that should be considered when designing learning opportunities. As adult learners, preservice teachers come with existing knowledge, are often self-directed, and prefer to be actively engaged and participate in collaborative approaches to learning (Knowles et al., 2011). To support adult learners, instructors act as facilitators (Chen, 2014) and provide supports, which allow learners to gain feedback and collaboration skills (Bonk et al., 2004).

Teacher education programs frequently utilize instructional activities to help preservice teachers develop the collaborative and feedback skills they will need in the classroom (Bentley-Williams et al., 2017). Reflection and evaluation practices, such as the use of peer and self-review, support collaborative and feedback skills, have been linked to improved student outcomes (Andrade & Valcheva, 2009; Mitchell et al., 2017; Søndergaard & Mulder, 2012), and have been shown to support adult learners in gaining skills and knowledge (Gulikers et al., 2009; Sluijsmans et al., 2002). Similarly, teachers report the value of peer and self-review (Budge & Gopal, 2009) and indicate that these practices facilitate a cycle of continuous feedback, which allows for continued growth (Pawan & Fan, 2014).

Teacher educators, however, report difficulty implementing peer and self-review with preservice teachers (Boase-Jelinek et al., 2013; Meerah & Halim, 2011; Søndergaard & Mulder, 2012; Tricarico & Yendol-Hoppey, 2012). Causes of these difficulties include student characteristics (e.g., lack of skills or knowledge to conduct a review), logistics of the peer review process (e.g., organizing exchange of materials and facilitating anonymous peer reviews; Boase-Jelinek et al., 2013), and teacher educator limitations (e.g., timeliness of feedback and limited expertise in content areas; Hatziapostolou & Paraskakis, 2010).

Teacher educators may remediate these challenges through the use of instructional supports, including instructor-developed guides and rubrics for peer and self-review (Orsmond et al., 2000) as well as the use of online review tools (Kong et al., 2009; Søngergaard & Mulder, 2012). The examination of one such online tool, Peermark, with a checklist and rubric to guide review for preservice teachers, was the focus of this experimental study. In the sections that follow, we report our exploration of the use of Peermark to support peer-review with preservice teachers and compare peer review to self-review.

Literature Review

Self-Review

Reflective practices have been identified as a key instructional component within teacher education programs (Darling-Hammond & Lieberman, 2013). Self-review, one type of reflective practice, is a method that can be utilized with preservice teachers to help develop the skills of providing and incorporating feedback. Self-review promotes learning, motivation, and active involvement in the learning process (Baecher et al., 2013; Case, 2007; Cheung, 2009). Self-review also helps preservice teachers identify their strengths and weaknesses (Andrade & Valtcheva, 2009), and supports the development of autonomy, self-regulation, self-reflection, self-evaluation skills, and goal setting (Baecher et al., 2013; Carless et al., 2011; Case, 2007; Hanrahan & Isaacs, 2001). The use of self-review can also save time and effort for the instructor, while still leading to high-quality outcomes for preservice teachers (Belski, 2009; Bonk et al., 2004).

Although the use of self-review and other reflective practices is beneficial, some preservice teachers may be reluctant to engage fully in the self-review process or may not be skilled enough to accurately identify their own limitations, in part, because they may lack self-regulation skills (Carless et al., 2011). Preservice teachers who fail to learn self-evaluation skills may struggle to incorporate feedback from others and improve their performance. Teacher educators may need to provide instructional supports to help preservice teachers in developing self-evaluation practices, such as the use of clear rubrics to help guide the review process (Kong et al., 2009).

Peer Review

Peer review is an approach similar to self-review that helps preservice teachers develop a number of essential skills (Britton & Anderson, 2010). Much like self-review, peer review supports critical thinking, self-reflection, and self-evaluation (Søndergaard & Mulder, 2012), allows preservice teachers to learn self-assessment skills (Lundstrom & Baker, 2009), and engage in higher level thinking (Odom et al., 2009). Peer review also supports the development of collaborative and assessment skills (Freeman, 1995; Hanrahan & Isaacs, 2001). Peer review can improve the timeliness of feedback, as it is often possible for peers to provide feedback more rapidly than an instructor who is responsible for administering feedback for all students in a course (Belski, 2009).

Engaging students in peer review actively encourages the retention of skills and increases preservice teacher involvement in the learning activity (Carless et al., 2011). Additionally, peer review is often seen as more helpful than teacher feedback because it is written using common language (Nicol et al., 2014), includes review from multiple individuals – increasing the quantity of feedback – and can include instructional supports that ultimately result in more thorough feedback than instructors typically provide to preservice teachers (Cho & MacArthur, 2010).

Peer review also exposes learners to many different ways of thinking (Bartlett & Spicer, 2015), a limitation of traditional faculty feedback. As such, peer review is an approximation of real-world experiences, in which teachers receive feedback from a variety of sources and incorporate that feedback to improve practice (Nicol et al., 2014).

Benefits of Peer Review in Teacher Education

Peer review may be especially helpful when training preservice teachers to work with children with disabilities. Introductory special education courses are often held in large university course sections, in which subgroups of preservice teachers have particular learning goals and would benefit from opportunities to practice with relevant, content specific materials.

For example, many programs require that all teacher certification candidates (e.g., science, math, or art education) take a single common class to prepare them to work with students with disabilities (Harvey et al., 2010). While the value of incorporating peer review has been well recognized, instructors often find managing the peer review process challenging in large university classes or classes with a considerable diversity in preservice teacher expertise and content focus.

Within teacher education, where preservice teachers are often combined into the same courses but with different teacher certification goals, instructor feedback can be complex and may require content knowledge that is outside the expertise of the instructor. Peer review provides an alternative, where in an ideal situation, art education majors would practice and receive feedback from other art education majors on materials related to art education, and science education majors would provide and receive feedback with materials related to science education, and so on. Innovative methods, such as the use of online peer review tools, have been identified for providing peer feedback in an effective, efficient, and meaningful way for preservice teachers (Avery & Meyer, 2012; Lynch et al., 2012).  

Technologies to Support Peer Review

Online peer review technologies support teacher educators in providing feedback using alternative methods. Online peer review allows teacher educators to monitor the peer review process and ensure all preservice teachers are appropriately engaged and using peer feedback to improve their performance (Hatziapostolou & Paraskakis, 2010). Online peer review also supports double-blind review, where neither the author nor reviewer know the identity of each other (Dahl, 2007) and instructors can assign peer reviewers (see Gehringer, 2011), which might include assignment of peers with similar expertise to evaluate assignments.

To date, online peer review has been suggested as a promising approach in the areas of computer science (Hämäläinen et al., 2011), law (Goldin & Ashley, 2010), English composition (Tsai, 2017), psychology (Jhangiani, 2016), architecture (Thompson & McGregor, 2009), and biology (Liang & Tsai, 2010). Within these studies, peer review was shown to result in increased understanding of course content (Jhangiani, 2016), and improved writing skills (Liang & Tsai, 2010). Students were able to identify concerns related to content within peer’s assignments (Goldin & Ashley, 2010), found written comments from peers to be helpful, and did not find the peer review process burdensome (Hämäläinen et al., 2011). Students also noted the anonymity of online peer review was important (Thompson & McGregor, 2009).

The use of online peer review has also been explored with preservice teachers and appears to be a promising, although understudied, approach (Avery & Meyer, 2012; Lynch et al., 2012). Online peer review has been used to support preservice teacher creation of project-based engineering activities (Lynch et al., 2012) and to design and carry out scientific experiments in instructional settings (Avery & Meyer, 2012).

Results from studies utilizing online peer review with preservice teachers indicate improved learning, increased self-efficacy, and overall satisfaction with the peer review process (Avery & Meyer, 2012; Lynch et al., 2012). Yet limited research has focused on preservice teachers and content areas such as math and social studies. No study to date has utilized commercially available online peer review tools with preservice teachers or addressed preservice teacher skills to support students with disabilities.

Peer review could be especially beneficial to prepare preservice teachers to support students with disabilities, given the collaborative approaches required to provide high-quality education to these students (Alquraini & Gut, 2012). Given the lack of research with preservice teachers and the potential benefits of online peer review with this population, further exploration of online peer review tools and use of online peer review to support activities appropriate for students with disabilities is warranted.

Commercially available online peer review tools, often available within online course management systems on college campuses, may help support instructors in facilitating peer review within their courses. Turnitin® (2015), an online assignment submission and grading tool that is widely available in institutes of higher education, has a peer review feature called Peermark. Through Peermark, preservice teachers can submit assignments within existing course management systems, and instructors can facilitate double-blind peer review as part of the learning process prior to final assignment submission (Dahl, 2007). The goal of the tool is to allow preservice teachers to receive and incorporate feedback without added workload on the instructor (Han, 2012). The tool also mirrors professional practice and builds important skills by providing preservice teachers with opportunities to give and receive feedback.

Rationale for Current Study

Given the lack of research exploring online peer review tools with preservice teachers, and the potential benefits of incorporating this practice into preservice teacher education to meet the needs of adult learners, we explored the use of peer review with preservice teachers enrolled in a large undergraduate, introductory level special education class. Although this course included multiple sections, each contained preservice teachers in a variety of content areas (e.g., science, math, and art education).

Graphic Organizers

In this experimental study, we carefully selected a learning activity (i.e., graphic organizers) that is an important teaching strategy across a range of content areas, but commonly misused without sufficient instructor feedback (Dexter & Hughes, 2011). Graphic organizers,visual displays that can support student understanding of the relationship between facts and concepts (Akhondi et al., 2011; see Figure 1 for examples of three commonly used graphic organizers), are a beneficial tool to use in inclusive educational settings where students with disabilities are instructed alongside their typically developing peers in the general education classroom (Knight et al., 2013).

Figure 1    Graphic Organizer Examples

Graphic organizers have been shown to support learning in social studies, science, English/language arts, and mathematics classes across a variety of grade levels (Barton-Arwood & Little, 2013; Dexter & Hughes, 2011). They are commonly used in classrooms to support comprehension (Stetter & Hughes, 2010), but students require explicit instruction to effectively utilize graphic organizers (Dexter & Hughes, 2011), and preservice teachers require systematic training to provide effective instruction to students in comprehension strategies, including graphic organizers (Mason & Hedin, 2011).

Unfortunately, research has shown that students receive little explicit instruction on graphic organizers even when present in textbooks (Catley & Novick, 2009). Therefore, the learning activity, Creation of Graphic Organizers, was specifically selected for this study because of its usefulness to support students with various disabilities in a variety of content areas (Dexter & Hughes, 2011) and the need for systematic training to help preservice teachers learn how to implement graphic organizers effectively within a lesson (Mason & Hedin, 2011). In addition to the creation of graphic organizers, we included a peer review element to support preservice teachers’ explicit understanding of using graphic organizers for students with disabilities in specific content areas. 

Theoretical Framework

Given the characteristics of preservice teachers as adult learners (e.g., self-directed an having preference for collaborative approaches to learning), we framed our study around adult learning theory, which provided insights into adult learner characteristics and ways instructors can best meet their needs (Bonk et al., 2004; Chen, 2014). In a model outlined by Taylor and Hamdy (2013), adult learners start with existing knowledge then engage in a task where the instructor provides resources to support success. Each learner approaches the task with different levels of development, motivation, and preferences and approaches, which can result in dissonance.

To resolve this dissonance, the learner engages in activities such as discussion, self-review, and peer review. This process allows learners to incorporate new skills and insights into future learning. To support adult learners, instructors act as facilitators (Chen, 2014) and provide supports, which in turn, allows learners to gain feedback and collaboration skills (Bonk et al., 2004). In an effort to reduce adult learner (i.e., preservice teacher) dissonance (Taylor & Hamdy, 2013) within this study, the researchers (as facilitators; Chen, 2014) introduced a generalizable strategy to support students with disabilities across content areas (i.e., graphic organizers) and provided opportunities for peer review as a collaborative effort (Bonk et al., 2004; Taylor & Hamdy, 2013), with an aim to support further understanding of graphic organizer use and effectiveness (see Figure 2). Because instructors face challenges providing feedback in areas outside their expertise, a focus on graphic organizers appeared to be an area where peer review might be especially beneficial.

Figure 2   Theoretical Framework for Current Study

Existing literature has shown that the use of adult learning theory principles can support teacher development of instructional, collaborative, and feedback skills. Specifically, research indicates the benefits of incorporating adult learning theory within professional development practices for teachers. This research included the benefits of using adult learning theory to support instructional coaching (Heard & Peltier, 2021) and professional learning communities (Peppers, 2015), which has led to increased skills (e.g., opportunities to respond, behavior specific praise, and collaboration) in educators. Additionally, a review of professional development programs for teachers conducted by Darling-Hammond et al. (2017) highlighted that effective professional development for teachers utilizes adult learning principles.

Research Questions

 In this study, we examined the impact of online peer review, using the PeerMark feature in TurnItIn (2015) with preservice teachers to support the creation of graphic organizers applicable to their educational concentration areas (e.g., science, math, or art education). We sought to answer the following research questions: (a) Does the use of online peer review improve the quality of preservice teacher graphic organizers from pre- to postreview? (b) Are there differences in mean gain scores for preservice teacher graphic organizers when comparing treatment (i.e., online peer review) and control groups (i.e., self-review)?  (c) Are differences in mean gain scores also observed for preservice teacher graphic organizers when comparing treatment and control groups within different content areas (e.g., preservice math teachers in the treatment compared with preservice math teachers in control)?

Methods         

To answer these research questions, we used a pretest-posttest design with random assignment to the experimental (i.e., online peer review) or control group, in which preservice teachers were directed to make use of a rubric (designed by the instructor and third author of this article) to self-evaluate and make changes to their graphic organizer (i.e., self-review).

Setting

The study was conducted in an undergraduate special education course at a research-intensive university in the northeastern United States. The course provided content in secondary education methods for students with disabilities and was required for all preservice teachers who were seeking secondary certification. The third author of this article, an assistant professor in special education with experience teaching secondary students with special needs and 4 years of experience teaching the course, was the course instructor.

The course utilized a hybrid approach with online and face-to-face instruction. Each week, prior to class, preservice teachers engaged in online instruction, including lectures and questions related to the lecture. Then, once a week, preservice teachers met face-to-face with the instructor and received and participated in an in-class activity related to the course content for that week. After the face-to-face class, preservice teachers completed and submitted a learning activity demonstrating that week’s learning using an online course management system where they then received feedback from the course instructor using a rubric that was shared with students.

Participants

Prior to the start of the study, we received approval through an institutional review board. All preservice teachers within the secondary special education methods course sections were invited to participate through an announcement in their face-to-face class. During recruitment, participation requirements were detailed, informed consent was provided, and the risks and benefits of participation were detailed. Preservice teachers who did not provide consent were still provided with the same materials and activities in the course and were informed that nonparticipation would not affect their grade in the course. However, their data were not included in this study. Preservice teacher participants were randomly assigned to treatment (n = 40) and control (n = 36) groups. See Table 1 for information regarding participant education concentration and group assignment.

Table 1   Participant Education Concentration and Group Assignment

 
Education Concentration
Treatment

(Peer Review)
Control

(Self-Review)
Science52
Social Studies88
English55
Math21
Music107
World Languages66
Art23
Communication Sciences and Disorders24
Total4036
Note. All preservice teachers were randomly assigned to treatment and control groups. Preservice teachers were all junior undergraduate academic status or higher.

Procedures

As part of the typical structure of the course, preservice teachers engaged in activities (e.g., lectures, readings, and video models of skills) outside of the classroom and within the classroom (e.g., hands-on activities exploring various instructional approaches and application activities from lecture). As part of this study and the course, preservice teachers were provided with online content about graphic organizers (i.e., types of graphic organizers, uses in the classroom, and benefits of graphic organizers) prior to attending class. This content included an online lecture, guided notes that accompanied the online lecture, and online questions related to the lecture.

Prior to the in-class activity, all students were expected to watch the online lecture, complete guided notes, and respond to the online questions. Next, preservice teachers participated in a face-to-face activity where they learned how to create a lesson plan using a graphic organizer. The activity included instruction during class about how to use the checklist (see example in Figure 3) and rubric to evaluate their work or the work of peers and how to submit their assignment using the online peer review system TurnItIn.

Figure 3   Peer/Self-Review Checklist

After the in-class activity, all preservice teachers were required to create a new graphic organizer for students with disabilities in a secondary setting for their content area as a final assignment. All preservice teachers used the online review system TurnItIn within the course management system to submit their graphic organizer. Preservice teachers also used the peer/self-review checklist provided by the instructor to guide their suggestions/revisions (see Figure 3) as was done during the in-class activity.

All participating preservice teachers were randomly assigned to either the treatment or control group. Preservice teachers in the control group completed their graphic organizer assignment and were instructed to conduct a self-review of the assignment using the checklist before submission. Preservice teachers in the treatment group were assigned to review a graphic organizer from a peer in the class, with their graphic organizer reviewed by peers with similar content focus (e.g., science education majors reviewed assignments from other science education majors).

A double-blind peer review process was used to ensure anonymity for students in the treatment group. Using the checklist and rubric, peer reviewers made suggestions on potential improvements to the assignment. Students in the control group conducted a self-review using the checklist and rubric provided to all students. Preservice teachers in treatment and control groups were both required to submit initial and reviewed assignments (see Figure 4 for research procedure), an effort that was made to ensure that students all had access to instructional materials to complete the activity successfully and reduce the likelihood of skewed study results between the treatment (peer review) and control (self-review) groups. The graphic organizer was a required assignment for all students in the course.

Figure 4   Control and Treatment Group Graphic Organizer Assignment Process

Data Collection

Data for this study were collected from participating preservice teachers within the online course management system. All initial and final submissions of the graphic organizers were gathered, scored, and compared (pre-post comparison) for participating preservice teachers.

Data Analysis

Prereview graphic organizers and postreview graphic organizers were scored by the third author using an instructor-created grading rubric (see Table 2). To ensure accuracy in scoring, a research assistant, blind to the purpose of the study and group assignment of the participants, independently scored graphic organizers using the same rubric as the third author. Reliability was calculated by taking the score agreement for graphic organizers and dividing it by the total number of graphic organizers within the study. Reliability between the primary scorer and the secondary scorer had a mean of 95%. All disagreements were discussed until 100% agreement was reached.

Table 2   Graphic Organizer Assignment Rubric

ItemPoints possibleHighly rated assignment description
Copy of content area text2Preservice teacher provided a copy (or reference) of the content area text for which the graphic organizer was selected.
Identification of verbal structure4Preservice teacher identified and labeled the verbal structure used in the selected text. If more than one verbal structure was present the preservice teacher selected the one that best suited the content.
Identification of graphic organizer type4Preservice teacher selected an appropriate graphic organizer for the text using the decision tree provided in course materials and identified the type of graphic organizer that was used.
Completed graphic organizer10Preservice teacher created or located a graphic organizer that was appropriate for the selected content area text. The relationships between information in the graphic organizer were clear and accurate. The preservice teacher cut unnecessary information (“fluff”).
TOTAL20

Descriptive statistical analyses were used within this study to compare the mean pre- and postreview scores of preservice teachers in the treatment and control groups. Mean scores, changes in mean score, standard deviation, and t-test analyses were conducted to determine if gains were made in mean graphic organizer scores and if means between the treatment and control groups differed.

Within-group comparisons were also conducted to determine differences within content areas (e.g., preservice math teachers in the treatment group compared with preservice math teachers in the control group). Then Hedge’s g was conducted to determine effect size of mean group differences. As suggested by Hedges (1981), Hedge’s g calculations were used due to the small sample sizes of comparison groups. Effect sizes can be interpreted as follows: .81 or above are large, .51-.80 are medium, and .50 and below are small (Cohen, 1988). Effect size calculations were conducted on between- and within-group comparisons on the following: (a) overall treatment and control comparison of finalized scores, (b) initial and finalized scores within treatment and control groups by content area, and (c) finalized score comparisons between treatment and control groups by content area for areas with sufficient participants in the treatment and control groups.

Results

A total of 76 preservice teachers participated in the study. Results indicated that, although treatment and control groups had similar prereview graphic organizer scores, postreview graphic organizer scores were significantly higher for the treatment group. Specifically, preassessment scores between groups showed no significant differences between the treatment and control groups (treatment = 79.8%, control = 78.3%; t = 0.6977, p = 0.48). However, there were significant differences (17.5%) between postreview scores for treatment and control groups (control = 78.3%, treatment = 95.8%; t = 10.01, p = 0.0001).

Analyses using Hedge’s g (see Table 3) indicated that the treatment group consistently outperformed the control group as indicated by effect sizes for growth of scores from prereview to postreview. Between group comparisons were g = 2.27, and within-group growth comparisons were g = 1.96 for treatment and g = 0.00 for control.

Table 3   Hedge’s g Effect Size Results for Within- and Between-Group Comparisons

Group nOriginal Graphic OrganizerEdited Graphic OrganizerMean GainWithin Group ESBetween Group ES
  MSDMSD   
Overall
Treatment4015.961.9919.151.12+3.191.962.27
Control3615.651.8715.651.870.000.00 
By Content Area
Science
Treatment514.701.9919.001.00+4.302.471.88
Control215.002.8315.002.830.000.00 
Social Studies
Treatment814.501.5818.751.39+4.252.702.07
Control815.191.8315.191.830.000.00 
English
Treatment517.102.0719.400.89+2.301.302.54
Control516.001.4616.001.460.000.00 
Math
Treatment216.500.7120.000.00+3.503.98NA
Control117.000.0017.000.000.000.00 
Music
Treatment1016.602.0819.201.23+2.601.461.70
Control715.792.5515.792.550.000.00 
World Language
Treatment616.252.1419.331.21+3.081.642.69
Control614.831.8114.831.810.000.00 
Art
Treatment216.000.7118.000.00+2.002.281.03
Control316.001.7316.001.730.000.00 
CSD
Treatment217.50.71200.00+2.502.852.20
Control416.881.3116.881.310.000.00 
Note. ES = effect size; n = number; M = mean; SD = standard deviation; NA = not applicable; CSD = communication sciences and disorders.

Analyses were also conducted on pre-post measures for groups by content area. Within-group comparisons and between-group comparisons were made for all but one content area due to small sample sizes (i.e., math; see Table 3). This pattern of performance by the treatment group in each of these content areas was consistent and mirrored the results from the overall group results. Specifically, for the content area of science within-group comparisons were g = 1.88, between-group comparisons were g = 2.47 for treatment, and g = 0.00 for control. Social studies within-group comparisons were g = 2.07, between-group comparisons were g = 2.70 for treatment, and g = 0.00 for control.

For English, within-group comparisons were g = 2.54, between-group comparisons were g = 1.30 for the treatment group, and g = 0.00 for the treatment group.  Music within-group comparisons were g = 1.70, between-group comparisons were g = 1.46 for treatment, and g = 0.00 for control. World language within-group comparisons were g = 2.69, between-group comparisons were g = 1.64 for treatment, and g = 0.00 for control. Results indicate large effect sizes for all groups.

Discussion

This study provides important new insights related to online peer review. First, the study was conducted with preservice teachers, an underresearched group in the area of online peer review, using a widely available peer review tool (i.e., TurnitIn Peermark), which has not been explored within other studies in this area. We utilized a learning activity that is traditionally challenging for both instructors and preservice teachers.

In answer to our first research question (Does the use of online peer review improve the quality of preservice teacher graphic organizers from pre- to postreview?), we found that online peer review can significantly improve preservice teacher performance on a challenging teaching practice (e.g., creation of graphic organizers). For our second research question (Are there differences in mean gain scores for preservice graphic organizers when comparing treatment and control groups?), we found that preservice teachers in the treatment group (i.e., peer review) demonstrated extremely large effect sizes from within group analyses, while preservice teachers assigned to control group (i.e., self-review) demonstrated no growth from pre- to postreview.

The same findings were noted when looking at group differences by content area as part of the third research question (Are differences in mean gain scores also observed for preservice teacher graphic organizers when comparing treatment and control groups within different content areas?). Overall, our findings suggest that PeerMark is a promising instructional tool to support online peer review for preservice teachers. Preservice teachers showed skill improvements from online peer review as part of their preservice learning activities, and peer review helped support preservice teachers in producing higher quality educational products.

Results found in this study are consistent with adult learning theory and results from previous studies (e.g., Avery & Meyers, 2012; Lynch et al., 2012), in which peer review was noted as beneficial for preservice teachers, but this study extends the literature in a number of important ways. First, this study adds to the online peer review literature for preservice teachers, in the area of special education knowledge and skills, including content specific areas in which children with disabilities are provided instruction (e.g., math, science, social studies).

Similar to the work of Avery and Meyer (2012), this study was implemented with preservice teachers and benefits of online peer review were noted for both teaching and content related skills. However, our study explored content areas beyond science (e.g., math, social studies, and art). This study also extends the literature through the use of a widely available online peer review tool. Previous studies most frequently used online peer review tools that are not widely available in universities across the United States (Avery & Meyer, 2012; Thompson & McGregor, 2009).

The use of a widely available online peer review tool within this study, such as Peermark, is advantageous as it has the possibility to be implemented more readily within existing teacher education programs. The use of commercially available online peer review tools might permit universities to use the online peer review systems throughout the students’ education, thus reinforcing the development of reflective and feedback skills. Online peer review tools may also be of benefit for students in secondary settings, and preservice exposure to such tools might help future teachers consider incorporating peer review into their instructional practices.

This study also provides evidence that online peer review can be effective even when implemented in the context of challenging tasks, such as graphic organizers. Online peer review resulted in higher quality graphic organizers by preservice teachers when compared to self-review of graphic organizers, supporting the idea that peer review helps increase collaborative skills within preservice teachers (Li et al., 2012), which are critical to the education of students with disabilities (Alquraini & Gut, 2012). Instructional supports were provided within the course, however, to ensure student success within the peer and self-review process. They included the use of a checklist and detailed rubric (see Table 2), as well as practice conducting self-review and submitting materials in the online review system. Review without such supports may have resulted in different outcomes.

Despite explicit activity instructions detailing the self-review procedures and use of the online submission system and prior in-class activity carrying out this skill with supervision of the instructor, preservice teachers in the control group did not make substantial changes to their graphic organizer assignments during the review period. While they provided two submissions, the changes made within postsubmissions did not result in higher grades. This finding was unexpected.

Although benefits are noted within the literature related to self-review (Belski, 2009), such benefits were not noted within this study. The reason for a lack of growth in the control group with students who conducted self-review is unclear. The instructor provided the same supports to preservice teachers in the peer review and self-review groups during the in-class activity (e.g., how to use the rubric to guide revisions and feedback and how to submit pre- and postassignments online using the Turnitin system).

Limited self-review may have occurred because preservice teachers within the control group (a) were not motivated to perform self-review (see Case, 2007); (b) felt as if self-review was not necessary (e.g., were not able to identify their weaknesses; see Careless et al., 2011); /or (c) lacked accountability to evaluate their own work (as in Dettori et al., 2006). Perhaps peer review was more comfortable for the participants within this study or was a more accepted practice within the teacher training program than self-review.

Dettori et al. (2006) suggested that the skills required to self-review might require continued development during preservice education, especially those required for evaluating performance on novel tasks, so perhaps the practice of self-review requires additional instructional support to ensure preservice teacher success. Future research should examine more extensive instructional supports for developing self-review skills and incorporate qualitative measures to better understand the perceived benefits and limitations of online peer review versus self-review.

The results from this study also provide evidence that online peer review may improve preservice teacher performance on learning activities regardless of their primary content area. Performance results were disaggregated by primary content area. Graphic organizers from treatment group preservice teachers demonstrated large effect sizes. This finding provides support for the recommendation that peer reviewers be grouped by content area expertise. This practice of conscious assignment of peer reviews by the instructor with peer reviewers from similar content areas may be especially beneficial in large introductory education classes where preservice teachers from various content areas are enrolled in the same course, further research is needed to fully understand this benefit.

Anecdotal data from the course instructor indicated benefits to peer reviews from students with both similar (e.g., same content area) and dissimilar backgrounds (e.g., different content area). The instructor noted that students from content areas may be better at providing feedback related to the content of the graphic organizer, while students from different content areas may be better at providing feedback to simplify language and reduce jargon in the graphic organizer. The differences in peer review from similar and different content area students should be explored in future research.

Limitations

Although this study provides important new insights and findings, its limitations should be noted. First, students within the study were provided with instructional supports to ensure success in the peer review and self-review process. As such, the level of support an instructor might need to support students is important to consider within peer review processes to ensure student success.

This study examined the use of online peer review technology on preservice teacher development, but results reported are from one activity within multiple sections of a single university course. Additionally, this study is the first that has utilized online peer review to support preservice teachers in obtaining skills to support students with disabilities. As such, further research is needed to understand fully the impact of online peer review and its usefulness to prepare preservice teachers in their instruction of students with disabilities. This research should include examining the use of peer review on a range of learning activities. To investigate more fully, the implementation of social validity measures with preservice teachers should be included to determine their experience and perceived benefits and challenges of online peer review. Future research might explore the content within peer reviews to determine patterns of feedback provided by preservice teachers during peer review.

Implications for Practice

Given the results of this study and the potential benefits of online peer review both on the academic performance and skills required for preservice teachers, teacher educators should consider implementing peer review into teacher preparation courses. Within this study, it was found that preservice teachers who engaged in online peer review produced higher quality graphic organizers, a tool that has been widely used to support comprehension for students with disabilities in science, social studies, language arts, and mathematics classes (Barton-Arwood & Little, 2013; Dexter & Hughes, 2011). Similarly, in previous research, teachers who engaged in reflective processes, including peer review, have demonstrated improved educational products and practice (see Avery & Meyer, 2012; Lynch et al., 2012).

The application of peer review may look different across different content areas, with adjustments made based on specific learning outcomes for students. For example, peer review could be used with preservice social studies teachers to support the development of small group activities to help students develop an understanding of cultural and religious practices across the world. Preservice teachers could provide feedback prior to implementation of the activity in a field placement. Given the various applications, instructors are encouraged to explore how peer review might be implemented within their specialties to support reflection and improve preservice teacher skills and educational outcomes for all students.

Online peer review can support instructors in facilitating double-blind peer reviews, and online peer review technologies are available at many universities (Dahl, 2007). Although we utilized the PeerMark feature in TurnItIn, a widely available tool at many universities, other peer review tools within course management systems (e.g., peer review options within Blackboard, Canvas, and Moodle), as well as traditional peer review, where student work is exchanged manually by the instructor, may also be an option to support preservice teachers in developing collaborative and reflective skills. Peer review can also be implemented in instances in which distance learning is required, such as during the COVID-19 pandemic.

Instructors may not need to invest a lot of time or money to incorporate the peer review process in their classrooms, especially since many tools are readily available. However, teacher educators should consider the use of instructional supports with preservice teachers to support peer and self-review processes, such as those provided within this study (i.e., rubric to support scoring and preview of the review and submission process in class).

Conclusion

Overall, this study provides additional evidence to support the use of online peer review as an effective tool to improve preservice teacher performance. Within this study, the use of online peer review supported the creation of graphic organizers by preservice teachers, a teaching strategy that can be challenging to master. Preservice teachers who engaged in online peer review outperformed preservice teachers in the control group (i.e., self-review) with large effect sizes noted for all content areas. Future research should continue to explore the impact of online peer review and its benefits for preservice teachers across a range of teaching practices and incorporate qualitative or social validity measures to understand the process preservice teachers engage in when conducting both peer and self-reviews.

References

Akhondi, M., Malayeri, F., & Samad, A. (2011). How to teach expository text structure to facilitate reading comprehension. The Reading Teacher, 64(5), 368–372. https://doi.org/10.1598/RT.64.5.9

Alquraini, T., & Gut, D. (2012). Critical components of successful inclusion of students with severe disabilities. International Journal of Special Education, 27, 1-18. https://www.learntechlib.org/p/55011/

Andrade, H., & Valtcheva, A. (2009). Promoting learning and achievement through self-assessment. Theory into Practice, 48(1), 12-19. https://doi.org/10.1080/00405840802577544

Avery, L. M., & Meyer, D. Z. (2012). Teaching science as science is practiced: Opportunities and limits for enhancing preservice elementary teachers’ self‐efficacy for science and science teaching. School Science and Mathematics, 112(7), 395-409. https://doi.org/10.1111/j.1949-8594.2012.00159.x

Baecher, L., Kung, S. C., Jewkes, A. M., & Rosalia, C. (2013). The role of video for self-evaluation in early field experiences. Teaching and Teacher Education, 36, 189-197. https://doi.org/10.1016/j.tate.2013.08.001

Bartlett, I., & Spicer, D. (2015). Everyone’s a critic: The power of peer review. In C. Klopper & S. Drew (Eds.), Teaching for learning and learning for teaching (pp. 155-167). Sense Publishers.

Barton-Arwood, S. M., & Little, M. A. (2013). Using graphic organizers to access the general curriculum at the secondary level. Intervention in School and Clinic, 49, 6-13. https://doi.org/10.1177/1053451213480025

Belski, I (2009, November). Improving student satisfaction with feedback by engaging them in self-assessment and reflection. In ATN Assessment Conference 2009: Assessment in Different Dimensions (pp. 17-27). 

Bentley-Williams, R., Grima-Farrell, C., Long, J., & Laws, C. (2017). Collaborative partnership: Developing preservice teachers as inclusive practitioners to support students with disabilities. International Journal of Disability, Development and Education64(3), 270-282. https://doi.org/10.1080/1034912X.2016.1199851

Boase-Jelinek, D., Parker, J., & Herrington, J. (2013). Student reflection and learning through peer reviews. Issues in Educational Research, 23(2), 119-131. http://www.iier.org.au/iier23/boase-jelinek.html

Bonk, C., Wisher, R., & Lee, J. (2004).  Moderating learner-centered e-learning: Problems and solutions, benefits and implications. In T. Roberts (Ed.), Online collaborative learning: Theory and practice (pp. 54-85).  Idea Group Publishing.

Britton, L. R., & Anderson, K. A. (2010). Peer coaching and preservice teachers: Examining an underutilized concept. Teaching and Teacher Education, 26, 306-314. https://doi.org/10.1016/j.tate.2009.03.008

Budge, K., & Gopal, S. (2009, November). Feedback: Working from the student perspective. In ATN Assessment Conference 2009: Assessment in Different Dimensions (pp. 74-83).  

Carless, D., Salter, D., Yang, M., & Lam, J. (2011). Developing sustainable feedback practices. Studies in Higher Education, 36(4), 395-407. https://doi.org/10.1080/03075071003642449

Case, S. (2007). Reconfiguring and realigning the assessment feedback processes for an undergraduate criminology degree. Assessment & Evaluation in Higher Education, 32(3), 285-299. https://doi.org/10.1080/02602930600896548

Catley, K. M., & Novick, L. R. (2009). Digging deep: Exploring college students’ knowledge of macroevolutionary time. Journal of Research in science teaching, 46(3), 311-332. https://doi.org/10.1002/tea.20273

Chan, P. E., Konrad, M., Gonzalez, V., Peters, M. T., & Ressa, V. A. (2014). The critical role of feedback in formative instructional practices. Intervention in School and Clinic, 50(2), 96-104. https://doi.org/10.1177/1053451214536044

Chen, J. C. (2014). Teaching nontraditional adult students: Adult learning theories in practice. Teaching in Higher Education, 19(4), 406-418.

Cheung, R. (2009). The use of self-assessment to foster students’ learning in teacher education: An experience in teaching practice. Action in Teacher Education, 31(1), 49-57. https://doi.org/10.1080/01626620.2009.10463510

Cho, K., & MacArthur, C. (2010). Student revision with peer and expert reviewing. Learning and Instruction, 20(4), 328-338. https://doi.org/10.1016/j.learninstruc.2009.08.006

Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Erlbaum.

Dahl, S. (2007). Turnitin®: The student perspective on using plagiarism detection software.Active Learning in Higher Education, 8, 173-191. https://doi.org/10.1177/1469787407074110

Darling-Hammond, L., Hyler, M. E., Gardner, M. (2017). Effective teacher professional development. Learning Policy Institute. https://learningpolicyinstitute.org/sites/default/files/productfiles/Effective_Teacher_Professional_Development_REPORT.pdf

Darling-Hammond, L., & Lieberman, A. (Eds.). (2013). Teacher education around the world: Changing policies and practices. Routledge.

Dettori, G., Giannetti, T., & Persico, D. (2006). SRL in online cooperative learning: Implications for pre‐service teacher training. European Journal of Education, 41, 397-414. https://doi.org/10.1111/j.1465-3435.2006.00273.x

Dexter, D. D., & Hughes, C. A. (2011). Graphic organizers and students with learning disabilities: A meta-analysis. Learning Disability Quarterly, 34(1), 51-72. https://doi.org/10.1177/073194871103400104

Douglas, S. N., Chapin, S., & Nolan, J. (2016). Special education teachers’ experiences supporting and supervising paraeducators: Implications for special and general education settings. Teacher Education and Special Education, 39(1), 60-74. https://doi.org/10.1177/0888406415616443

Freeman, M. (1995). Peer assessment by groups of group work. Assessment und Evaluation in Higher Education, 20, 289-300.

Friend, M. P., & Bursuck, W. D. (2012). Including students with special needs: A practical guide for classroom teachers (6th ed.). Pearson.

Gehringer, E. F. (2011, June). Experience with software support for managing student-authored wiki textbooks. In Proceedings of the 2011 American Society for Engineering and Education annual conference. (pp. 26-29).

Goldin, I. M., & Ashley, K. D. (2010, June). Eliciting informative feedback in peer review: importance of problem-specific scaffolding. In V. Aleven, J. Kay, & J. Mostow (Eds.), Intelligent tutoring systems (pp. 95-104). Springer.

Gulikers, J., Sluijsmans, D., Baartman, L., & Bartolo, P. (2009). The power of assessment in teacher education. In P. Bartolo, J. McCall, M. Montane, L. Shagrir, & I. Zogla, I. (Eds.), Becoming a teacher educator: Theory and practice for teacher educators (pp. 173-190). Springer.

Hämäläinen, H., Hyyrynen, V., Ikonen, J., & Porras, J. (2011). Applying peer-review for programming assignments. International Journal for Information Technologies and Security, 1, 3-17.

Hammond, Z. (2014). Culturally responsive teaching and the brain: Promoting authentic engagement and rigor among culturally and linguistically diverse students. Corwin.

Han, A. (2012, June 10-14). Using calibrated peer review to encourage writing. In Proceedings of the 2012 Association of Small Computer Users in Education Summer Conference (pp. 23-32).

Hanrahan, S., & Isaacs, G. (2001) Assessing self- and peer assessment: The students’ views. Higher Education Research & Development, 20(1), 53-70. https://doi.org/10.1080/07294360123776

Harvey, M., Yssel, N., Bauserman, A., & Merbler, J. (2010). Preservice teacher preparation for inclusion: An exploration of higher education teacher-training institutions. Remedial and Special Education31, 24-33. https://doi.org/10.1177/0741932508324397

Hatziapostolou, T., & Paraskakis, I. (2010). Enhancing the impact of formative feedback on student learning through an online feedback system. Electronic Journal of e-Learning, 8(2), 111-122. https://www.learntechlib.org/p/55178/

Heard, K., & Peltier, C. (2021). Using video-analysis in the professional development of a classroom teacher. Preventing School Failure: Alternative Education for Children and Youth, 65(2), 163-174. https://doi.org/10.1080/1045988X.2020.1864714

Hedges, L. V. (1981). Distribution theory for Glass’s estimator of effect size and related estimators. Journal of Educational and Behavioral Statistics, 6, 107-128. https://doi.org/10.3102/10769986006002107

Jhangiani, R. S. 2016. The impact of participating in a peer assessment activity on subsequent academic performance. Teaching of Psychology, 43 (3), 180–186. https://doi.org/10.1177/0098628316649312

Knight, V. F., Spooner, F., Browder, D. M., Smith, B. R., & Wood, C. L. (2013). Using systematic instruction and graphic organizers to teach science concepts to students with autism spectrum disorders and intellectual disability. Focus on Autism and Other Developmental Disabilities, 28(2), 115-126. https://doi.org/10.1177/1088357612475301

Knowles, M.S., Holton, E.F., Swanson, R.A. (2011) The adult learner: The definitive classic in adult education and human resource development (7th ed). Elsevier.

Kong, S., Shroff, R. H., & Hung, H. (2009). A web enabled video system for self-reflection by student teachers using a guiding framework. Australasian Journal of Educational Technology, 25(4), 544-558. https://doi.org/10.14742/ajet.1128

Li, C., Dong, Z., Untch, R. H., & Chasteen, M. (2012, January). Facilitating peer review in an online collaborative learning environment for computer science students. In Proceedings of the International Conference on Frontiers in Education: Computer Science and Computer Engineering (pp. 1-7). 

Liang, J. C., & Tsai, C. C. (2010). Learning through science writing via online peer assessment in a college biology course. The Internet and Higher Education, 13, 242-247. https://doi.org/10.1016/j.iheduc.2010.04.004

Lundstrom, K., & Baker, W. (2009). To give is better than to receive: The benefits of peer review to the reviewer’s own writing. Journal of second language writing, 18(1), 30-43. https://doi.org/10.1016/j.jslw.2008.06.002

Lynch, R., McNamara, P. M., & Seery, N. (2012). Promoting deep learning in a teacher education programme through self-and peer-assessment and feedback. European Journal of Teacher Education, 35(2), 179-197.

Mason, L. H., & Hedin, L. R. (2011). Reading science text: Challenges for students with learning disabilities and considerations for teachers. Learning Disabilities Research & Practice, 26, 214-222. https://doi.org/10.1111/j.1540-5826.2011.00342.x

Meerah, T. S. M., & Halim, L. (2011). Improve feedback on teaching and learning at the university through peer group. Procedia-Social and Behavioral Sciences, 18, 633-637. https://doi.org/10.1016/j.sbspro.2011.05.093

Mitchell, D. E., Howard, B., Meetze-Hall, M., Hendrick, L. S., & Sandlin, R. (2017). The new teacher induction experience: Tension between curricular and programmatic demands and the need for immediate help. Teacher Education Quarterly, 44(2), 79-104. https://www.jstor.org/stable/90010519

Nicol, D., Thomson, A., & Breslin, C. (2014). Rethinking feedback practices in higher education: a peer review perspective. Assessment & Evaluation in Higher Education, 39(1), 102-122. https://doi.org/10.1080/02602938.2013.795518

Nierengarten, G. (2013). Supporting co-teaching teams in high schools: Twenty research-based practices. American Secondary Education, 42, 73-83. https://www.jstor.org/stable/43694178

Nolan, J., & Hoover, L. A. (2010). Teacher supervision and evaluation. Wiley.

Odom, S., Glenn, B., Sanner, S., & Cannella, K. A. (2009). Group peer review as an active learning strategy in a research course. International Journal of Teaching and Learning in Higher Education, 21(1), 108-117.

Orsmond, P., Merry, S., & Reiling, K. (2000) The use of student derived marking criteria in peer and self-assessment. Assessment and Evaluation in Higher Education, 25, 23-38.

Pawan, F., & Fan, W. (2014). Sustaining expertise through collaborative/peer-mediated and individual reflections: The experiences of Chinese English language teachers. Teacher Education Quarterly, 41(4), 71-88. http://www.jstor.org/stable/teaceducquar.41.4.71

Peppers, G. J. (2015). Teachers’ perceptions and implementation of professional learning communities in a large suburban high school. National Teacher Education Journal, 8(1), 25-31.

Sluijsmans, D. M., Brand-Gruwel, S., van Merriënboer, J. J., & Bastiaens, T. J. (2002). The training of peer assessment skills to promote the development of reflection skills in teacher education. Studies in Educational Evaluation, 29(1), 23-42.

Solis, M., Vaughn, S., Swanson, E., & McCulley, L. (2012). Collaborative models of instruction: The empirical foundations of inclusion and co‐teaching. Psychology in the Schools, 49(5), 498-510.

Søndergaard, H., & Mulder, R. A. (2012). Collaborative learning through formative peer review: Pedagogy, programs and potential. Computer Science Education, 22(4), 343-367. https://doi.org/10.1080/08993408.2012.728041

Stetter, M. E., & Hughes, M. T. (2010). Using story grammar to assist students with learning disabilities and reading difficulties improve their comprehension. Education and Treatment of Children, 33(1), 115-151. https://www.jstor.org/stable/42900057

Taylor, D., & Hamdy, H. (2013). Adult learning theories: Implications for learning and teaching in medical education, Medical Teacher, 35(11), e1561- e1572. https://doi.org/10.3109/0142159X.2013.828153

Thompson, D., & McGregor, I. (2009). Online self-and peer assessment for group work. Education + Training, 51, 434-447. https://doi.org/10.1108/00400910910987237

Tricarico, K., & Yendol-Hoppey, D. (2012). Teacher learning through self-regulation. Teacher Education Quarterly. 39(1), 139-158. https://www.jstor.org/stable/23479566

Tsai, M. H. (2017). The most preferred and effective reviewer of L2 writing among automated grading system, peer reviewer, and teacher. World Journal of Education, 7(4), 60.

TurnItIn [Computer software]. (2015). iParadigms.

 74 total views,  1 views today