{"id":11079,"date":"2021-10-22T18:21:11","date_gmt":"2021-10-22T18:21:11","guid":{"rendered":"https:\/\/citejournal.org\/\/\/"},"modified":"2022-02-18T20:46:14","modified_gmt":"2022-02-18T20:46:14","slug":"the-effects-of-robotics-professional-development-on-science-and-mathematics-teaching-performance-and-student-achievement-in-underserved-middle-schools","status":"publish","type":"post","link":"https:\/\/citejournal.org\/volume-21\/issue-4-21\/mathematics\/the-effects-of-robotics-professional-development-on-science-and-mathematics-teaching-performance-and-student-achievement-in-underserved-middle-schools","title":{"rendered":"The Effects of Robotics Professional Development on Science and Mathematics Teaching Performance and Student Achievement in Underserved Middle Schools"},"content":{"rendered":"\n

Proponents of integrating technology into science and mathematics curricula argue that it aids students in acquiring valuable disciplinary skills, such as logical analysis and critical thinking, and prepares them for real-world problem solving using modern tools (Castledine & Chalmers, 2011). Federal legislation in the United States, such as the America COMPETES Reauthorization Act<\/em> of 2010, has acknowledged the importance of developing these types of skills in schools.<\/p>\n\n\n\n

In 2018, the Committee on STEM Education of the National Science and Technology Council published a report charting a 5-year strategic plan for science, technology, engineering, and mathematics (STEM) education. In response to this plan, in-service teacher education has recently focused on initiatives that strengthen STEM subjects\u2019 cross-cutting curricula (see also K-12 Computer Science Framework Steering Committee, 2016), such as robotics instruction, which has been identified as an effective integrative approach to teaching STEM principles (Scaradozzi et al., 2019). This study investigated the effects of robotics professional development (PD) on middle level science and mathematics teachers\u2019 (N<\/em> = 11) robotics instruction and students\u2019 (N<\/em> = 291) mathematics achievement.<\/p>\n\n\n\n

Relevant Literature<\/h2>\n\n\n\n

Constructionism<\/h3>\n\n\n\n

Papert\u2019s early findings on computer programming instruction (1980) and constructionist learning (1993) have contributed substantially to the evolution of robotics education, emphasizing the combination of student-centered activities with mechanical tools to solve practical problems. For example, research by Mikropoulos and Bellou (2013) indicated that constructionism impacted robotics education significantly, as the majority of educational robotics studies in their sample utilized some type of constructionist approach.<\/p>\n\n\n\n

Constructionism is both a theory of learning and an instructional strategy (Ardito et al., 2014). Constructionism theorizes that knowledge is not simply transferred from the instructor to the student (Papert, 1980, 1993). Instead, learning is brought about through the construction, deconstruction, and reconstruction of students\u2019 understanding, based on experiences fostered by physical construction of learning artifacts (Kafai & Resnick, 1996; Mikropoulous & Bellou, 2013; Resnick & Silverman, 2005). Constructionism includes two entwined types of construction: the construction of products and the construction of meaning (Kafai & Resnick, 1996). The construction of the concrete objects aids in the construction of mental models (Mikropoulous & Bellou, 2013).<\/p>\n\n\n\n

In 1998, the LEGO\u00ae company released their constructible robotics kits \u2013 LEGO MINDSTORMS\u00ae \u2013 named after Papert\u2019s (1980) seminal work on constructionism entitled Mindstorms: Children, Computers and Powerful Ideas<\/em> (Chambers & Carbonaro, 2003). These LEGO MINDSTORMS kits were developed by some of Papert\u2019s prot\u00e9g\u00e9s as an archetypal constructionist learning tool (Ardito et al., 2014). Since their release, LEGO\u2019s MINDSTORMS kits and curricula have advanced to the forefront of robotics education (Eguchi, 2013, Martin et al., 2000) as well as student robotics competitions including FIRST LEGO League and World Robot Olympiad (Zhang & Wan, 2020).<\/p>\n\n\n\n

Research by Yolcu and Demirer (2017) analyzed studies about robotics education and found that over 66% of such studies utilized buildable LEGO robotics kits (over 40% used LEGO MINDSTORMS, in particular) and over 90% used LEGO or similar buildable robotics kits. To summarize, constructionism is heavily associated with educational robotics due to the constructable and customizable nature of educational robotics kits, like LEGO MINDSTORMS.<\/p>\n\n\n\n

Teachers and Robotics Professional Development<\/h3>\n\n\n\n

Researchers have noted that few studies have examined the impact of robotics PD on teachers (Kim et al., 2015; Yuksel et al., 2020). Studies that focused on training teachers in STEM concepts with educational robotics have had various aims and findings (Guven & Cakir, 2020; Kay et al., 2014; Kopcha et al., 2017; Scaradozzi et al., 2019; Sullivan & Moriarty, 2009). For example, Kopcha et al. (2017) and Scaradozzi et al. (2019) found that educational robotics STEM PD activities were effective in teaching integrative STEM principles to teachers.<\/p>\n\n\n\n

Similarly, studies have noted statistically significant programming and robotics knowledge increases among in-service teacher participants (Kay et al., 2014; Scaradozzi et al., 2019; Sullivan & Moriarty, 2009). Researchers have found that in-service teachers\u2019 confidence with robotics increased significantly because of workshops, as well (Kay et al., 2014; Scaradozzi et al., 2019; Sullivan & Moriarty, 2009).<\/p>\n\n\n\n

Sullivan and Moriarty (2009) suggested that the perceptions and practices among teachers learning about robotics and integrating robotics concepts into instruction may change through robotics experiences. Research by Guven and Cakir (2020) and Kopcha et al. (2017) found that the teachers integrated or intended to integrate robotics into their future instruction, which suggested that robotics were an efficient way to teach STEM concepts to teachers and influence their perceptions and practices. These studies exemplify the different aims and findings of literature exploring teachers and robotics PD.<\/p>\n\n\n\n

Students and Robotics<\/h3>\n\n\n\n

The impact of robotics integration in science and mathematics instruction has been recently investigated for students in numerous grade levels. Previous inquiry has examined the integration of robotics kits as constructionist tools for students to learn STEM content through hands-on programming tasks at the elementary and middle levels (Bers, 2010; Fessakis et al., 2013; Koumoullos, 2013; Mikropoulos & Bellou, 2013). Researchers have noted that these kits can be effective for younger learners because they integrate block-based programming languages that diminish the tedium of coding text line-by-line and the associated syntax errors that novice programmers often make (Falloon, 2016; Kim et al., 2018).<\/p>\n\n\n\n

Studies have indicated that robotics activities develop student problem-solving abilities (Bers et al., 2014; Datteri et al., 2013) and increase meaningful learning (Kaloti-Hallak et al., 2019). Beyond achievement gains, Yesharim and Ben-Ari (2018) noted that students learning computer science constructs with robotics demonstrated high motivation to succeed. Other researchers have detected positive effects of robotics on students\u2019 STEM self-efficacy (Hall-Lay, 2018; Leonard et al., 2016). Williams et al. (2012) studied the impact of robotics on science and mathematics understanding of elementary, middle, and high schoolers. From the pretest to the posttest, students\u2019 mathematics understanding increased 25% and their science understanding increased 47%.<\/p>\n\n\n\n

While studies specifically evaluating robotics in middle school student populations are scarce (Casler-Failing, 2018), such studies have shown mathematics gains among students (Ardito et al., 2014; Casler-Failing, 2017; Castledine & Chalmers, 2011). A study by Ardito et al. (2014) investigated the impact of robotics on sixth graders\u2019 mathematics achievement. The study was conducted in the mathematics and science classrooms and utilized programming problem-solving activities and challenges linked to algebra, measurement, and probability. The results of the study indicated that students\u2019 achievement on a state standardized mathematics test in algebra, measurement, and probability improved, but not to statistically significant levels.<\/p>\n\n\n\n

Further, a study by Castledine and Chalmers (2011) examined the correlation between sixth-grade mathematics students\u2019 problem-solving decisions related to speed, distance, time, and angles in robotics programming races and mazes and their abilities to translate those strategies to authentic mathematics problems. Students exhibited growth in their problem-solving skills in mathematics because of robotics learning activities.<\/p>\n\n\n\n

At the seventh-grade level, mathematics students who were learning graphing, measurement, scaling, speed, distance, and time through robotics activities in a study by Casler-Failing (2017) showed improvement in their understanding of proportional reasoning skills, especially among low-performing students. Eighth grade science students in research by Williams et al. (2012) showed learning growth in their understanding of the mathematics and science concepts of force, velocity, and acceleration after 90 minutes of hands-on robotics activities. These studies investigated the impacts of robotics integration on students in science and mathematics at numerous grade levels.<\/p>\n\n\n\n

The Present Study<\/h3>\n\n\n\n

The research reported here adds to the limited literature on the impact of robotics PD on teachers (Kim et al., 2015; Yuksel et al., 2020), as well as the limited literature specifically analyzing how educational robotics impact middle school students\u2019 mathematics achievement (Casler-Failing, 2018). Three novel aspects of this study distinguish it from previous research: (a) the context, (b) the length of the treatment, and (c) the use of nationally normed and demographically matched control samples.<\/p>\n\n\n\n

First, this study focused on a novel context for educational robotics research: underserved middle schools. Second, this study did not simply focus on the short-term impacts of educational robotics. In this study, teacher-participants took part in over 75 contact hours of extensive PD, and teacher-participants and student-participants were evaluated over the course of a year. Finally, the use of both nationally normed and demographically matched control samples has provided two sets of control groups with which to compare the student participants\u2019 results. The national norm data were used to contrast the student-participants\u2019 mathematics growth against the rest of the country among sixth, seventh, and eighth graders. The control sample data were used to precisely contextualize the student-participants\u2019 mathematics growth against students\u2019 growth from similarly disadvantaged schools with demographically matched backgrounds.<\/p>\n\n\n\n

The research questions in this study were as follows:<\/p>\n\n\n\n

  1. What are the effects of robotics professional development sessions on middle school science and mathematics teachers\u2019 teaching performance?<\/li>
  2. How do robotics professional development sessions for middle school science and mathematics teachers impact students\u2019 mathematics achievement?<\/li><\/ol>\n\n\n\n

    Methodology<\/h2>\n\n\n\n

    The PD sessions occurred across the span of an academic year, bookended by week-long summer PD sessions. In turn, students were taught by the teachers who participated in the PD sessions and used the teachers\u2019 robotics kits in robotics-centric science and mathematics lessons. Using quantitative methods, the researchers evaluated teaching performance and student mathematics growth. Specific methodological details will be explained in the paragraphs below.<\/p>\n\n\n\n

    Setting and Participants<\/h3>\n\n\n\n

    For this grant-funded project, the researchers partnered with a regional public school district identified as high-needs in the southeastern United States for its historically low socioeconomic status and low student achievement. The grant\u2019s call focused on increasing academic achievement in the state by improving teacher quality. The researchers identified this district based on the project\u2019s potential to have a more meaningful impact supporting a high-needs school district, as opposed to others in the region. At the time of the study, the district served 5,200 students; 90% of students lived in poverty, and it had a 63% senior graduation rate. Both teachers and students from this school district served as this study\u2019s participants. Informed consent was obtained from the teacher-participants and student-participant consent was managed by the individual schools.<\/p>\n\n\n\n

    Teacher-Participants<\/em><\/h3>\n\n\n\n

    After a district-wide survey of interest in robotics PD, administrators selected participants who (a) taught middle school science or mathematics (other subject area teachers who expressed interest were excluded) and (b) were willing to participate in the year-long study. All the teachers who had expressed interest and met these criteria were selected.<\/p>\n\n\n\n

    In total, the district selected 15 science and mathematics teachers spread among four of the district\u2019s middle schools. There was attrition of teacher-participants over the year-long duration of the study. Two of the 15 teacher-participants were lost due to career advancement, one dropped out to focus on 1st-year teaching responsibilities, and one could not attend the second summer of PD experiences due to a family emergency. Of the remaining 11 participants (four male and seven female teachers), each taught sixth (three), seventh (two), or eighth (six) grade science or mathematics.<\/p>\n\n\n\n

    Seven teacher-participants taught mathematics, and four taught science. Four teachers identified themselves as Black, four as White, and three as Asian. All participants received a robotics kit, a laptop, a stipend, supplementary sensors, robotics classroom integration books, and three graduate course credits for their participation that could be applied toward continuing education or a degree program.<\/p>\n\n\n\n

    Student-Participants<\/em><\/h3>\n\n\n\n

    Student-participants also took part in this study. All the students in science and mathematics classes taught by the teachers participating in this study were used as a convenience sample. The student-participants represented 13 classes: five science and eight mathematics. The sample consisted of 171 female and 120 male student-participants. Of the 291 student-participants, 207 identified as Black, 74 identified as White, five identified as Hispanic, and five identified as Asian. More detailed demographic data is shared in Table 1.<\/p>\n\n\n\n

    Table 1<\/strong>
    Student-Participant Demographic Information<\/em><\/p>\n\n\n\n\n\n\n\t\n\n\t\n\t\n\t\n\t\n\t
    Grade<\/th>Male<\/th>Female<\/th>Total<\/th>\n<\/tr>\n<\/thead>\n
    <\/td>Asian<\/td>Black<\/td>Hispanic<\/td>White<\/td>Asian<\/td>Black<\/td>Hispanic<\/td>White<\/td><\/td>\n<\/tr>\n
    6<\/td>0<\/td>22<\/td>0<\/td>13<\/td>2<\/td>27<\/td>1<\/td>17<\/td>82<\/td>\n<\/tr>\n
    7<\/td>0<\/td>14<\/td>0<\/td>2<\/td>0<\/td>21<\/td>1<\/td>10<\/td>48<\/td>\n<\/tr>\n
    8 <\/td>0<\/td>62<\/td>1<\/td>6<\/td>3<\/td>61<\/td>2<\/td>26<\/td>161<\/td>\n<\/tr>\n
    Total<\/td>0<\/td>98<\/td>1<\/td>21<\/td>5<\/td>109<\/td>4<\/td>53<\/td>291<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n\n\n\n

     <\/em><\/h4>\n\n\n\n

    Trainers and Coaches<\/h3>\n\n\n\n

    Two trainers were selected to deliver the PD content to teacher-participants. Both were full-time university faculty members credentialed in robotics instruction, one from the college of science and the other from the college of education. In addition to these two trainers, the researchers also hired three experienced robotics coaches from a different school district to provide scaffolding, instructional design, and PD support to the participating teachers. Both the trainers and the coaches received a stipend as compensation for their services throughout the project.<\/p>\n\n\n\n

    Research Design<\/h3>\n\n\n\n

    Teacher-participants began the year-long series of PD sessions with a 1-week (35 hours) campus-based summer workshop that integrated formal robotics technology and pedagogy lessons. Constructionism (Papert, 1993) served as the theoretical framework for the PD curriculum, based on its alignment in the literature to educational robotics. The PD sessions were designed with a constructionist framework, as teacher-participants constructed their robots to adapt to different problem scenarios utilizing mathematics and science knowledge.<\/p>\n\n\n\n

    This curriculum incorporated the constructionist facets of knowledge construction through physical construction as teacher-participants built and customized their robots to solve problems, as well as a collaborative environment (Papert, 1980, 1993). Teacher-participants could then teach with these same constructionist practices in their own classrooms, facilitating learning through activities that required their students to build and customize their robots to solve authentic problems, such as mazes, in a collaborative environment. The PD curriculum was reviewed for face validity before implementation by four experts: three with expertise in robotics education and one with expertise in education. These lessons were led by the two trainers and included independent practice activities and challenges.<\/p>\n\n\n\n

    Individual support of the teacher-participants was facilitated by the three coaches. LEGO MINDSTORMS EV3 robotics kits were utilized by teacher-participants for all instructional activities. The LEGO MINDSTORMS EV3 kits included a programmable control unit, motors, sensors, building blocks, gears, and other mechanical pieces. LEGO MINDSTORMS EV3 kits were selected due to their developmental appropriateness for middle school students (Martin et al., 2000), the population taught by the teacher-participants.<\/p>\n\n\n\n

    The first campus-based summer week-long PD series focused on related science, mathematics, and robotics principles, specifically odometry, dead reckoning, sensors, flow control, data wires, gears, and problem-solving. The instruction included a blend of lecture, demonstration, and discussion, followed by hands-on individual or team activities that included relevant programming challenges. Each day\u2019s lesson topic, its associated science and mathematics topics, and challenges are outlined in Table 2.<\/p>\n\n\n\n

    Table 2<\/strong>
    First Campus-Based Summer Workshop Activities and Challenges<\/em><\/p>\n\n\n\n\n\n\n\t\n\n\t\n\t\n\t\n\t\n\t
    Lessons<\/th>Science and Mathematics Topics<\/th>RoboMaze Challenges<\/th>\n<\/tr>\n<\/thead>\n
    Dead Reckoning<\/td>Odometry; Calculating wheel circumference and distance per rotation; Dead reckoning; Debugging; Pseudocode<\/td>Navigate with dead reckoning; Navigate with dead reckoning (black diamond)<\/td>\n<\/tr>\n
    Flow Control <\/td>Rotations and distance; Programming loop, switch, and wait; Touch sensory input<\/td>Navigate with touch sensor<\/td>\n<\/tr>\n
    Sensors<\/td>Sound waves; Sonar; Programming switch; Light intensity; Light reflection<\/td>Navigate with ultrasonic sensor; Navigate with color sensor<\/td>\n<\/tr>\n
    Data Wires and Gears<\/td>Programming decision making; Gears; Gear ratios; Transmitting data; Programming loops; Positive and negative integers; Calculating time and speed<\/td>Navigate with all sensors with obstacle included<\/td>\n<\/tr>\n
    The Challenge<\/td>Cumulative science and mathematics concepts<\/td>Navigate with obstacle included for time (black diamond)<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n\n\n\n

    The programming challenges during these sessions focused on applying the targeted science and mathematics principles to solve problems in the RoboMaze. The RoboMaze required participants to navigate their robots by applying science and mathematics concepts, such as calculating distance, applying sonar, and engineering gearing ratios. In addition, problem-solving in the RoboMaze required participants to write and customize code written in a block-based programming language that controlled the robots.<\/p>\n\n\n\n

    Multiple RoboMazes were constructed from 4×8 foot melamine sheets and 2 ft x 4 ft boards to facilitate efficient access by all teacher-participants. As depicted in Figure 1, the RoboMaze could be navigated from top left to bottom right for a moderate challenge (Start and Finish for the moderate challenge are denoted on the schematic), or from bottom left to top right for a more difficult challenge with more turns (marked by the black diamonds). The RoboMaze required teacher-participants to utilize the programming, mathematics, and science knowledge they had built in each lesson to successfully navigate their robot through the maze.<\/p>\n\n\n\n

    Figure 1<\/strong>
    The RoboMaze Path<\/em><\/p>\n\n\n\n

    \"\"
    The path from the noted Start and Finish locations requires fewer turns and is, thus, a lower difficulty than the black diamond path between the opposite corners.<\/figcaption><\/figure>\n\n\n\n

    During the ensuing fall and spring semesters of the academic year, teacher-participants were observed utilizing robotics in their classrooms by the researchers and received additional training and evaluation. Teacher-participants were required to attend two in-person robotics workshops with the trainers, participate in two live webinars for additional training, and attend a state educational technology conference that showcased numerous robotics sessions.<\/p>\n\n\n\n

    During the face-to-face workshops and live webinars, the teacher-participants were introduced to new programming concepts, sensors, challenges, club resources, and in-class robotics integration strategies. New robotics integration books and sensors, such as the temperature and infrared sensor\/beacon, were distributed during the fall and spring workshops to expand teacher-participants\u2019 integration of robotics in the classroom. In addition, teacher-participants shared their experiences of teaching with robotics among their teacher-participant peers both in-person in the workshops and through a social media group created for the teacher-participants.<\/p>\n\n\n\n

    The culminating challenge during a workshop in the spring semester of the academic year was the LEGO MINDSTORMS EV3 Animal Allies Challenge, depicted in Figure 2. In this challenge, teacher-participants were given various tasks to program their robots to complete while navigating an obstacle course. Teacher-participants were also assigned homework throughout the project tenure, such as lesson plans, implementation videos, and critical reflections. In-class observations took place during the fall and spring semesters during the academic year.<\/p>\n\n\n\n

    Figure 2<\/strong>
    Teacher-Participants Discuss Problem-Solving in the Animal Allies Challenge During a Spring Workshop<\/em><\/p>\n\n\n\n

    \"\"<\/figure>\n\n\n\n

    The following summer, a final 1-week (35 hours) series of PD was conducted on campus using the same integrated model as the first summer. This summer workshop combined advanced robotics technology training with advanced instructional design training. The topics of instruction in the second summer workshop focused on training teachers to teach engineering design to their students. The engineering design process of planning, design, implementation, and improvement was taught to teacher-participants to integrate into science and mathematics. Engineering design was chosen as the next step in the curriculum to contextualize robotics instruction in real-world problem-solving. The lessons in the second summer workshop utilized more advanced problem-solving scenarios and various obstacle courses. Best practices for using robotics in the classroom were also analyzed. The culminating challenge for the second summer workshop was the LEGO MINDSTORMS Education EV3 Space Challenge. Shown in Figure 3, this challenge contained various tasks and obstacles for the teacher-participants to solve.<\/p>\n\n\n\n

    Figure 3<\/strong>
    A Teacher-Participant Brainstorms a Solution to a Problem in the LEGO MINDSTORMS EV3 Space Challenge During the Second Summer\u2019s Week of PD<\/em><\/p>\n\n\n\n

    \"\"<\/figure>\n\n\n\n

    Data Sources  <\/h3>\n\n\n\n

    Quantitative methods were used for this study. Quantitative data were gathered through multiple instruments, which included pre\/post teacher robotics teaching competency surveys, teacher lesson observations, and pre\/post MAP exams for students. As outlined in Table 3, these data sources were used to answer the two research questions.<\/p>\n\n\n\n

    Table 3<\/strong>
    Research Questions, Data Sources, and Data Analysis Method Alignment<\/em><\/p>\n\n\n\n\n\n\n\t\n\n\t\n\t
    Research Questions <\/th>Data Sources<\/th>Data Analysis<\/th>\n<\/tr>\n<\/thead>\n
    RQ 1: What are the effects of robotics professional development sessions on middle school science and mathematics teachers\u2019 teaching performance<\/td>Pre\/post robotics teaching competency surveys
    \n Teaching observations <\/td>    		Descriptive statistics
    \n Paired samples t<\/em>-tests<\/td>\n<\/tr>\n
    RQ 2: How do robotics professional development sessions for middle school science and mathematics teachers impact students\u2019 mathematics achievement? <\/td>Pre\/post MAP exams<\/td>Descriptive statistics
    \n Paired samples t<\/em>-tests<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n\n\n\n

    Pre\/Post Robotics Teaching Competency Surveys<\/em><\/h3>\n\n\n\n

    Teacher-participants self-assessed their robotics\u2019 capabilities with 20-statement pre\/post robotics teaching competency surveys. The first survey was given before the series of PD began, and the second survey was given after all PD sessions had been completed a year later. The teacher-participants assessed themselves on five categories, consisting of four statements each. This instrument used a 4-point Likert scale, with 1 being the lowest and 4 being the highest level of competency. As recommended by Cronbach (1950) and Nunnally (1967), a 4-point forced choice Likert scale was used to prevent participants from giving a response set of neutral answers. These five categories of statements were (a) hands-on robotics project curriculum planning (e.g., \u201cKnowledge of how to integrate robotics into my curriculum\u201d), (b) robotics and problem-solving skills (e.g., \u201cUse of robotics technology to facilitate higher order and complex thinking skills\u201d), (c) robotics and science inquiry (e.g., \u201cUse of the science inquiry process to debug programs\u201d), (d) robotics and design skills (e.g., \u201cCreating and building stable structures with LEGO or other materials\u201d), and (e) robotics and philosophical issues (e.g., \u201cUnderstanding of the safe and responsible use of robotics in the classroom\u201d).<\/p>\n\n\n\n

    This instrument was evaluated for face validity by two external consultants, one with expertise in the field of educational robotics and the other with expertise in science. The Cronbach\u2019s alpha values for internal consistency on the pre- (\u03b1 = .972) and post- (\u03b1 = 0.955) surveys indicated a good reliability (DeVellis, 2003).<\/p>\n\n\n\n

    Teaching Observations<\/em><\/h3>\n\n\n\n

    To gauge teacher-participants\u2019 teaching performance, this study utilized a modified version of the Assisting, Developing, and Evaluating Professional Teaching (ADEPT) teaching observation rubric (South Carolina Department of Education, 2018). We modified the ADEPT rubric to target indicators grouped in four categories: (a) Standards and Objectives, (b) Student Instruction, (c) Academic Engagement, and (d) Teacher Content Knowledge. These refined categories were designed to feature indicators pertinent to this study (e.g., the performance indicators of Problem Solving, Thinking: Types of Thinking, Teacher Content Knowledge: Connecting Concepts, and Activities and Materials) and yield additional insight and detail for evaluation purposes. The ADEPT rubric was selected because it was the state\u2019s instrument used to evaluate teachers, and it provided considerable fine grain data regarding teaching performance.<\/p>\n\n\n\n

    Observations occurred at two times during the project, once in the fall and again at the end of the year in the spring. Each observation took 30 minutes. Data were collected by the researchers in pairs and then combined to avoid representing only the subjectivities of a single researcher (as recommended by Barry et al., 1999; Saldana, 2015). We rated each indicator using the instrument\u2019s 4-point scale, where 4 represented the highest evaluation and 1 the lowest. Proficiency on a teaching performance indicator in the ADEPT rubric is a score of 3 on a 4-point scale. Interrater reliability was calculated for paired observation scores and yielded an agreement coefficient of .86.<\/p>\n\n\n\n

    Pre\/Post MAP Exams<\/em><\/h3>\n\n\n\n

    Student-participants (N<\/em> = 291) were evaluated by the growth of their mathematics scores on a standardized test, the Northwest Evaluation Association\u2019s (NWEA) Measure of Academic Progress (MAP) exam. The MAP exam is produced by the NWEA, a non-profit testing association. The MAP exam is a dynamic computer-based standardized test which evaluates students\u2019 growth in the areas of reading, mathematics, and science. Due to a testing administration error, the schools provided the researchers with incomplete science data that could not be used. Therefore, the researchers focused the analyses on mathematics scores only. Approximately 70% of the standardized test items in the MAP exam are mathematics questions, and these are out of 300 possible points. The MAP exam is designed to track progress across multiple grade levels, and on average, students score from 140 to 190 in third grade and between 240 to 300 by high school (NWEA, 2019). Student-participants in classrooms taught by the teacher-participants were assessed twice using the MAP exam, once at the beginning of the academic year and once at the end.<\/p>\n\n\n\n

    Results<\/h2>\n\n\n\n

    Teacher-Participants<\/h3>\n\n\n\n

    Pre\/Post Robotics Teaching Competency Surveys<\/em><\/h3>\n\n\n\n

    Teacher-participants completed robotics teaching competency self-assessment surveys in which they evaluated their own knowledge and teaching application of robotics both before and after the series of PD. Before the PD activities began, all the teacher-participants reported that they had almost no knowledge or competency related to robotics. The second administration of the survey was given a year later after the teacher-participants had completed all the robotics PD sessions.<\/p>\n\n\n\n

    Teacher-participants\u2019 mean competency survey scores were compared. A Shapiro-Wilk normality test (p<\/em> > .05) was used to determine the normality of the difference between the presurvey and postsurvey data. The Shapiro-Wilk test is the most accurate method for evaluating the normality of data for sample sizes less than 50 (Liang et al., 2019), and it was necessary to assure that the data met all assumptions for the statistical analysis applied (Field, 2009; Stehlik-Barry & Babinec, 2017). The results (p<\/em> = .056) indicated that the data were normally distributed. Thus, the parametric paired samples t<\/em>-test was used. Results of a paired samples t<\/em>-test (p<\/em> < .05) indicated that teacher-participants\u2019 robotics teaching competency increased significantly from the presurvey (M<\/em> = 1.55, SD<\/em> = .52) to the postsurvey (M<\/em> = 2.45, SD<\/em> = .54), t<\/em>(10) = 4.33, p<\/em> = .001, Cohen\u2019s d<\/em> = 1.31. As shown in Table 4, the effect size (d<\/em> = 1.31) was found to exceed Cohen\u2019s (1988) convention for a large effect (d<\/em> = .80).<\/p>\n\n\n\n

    Table 4<\/strong>
    Paired Samples t-Test \u2013 Robotics Teaching Competency Surveys<\/em><\/p>\n\n\n\n\n\n\n\t\n\n\t\n\t\n\t
    Presurvey<\/th>Postsurvey<\/th>t<\/em><\/th>df<\/em><\/th>p<\/em><\/th>d<\/em><\/th>\n<\/tr>\n<\/thead>\n
    M<\/em><\/td>SD<\/em><\/td>M<\/em><\/td>SD<\/em><\/td><\/td><\/td><\/td><\/td>\n<\/tr>\n
    1.55<\/td>.52<\/td>2.45<\/td>.54<\/td>4.33<\/td>10<\/td>.001*<\/td>1.31<\/td>\n<\/tr>\n
    Note.<\/em> Out of 4-point scale.
    \n* Indicates the differences between pretest and posttest is significant p<\/em> < .05.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n\n\n\n

    Teaching Observations<\/em><\/h3>\n\n\n\n

    Teacher-participants were formally observed by the researchers while conducting robotics-integrated lessons. The teacher-participants\u2019 observation scores were evaluated at the total, category, and performance indicator levels. First, descriptive statistics were tabulated with the observation scores from each researcher paired and averaged for each utilized ADEPT item, category, and the total (Table 5).<\/p>\n\n\n\n

    Table 5<\/strong>
    Descriptive Statistics \u2013 Teacher Observation Scores<\/em><\/p>\n\n\n\n\n\n\n\t\n\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t
    ADEPT Indicators<\/strong><\/th>Fall Observation<\/strong><\/th>Spring Observation<\/strong><\/th>Gain<\/strong><\/th>\n<\/tr>\n<\/thead>\n
    \u00a0<\/td>M<\/em><\/td>SD<\/em><\/td>M<\/em><\/td>SD<\/em><\/td>\u00a0<\/td>\n<\/tr>\n
    Communicating Learning Objectives and Standards <\/strong><\/td>2.62<\/td>0.74<\/td>3.39<\/td>0.89<\/td>0.77<\/td>\n<\/tr>\n
    Aligning Subobjectives<\/td>3.05<\/td>0.50<\/td>3.62<\/td>0.43<\/td>0.57<\/td>\n<\/tr>\n
    Connecting Learning Objectives<\/td>3.09<\/td>0.49<\/td>3.82<\/td>0.34<\/td>0.73<\/td>\n<\/tr>\n
    Student Performance Expectations<\/td>3.05<\/td>0.84<\/td>3.58<\/td>0.34<\/td>0.53<\/td>\n<\/tr>\n
    Student Mastery<\/td>3.58<\/td>0.54<\/td>2.85<\/td>0.68<\/td>-0.73<\/td>\n<\/tr>\n
    Standards and Objectives Category<\/strong><\/td>3.11<\/td>0.56<\/td>3.39<\/td>0.70<\/td>0.28<\/td>\n<\/tr>\n
    Motivating Students: Engaging Students <\/td>3.09<\/td>0.74<\/td>3.50<\/td>0.34<\/td>0.41<\/td>\n<\/tr>\n
    Motivating Students: Learning Experiences<\/td>3.02<\/td>0.81<\/td>3.50<\/td>0.71<\/td>0.48<\/td>\n<\/tr>\n
    Presenting Instructional Content<\/td>2.91<\/td>0.69<\/td>3.29<\/td>1.00<\/td>0.38<\/td>\n<\/tr>\n
    Lesson Structure and Pacing: Structure<\/td>3.43<\/td>0.37<\/td>2.86<\/td>1.00<\/td>-0.57<\/td>\n<\/tr>\n
    Lesson Structure and Pacing: Pacing<\/td>2.77<\/td>0.75<\/td>3.02<\/td>0.69<\/td>0.25<\/td>\n<\/tr>\n
    Lesson Structure and Pacing: Routines, Transitions<\/td>3.27<\/td>0.85<\/td>2.97<\/td>0.81<\/td>-0.30<\/td>\n<\/tr>\n
    Activities and Materials
    \n <\/td>    		3.37<\/td>0.57<\/td>3.58<\/td>0.65<\/td>0.21<\/td>\n<\/tr>\n
    Instructional Plans: Activities, Materials, Assessments<\/td>3.05<\/td>0.50<\/td>2.90<\/td>0.22<\/td>-0.15<\/td>\n<\/tr>\n
    Student Work: Assignments<\/td>2.88<\/td>0.60<\/td>3.58<\/td>0.70<\/td>0.70<\/td>\n<\/tr>\n
    Student Work: Drawing and Supporting Conclusions<\/td>2.80<\/td>0.68<\/td>3.10<\/td>0.78<\/td>0.30<\/td>\n<\/tr>\n
    Student Work: Connecting Learning<\/td>2.69<\/td>0.88<\/td>3.31<\/td>0.67<\/td>0.62<\/td>\n<\/tr>\n
    Student Instruction Category<\/strong><\/td>3.07<\/td>0.49<\/td>3.34<\/td>0.68<\/td>0.27<\/td>\n<\/tr>\n
    Questioning<\/td>2.83<\/td>0.15<\/td>2.83<\/td>0.76<\/td>0.00<\/td>\n<\/tr>\n
    Academic Feedback: Oral and Written Feedback<\/td>3.32<\/td>0.09<\/td>3.03<\/td>0.67<\/td>-0.29<\/td>\n<\/tr>\n
    Academic Feedback: Frequency of Feedback<\/td>3.09<\/td>0.07<\/td>3.33<\/td>0.86<\/td>0.24<\/td>\n<\/tr>\n
    Academic Feedback: Monitoring Student Progress<\/td>3.22<\/td>0.10<\/td>3.58<\/td>0.57<\/td>0.36<\/td>\n<\/tr>\n
    Academic Feedback: Student Feedback<\/td>3.09<\/td>0.18<\/td>3.25<\/td>0.47<\/td>0.16<\/td>\n<\/tr>\n
    Thinking: Types of Thinking<\/td>3.14<\/td>0.24<\/td>3.40<\/td>0.70<\/td>0.26<\/td>\n<\/tr>\n
    Problem-Solving<\/td>3.27<\/td>0.24<\/td>3.67<\/td>0.67<\/td>0.40<\/td>\n<\/tr>\n
    Academic Engagement Category<\/strong><\/td>3.14<\/td>0.19<\/td>3.26<\/td>0.61<\/td>0.12<\/td>\n<\/tr>\n
    Teacher Content Knowledge: Overall <\/td>2.98<\/td>0.66<\/td>3.11<\/td>0.48<\/td>0.13<\/td>\n<\/tr>\n
    Teacher Content Knowledge: Instructional Strategies<\/td>2.73<\/td>0.74<\/td>3.06<\/td>0.86<\/td>0.33<\/td>\n<\/tr>\n
    Teacher Content Knowledge: Connecting Concepts<\/td>2.68<\/td>0.60<\/td>3.08<\/td>0.71<\/td>0.40<\/td>\n<\/tr>\n
    Teacher Content Knowledge Category<\/td>2.82<\/td>0.62<\/td>3.08<\/td>0.48<\/td>0.26<\/td>\n<\/tr>\n
    Total Score<\/strong><\/td>3.04<\/td>0.49<\/td>3.26<\/td>0.57<\/td>0.22<\/td>\n<\/tr>\n
    Note.<\/em> Out of 4-point scale. Gains are the differences between fall and spring means. <\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n\n\n\n

    Next, the means between the fall and spring observations were compared to evaluate teaching performance changes. The researchers ran Shapiro-Wilk tests to determine if the data complied with the assumptions for parametric statistical analysis. The data were normally distributed (p<\/em> > .05) for each of the categories as well as the total. Thus, parametric paired samples t<\/em>-tests were used. After testing the assumptions, paired samples t<\/em>-tests showed that observation scores increased from the fall (M<\/em> = 3.04, SD<\/em> = .49) to the spring (M<\/em> = 3.26, SD<\/em> = .57), t<\/em>(10) = 1.02, p<\/em> = .333. The results of the t<\/em>-tests showed no statistically significant differences between the observations, suggesting only slight gains in teaching performance.<\/p>\n\n\n\n

    Student-Participants<\/h3>\n\n\n\n

    Pre\/Post MAP Exams<\/em><\/h3>\n\n\n\n

    Natural gains in mathematics achievement were expected among students as they learned and progressed throughout the year. Therefore, two control samples were used for comparison to determine if additional growth could be attributed to the robotics PD over natural student gains. The NWEA (2019) publishes anonymous assessment data from over 10 million students from 49 states with which researchers can create demographically aligned and nationally normed control groups. We used this openly published data to create the control groups for this study. <\/em><\/strong><\/p>\n\n\n\n

    To contextualize student-participants\u2019 growth between the first and second exams in this study, means were used to compare student-participants\u2019 scores with two sets of data: (a) national norms and (b) a demographically matched control sample from current published NWEA MAP datasets. The national norm data were used to contextualize the student-participants\u2019 scores against the mathematics growth of the rest of the country among those grade levels. The control sample data were used to precisely contextualize the student-participants\u2019 mathematics growth against students\u2019 scores from similarly disadvantaged schools with demographically matched backgrounds. The control sample was comprised of randomly selected students who matched this study\u2019s student-participants demographically (grade level, gender, and ethnicity) from similarly disadvantaged schools. In Table 6, the student-participants in this study are referred to as Robotics, the national norm sample is referred to as National Norm, and the demographically matched control sample is referred to as Control Sample.<\/p>\n\n\n\n