Fegely, A., Winslow, J., Lee, C-Y. & Rubbo, L. J. (2021). The effects of robotics professional development on science and mathematics teaching performance and student achievement in underserved middle schools. Contemporary Issues in Technology and Teacher Education, 21(4). https://citejournal.org/volume-21/issue-4-21/mathematics/the-effects-of-robotics-professional-development-on-science-and-mathematics-teaching-performance-and-student-achievement-in-underserved-middle-schools

The Effects of Robotics Professional Development on Science and Mathematics Teaching Performance and Student Achievement in Underserved Middle Schools

by Alex Fegely, Coastal Carolina University; Joseph Winslow, Coastal Carolina University; Cheng-Yuan Lee, Coastal Carolina University; & Louis J. Rubbo, College of Southern Nevada

Abstract

This article reports findings from an exploratory study investigating the effects of robotics professional development sessions in underserved middle schools in the southeastern United States. Eleven middle-level science and mathematics teachers from a high-needs school district received year-long training in robotics technology and instructional integration. Teacher-participants were evaluated on their problem-solving abilities, critical thinking strategies, robotics knowledge, content knowledge, and instructional design through teaching observations and pre/post robotics teaching competency surveys. Student performance was measured by comparing student-participants’ mathematics score growth on a standardized test against nationally normed control group samples. Results from teacher-participants (N = 11) indicated that they significantly improved their robotics teaching competencies and demonstrated measurable gains in numerous teaching performance indicators. Results from student-participants (N = 291) revealed they experienced mathematics growth at a higher percentage than their control group counterparts at each grade level. Sixth graders improved at a year change rate higher than the control sample to match the national norm mean on the posttest. Seventh graders experienced a year change rate and posttest mean far exceeding the control group that approached the national norm. Eighth graders improved at a year change rate that exceeded the control group but was beneath the national norm.

Proponents of integrating technology into science and mathematics curricula argue that it aids students in acquiring valuable disciplinary skills, such as logical analysis and critical thinking, and prepares them for real-world problem solving using modern tools (Castledine & Chalmers, 2011). Federal legislation in the United States, such as the America COMPETES Reauthorization Act of 2010, has acknowledged the importance of developing these types of skills in schools.

In 2018, the Committee on STEM Education of the National Science and Technology Council published a report charting a 5-year strategic plan for science, technology, engineering, and mathematics (STEM) education. In response to this plan, in-service teacher education has recently focused on initiatives that strengthen STEM subjects’ cross-cutting curricula (see also K-12 Computer Science Framework Steering Committee, 2016), such as robotics instruction, which has been identified as an effective integrative approach to teaching STEM principles (Scaradozzi et al., 2019). This study investigated the effects of robotics professional development (PD) on middle level science and mathematics teachers’ (N = 11) robotics instruction and students’ (N = 291) mathematics achievement.

Relevant Literature

Constructionism

Papert’s early findings on computer programming instruction (1980) and constructionist learning (1993) have contributed substantially to the evolution of robotics education, emphasizing the combination of student-centered activities with mechanical tools to solve practical problems. For example, research by Mikropoulos and Bellou (2013) indicated that constructionism impacted robotics education significantly, as the majority of educational robotics studies in their sample utilized some type of constructionist approach.

Constructionism is both a theory of learning and an instructional strategy (Ardito et al., 2014). Constructionism theorizes that knowledge is not simply transferred from the instructor to the student (Papert, 1980, 1993). Instead, learning is brought about through the construction, deconstruction, and reconstruction of students’ understanding, based on experiences fostered by physical construction of learning artifacts (Kafai & Resnick, 1996; Mikropoulous & Bellou, 2013; Resnick & Silverman, 2005). Constructionism includes two entwined types of construction: the construction of products and the construction of meaning (Kafai & Resnick, 1996). The construction of the concrete objects aids in the construction of mental models (Mikropoulous & Bellou, 2013).

In 1998, the LEGO® company released their constructible robotics kits – LEGO MINDSTORMS® – named after Papert’s (1980) seminal work on constructionism entitled Mindstorms: Children, Computers and Powerful Ideas (Chambers & Carbonaro, 2003). These LEGO MINDSTORMS kits were developed by some of Papert’s protégés as an archetypal constructionist learning tool (Ardito et al., 2014). Since their release, LEGO’s MINDSTORMS kits and curricula have advanced to the forefront of robotics education (Eguchi, 2013, Martin et al., 2000) as well as student robotics competitions including FIRST LEGO League and World Robot Olympiad (Zhang & Wan, 2020).

Research by Yolcu and Demirer (2017) analyzed studies about robotics education and found that over 66% of such studies utilized buildable LEGO robotics kits (over 40% used LEGO MINDSTORMS, in particular) and over 90% used LEGO or similar buildable robotics kits. To summarize, constructionism is heavily associated with educational robotics due to the constructable and customizable nature of educational robotics kits, like LEGO MINDSTORMS.

Teachers and Robotics Professional Development

Researchers have noted that few studies have examined the impact of robotics PD on teachers (Kim et al., 2015; Yuksel et al., 2020). Studies that focused on training teachers in STEM concepts with educational robotics have had various aims and findings (Guven & Cakir, 2020; Kay et al., 2014; Kopcha et al., 2017; Scaradozzi et al., 2019; Sullivan & Moriarty, 2009). For example, Kopcha et al. (2017) and Scaradozzi et al. (2019) found that educational robotics STEM PD activities were effective in teaching integrative STEM principles to teachers.

Similarly, studies have noted statistically significant programming and robotics knowledge increases among in-service teacher participants (Kay et al., 2014; Scaradozzi et al., 2019; Sullivan & Moriarty, 2009). Researchers have found that in-service teachers’ confidence with robotics increased significantly because of workshops, as well (Kay et al., 2014; Scaradozzi et al., 2019; Sullivan & Moriarty, 2009).

Sullivan and Moriarty (2009) suggested that the perceptions and practices among teachers learning about robotics and integrating robotics concepts into instruction may change through robotics experiences. Research by Guven and Cakir (2020) and Kopcha et al. (2017) found that the teachers integrated or intended to integrate robotics into their future instruction, which suggested that robotics were an efficient way to teach STEM concepts to teachers and influence their perceptions and practices. These studies exemplify the different aims and findings of literature exploring teachers and robotics PD.

Students and Robotics

The impact of robotics integration in science and mathematics instruction has been recently investigated for students in numerous grade levels. Previous inquiry has examined the integration of robotics kits as constructionist tools for students to learn STEM content through hands-on programming tasks at the elementary and middle levels (Bers, 2010; Fessakis et al., 2013; Koumoullos, 2013; Mikropoulos & Bellou, 2013). Researchers have noted that these kits can be effective for younger learners because they integrate block-based programming languages that diminish the tedium of coding text line-by-line and the associated syntax errors that novice programmers often make (Falloon, 2016; Kim et al., 2018).

Studies have indicated that robotics activities develop student problem-solving abilities (Bers et al., 2014; Datteri et al., 2013) and increase meaningful learning (Kaloti-Hallak et al., 2019). Beyond achievement gains, Yesharim and Ben-Ari (2018) noted that students learning computer science constructs with robotics demonstrated high motivation to succeed. Other researchers have detected positive effects of robotics on students’ STEM self-efficacy (Hall-Lay, 2018; Leonard et al., 2016). Williams et al. (2012) studied the impact of robotics on science and mathematics understanding of elementary, middle, and high schoolers. From the pretest to the posttest, students’ mathematics understanding increased 25% and their science understanding increased 47%.

While studies specifically evaluating robotics in middle school student populations are scarce (Casler-Failing, 2018), such studies have shown mathematics gains among students (Ardito et al., 2014; Casler-Failing, 2017; Castledine & Chalmers, 2011). A study by Ardito et al. (2014) investigated the impact of robotics on sixth graders’ mathematics achievement. The study was conducted in the mathematics and science classrooms and utilized programming problem-solving activities and challenges linked to algebra, measurement, and probability. The results of the study indicated that students’ achievement on a state standardized mathematics test in algebra, measurement, and probability improved, but not to statistically significant levels.

Further, a study by Castledine and Chalmers (2011) examined the correlation between sixth-grade mathematics students’ problem-solving decisions related to speed, distance, time, and angles in robotics programming races and mazes and their abilities to translate those strategies to authentic mathematics problems. Students exhibited growth in their problem-solving skills in mathematics because of robotics learning activities.

At the seventh-grade level, mathematics students who were learning graphing, measurement, scaling, speed, distance, and time through robotics activities in a study by Casler-Failing (2017) showed improvement in their understanding of proportional reasoning skills, especially among low-performing students. Eighth grade science students in research by Williams et al. (2012) showed learning growth in their understanding of the mathematics and science concepts of force, velocity, and acceleration after 90 minutes of hands-on robotics activities. These studies investigated the impacts of robotics integration on students in science and mathematics at numerous grade levels.

The Present Study

The research reported here adds to the limited literature on the impact of robotics PD on teachers (Kim et al., 2015; Yuksel et al., 2020), as well as the limited literature specifically analyzing how educational robotics impact middle school students’ mathematics achievement (Casler-Failing, 2018). Three novel aspects of this study distinguish it from previous research: (a) the context, (b) the length of the treatment, and (c) the use of nationally normed and demographically matched control samples.

First, this study focused on a novel context for educational robotics research: underserved middle schools. Second, this study did not simply focus on the short-term impacts of educational robotics. In this study, teacher-participants took part in over 75 contact hours of extensive PD, and teacher-participants and student-participants were evaluated over the course of a year. Finally, the use of both nationally normed and demographically matched control samples has provided two sets of control groups with which to compare the student participants’ results. The national norm data were used to contrast the student-participants’ mathematics growth against the rest of the country among sixth, seventh, and eighth graders. The control sample data were used to precisely contextualize the student-participants’ mathematics growth against students’ growth from similarly disadvantaged schools with demographically matched backgrounds.

The research questions in this study were as follows:

  1. What are the effects of robotics professional development sessions on middle school science and mathematics teachers’ teaching performance?
  2. How do robotics professional development sessions for middle school science and mathematics teachers impact students’ mathematics achievement?

Methodology

The PD sessions occurred across the span of an academic year, bookended by week-long summer PD sessions. In turn, students were taught by the teachers who participated in the PD sessions and used the teachers’ robotics kits in robotics-centric science and mathematics lessons. Using quantitative methods, the researchers evaluated teaching performance and student mathematics growth. Specific methodological details will be explained in the paragraphs below.

Setting and Participants

For this grant-funded project, the researchers partnered with a regional public school district identified as high-needs in the southeastern United States for its historically low socioeconomic status and low student achievement. The grant’s call focused on increasing academic achievement in the state by improving teacher quality. The researchers identified this district based on the project’s potential to have a more meaningful impact supporting a high-needs school district, as opposed to others in the region. At the time of the study, the district served 5,200 students; 90% of students lived in poverty, and it had a 63% senior graduation rate. Both teachers and students from this school district served as this study’s participants. Informed consent was obtained from the teacher-participants and student-participant consent was managed by the individual schools.

Teacher-Participants

After a district-wide survey of interest in robotics PD, administrators selected participants who (a) taught middle school science or mathematics (other subject area teachers who expressed interest were excluded) and (b) were willing to participate in the year-long study. All the teachers who had expressed interest and met these criteria were selected.

In total, the district selected 15 science and mathematics teachers spread among four of the district’s middle schools. There was attrition of teacher-participants over the year-long duration of the study. Two of the 15 teacher-participants were lost due to career advancement, one dropped out to focus on 1st-year teaching responsibilities, and one could not attend the second summer of PD experiences due to a family emergency. Of the remaining 11 participants (four male and seven female teachers), each taught sixth (three), seventh (two), or eighth (six) grade science or mathematics.

Seven teacher-participants taught mathematics, and four taught science. Four teachers identified themselves as Black, four as White, and three as Asian. All participants received a robotics kit, a laptop, a stipend, supplementary sensors, robotics classroom integration books, and three graduate course credits for their participation that could be applied toward continuing education or a degree program.

Student-Participants

Student-participants also took part in this study. All the students in science and mathematics classes taught by the teachers participating in this study were used as a convenience sample. The student-participants represented 13 classes: five science and eight mathematics. The sample consisted of 171 female and 120 male student-participants. Of the 291 student-participants, 207 identified as Black, 74 identified as White, five identified as Hispanic, and five identified as Asian. More detailed demographic data is shared in Table 1.

Table 1
Student-Participant Demographic Information

GradeMaleFemaleTotal
AsianBlackHispanicWhiteAsianBlackHispanicWhite
602201322711782
70140202111048
806216361226161
Total0981215109453291

 

Trainers and Coaches

Two trainers were selected to deliver the PD content to teacher-participants. Both were full-time university faculty members credentialed in robotics instruction, one from the college of science and the other from the college of education. In addition to these two trainers, the researchers also hired three experienced robotics coaches from a different school district to provide scaffolding, instructional design, and PD support to the participating teachers. Both the trainers and the coaches received a stipend as compensation for their services throughout the project.

Research Design

Teacher-participants began the year-long series of PD sessions with a 1-week (35 hours) campus-based summer workshop that integrated formal robotics technology and pedagogy lessons. Constructionism (Papert, 1993) served as the theoretical framework for the PD curriculum, based on its alignment in the literature to educational robotics. The PD sessions were designed with a constructionist framework, as teacher-participants constructed their robots to adapt to different problem scenarios utilizing mathematics and science knowledge.

This curriculum incorporated the constructionist facets of knowledge construction through physical construction as teacher-participants built and customized their robots to solve problems, as well as a collaborative environment (Papert, 1980, 1993). Teacher-participants could then teach with these same constructionist practices in their own classrooms, facilitating learning through activities that required their students to build and customize their robots to solve authentic problems, such as mazes, in a collaborative environment. The PD curriculum was reviewed for face validity before implementation by four experts: three with expertise in robotics education and one with expertise in education. These lessons were led by the two trainers and included independent practice activities and challenges.

Individual support of the teacher-participants was facilitated by the three coaches. LEGO MINDSTORMS EV3 robotics kits were utilized by teacher-participants for all instructional activities. The LEGO MINDSTORMS EV3 kits included a programmable control unit, motors, sensors, building blocks, gears, and other mechanical pieces. LEGO MINDSTORMS EV3 kits were selected due to their developmental appropriateness for middle school students (Martin et al., 2000), the population taught by the teacher-participants.

The first campus-based summer week-long PD series focused on related science, mathematics, and robotics principles, specifically odometry, dead reckoning, sensors, flow control, data wires, gears, and problem-solving. The instruction included a blend of lecture, demonstration, and discussion, followed by hands-on individual or team activities that included relevant programming challenges. Each day’s lesson topic, its associated science and mathematics topics, and challenges are outlined in Table 2.

Table 2
First Campus-Based Summer Workshop Activities and Challenges

LessonsScience and Mathematics TopicsRoboMaze Challenges
Dead ReckoningOdometry; Calculating wheel circumference and distance per rotation; Dead reckoning; Debugging; PseudocodeNavigate with dead reckoning; Navigate with dead reckoning (black diamond)
Flow ControlRotations and distance; Programming loop, switch, and wait; Touch sensory inputNavigate with touch sensor
SensorsSound waves; Sonar; Programming switch; Light intensity; Light reflectionNavigate with ultrasonic sensor; Navigate with color sensor
Data Wires and GearsProgramming decision making; Gears; Gear ratios; Transmitting data; Programming loops; Positive and negative integers; Calculating time and speedNavigate with all sensors with obstacle included
The ChallengeCumulative science and mathematics conceptsNavigate with obstacle included for time (black diamond)

The programming challenges during these sessions focused on applying the targeted science and mathematics principles to solve problems in the RoboMaze. The RoboMaze required participants to navigate their robots by applying science and mathematics concepts, such as calculating distance, applying sonar, and engineering gearing ratios. In addition, problem-solving in the RoboMaze required participants to write and customize code written in a block-based programming language that controlled the robots.

Multiple RoboMazes were constructed from 4×8 foot melamine sheets and 2 ft x 4 ft boards to facilitate efficient access by all teacher-participants. As depicted in Figure 1, the RoboMaze could be navigated from top left to bottom right for a moderate challenge (Start and Finish for the moderate challenge are denoted on the schematic), or from bottom left to top right for a more difficult challenge with more turns (marked by the black diamonds). The RoboMaze required teacher-participants to utilize the programming, mathematics, and science knowledge they had built in each lesson to successfully navigate their robot through the maze.

Figure 1
The RoboMaze Path

The path from the noted Start and Finish locations requires fewer turns and is, thus, a lower difficulty than the black diamond path between the opposite corners.

During the ensuing fall and spring semesters of the academic year, teacher-participants were observed utilizing robotics in their classrooms by the researchers and received additional training and evaluation. Teacher-participants were required to attend two in-person robotics workshops with the trainers, participate in two live webinars for additional training, and attend a state educational technology conference that showcased numerous robotics sessions.

During the face-to-face workshops and live webinars, the teacher-participants were introduced to new programming concepts, sensors, challenges, club resources, and in-class robotics integration strategies. New robotics integration books and sensors, such as the temperature and infrared sensor/beacon, were distributed during the fall and spring workshops to expand teacher-participants’ integration of robotics in the classroom. In addition, teacher-participants shared their experiences of teaching with robotics among their teacher-participant peers both in-person in the workshops and through a social media group created for the teacher-participants.

The culminating challenge during a workshop in the spring semester of the academic year was the LEGO MINDSTORMS EV3 Animal Allies Challenge, depicted in Figure 2. In this challenge, teacher-participants were given various tasks to program their robots to complete while navigating an obstacle course. Teacher-participants were also assigned homework throughout the project tenure, such as lesson plans, implementation videos, and critical reflections. In-class observations took place during the fall and spring semesters during the academic year.

Figure 2
Teacher-Participants Discuss Problem-Solving in the Animal Allies Challenge During a Spring Workshop

The following summer, a final 1-week (35 hours) series of PD was conducted on campus using the same integrated model as the first summer. This summer workshop combined advanced robotics technology training with advanced instructional design training. The topics of instruction in the second summer workshop focused on training teachers to teach engineering design to their students. The engineering design process of planning, design, implementation, and improvement was taught to teacher-participants to integrate into science and mathematics. Engineering design was chosen as the next step in the curriculum to contextualize robotics instruction in real-world problem-solving. The lessons in the second summer workshop utilized more advanced problem-solving scenarios and various obstacle courses. Best practices for using robotics in the classroom were also analyzed. The culminating challenge for the second summer workshop was the LEGO MINDSTORMS Education EV3 Space Challenge. Shown in Figure 3, this challenge contained various tasks and obstacles for the teacher-participants to solve.

Figure 3
A Teacher-Participant Brainstorms a Solution to a Problem in the LEGO MINDSTORMS EV3 Space Challenge During the Second Summer’s Week of PD

Data Sources  

Quantitative methods were used for this study. Quantitative data were gathered through multiple instruments, which included pre/post teacher robotics teaching competency surveys, teacher lesson observations, and pre/post MAP exams for students. As outlined in Table 3, these data sources were used to answer the two research questions.

Table 3
Research Questions, Data Sources, and Data Analysis Method Alignment

Research QuestionsData SourcesData Analysis
RQ 1: What are the effects of robotics professional development sessions on middle school science and mathematics teachers’ teaching performancePre/post robotics teaching competency surveys
Teaching observations
Descriptive statistics
Paired samples t-tests
RQ 2: How do robotics professional development sessions for middle school science and mathematics teachers impact students’ mathematics achievement?Pre/post MAP examsDescriptive statistics
Paired samples t-tests

Pre/Post Robotics Teaching Competency Surveys

Teacher-participants self-assessed their robotics’ capabilities with 20-statement pre/post robotics teaching competency surveys. The first survey was given before the series of PD began, and the second survey was given after all PD sessions had been completed a year later. The teacher-participants assessed themselves on five categories, consisting of four statements each. This instrument used a 4-point Likert scale, with 1 being the lowest and 4 being the highest level of competency. As recommended by Cronbach (1950) and Nunnally (1967), a 4-point forced choice Likert scale was used to prevent participants from giving a response set of neutral answers. These five categories of statements were (a) hands-on robotics project curriculum planning (e.g., “Knowledge of how to integrate robotics into my curriculum”), (b) robotics and problem-solving skills (e.g., “Use of robotics technology to facilitate higher order and complex thinking skills”), (c) robotics and science inquiry (e.g., “Use of the science inquiry process to debug programs”), (d) robotics and design skills (e.g., “Creating and building stable structures with LEGO or other materials”), and (e) robotics and philosophical issues (e.g., “Understanding of the safe and responsible use of robotics in the classroom”).

This instrument was evaluated for face validity by two external consultants, one with expertise in the field of educational robotics and the other with expertise in science. The Cronbach’s alpha values for internal consistency on the pre- (α = .972) and post- (α = 0.955) surveys indicated a good reliability (DeVellis, 2003).

Teaching Observations

To gauge teacher-participants’ teaching performance, this study utilized a modified version of the Assisting, Developing, and Evaluating Professional Teaching (ADEPT) teaching observation rubric (South Carolina Department of Education, 2018). We modified the ADEPT rubric to target indicators grouped in four categories: (a) Standards and Objectives, (b) Student Instruction, (c) Academic Engagement, and (d) Teacher Content Knowledge. These refined categories were designed to feature indicators pertinent to this study (e.g., the performance indicators of Problem Solving, Thinking: Types of Thinking, Teacher Content Knowledge: Connecting Concepts, and Activities and Materials) and yield additional insight and detail for evaluation purposes. The ADEPT rubric was selected because it was the state’s instrument used to evaluate teachers, and it provided considerable fine grain data regarding teaching performance.

Observations occurred at two times during the project, once in the fall and again at the end of the year in the spring. Each observation took 30 minutes. Data were collected by the researchers in pairs and then combined to avoid representing only the subjectivities of a single researcher (as recommended by Barry et al., 1999; Saldana, 2015). We rated each indicator using the instrument’s 4-point scale, where 4 represented the highest evaluation and 1 the lowest. Proficiency on a teaching performance indicator in the ADEPT rubric is a score of 3 on a 4-point scale. Interrater reliability was calculated for paired observation scores and yielded an agreement coefficient of .86.

Pre/Post MAP Exams

Student-participants (N = 291) were evaluated by the growth of their mathematics scores on a standardized test, the Northwest Evaluation Association’s (NWEA) Measure of Academic Progress (MAP) exam. The MAP exam is produced by the NWEA, a non-profit testing association. The MAP exam is a dynamic computer-based standardized test which evaluates students’ growth in the areas of reading, mathematics, and science. Due to a testing administration error, the schools provided the researchers with incomplete science data that could not be used. Therefore, the researchers focused the analyses on mathematics scores only. Approximately 70% of the standardized test items in the MAP exam are mathematics questions, and these are out of 300 possible points. The MAP exam is designed to track progress across multiple grade levels, and on average, students score from 140 to 190 in third grade and between 240 to 300 by high school (NWEA, 2019). Student-participants in classrooms taught by the teacher-participants were assessed twice using the MAP exam, once at the beginning of the academic year and once at the end.

Results

Teacher-Participants

Pre/Post Robotics Teaching Competency Surveys

Teacher-participants completed robotics teaching competency self-assessment surveys in which they evaluated their own knowledge and teaching application of robotics both before and after the series of PD. Before the PD activities began, all the teacher-participants reported that they had almost no knowledge or competency related to robotics. The second administration of the survey was given a year later after the teacher-participants had completed all the robotics PD sessions.

Teacher-participants’ mean competency survey scores were compared. A Shapiro-Wilk normality test (p > .05) was used to determine the normality of the difference between the presurvey and postsurvey data. The Shapiro-Wilk test is the most accurate method for evaluating the normality of data for sample sizes less than 50 (Liang et al., 2019), and it was necessary to assure that the data met all assumptions for the statistical analysis applied (Field, 2009; Stehlik-Barry & Babinec, 2017). The results (p = .056) indicated that the data were normally distributed. Thus, the parametric paired samples t-test was used. Results of a paired samples t-test (p < .05) indicated that teacher-participants’ robotics teaching competency increased significantly from the presurvey (M = 1.55, SD = .52) to the postsurvey (M = 2.45, SD = .54), t(10) = 4.33, p = .001, Cohen’s d = 1.31. As shown in Table 4, the effect size (d = 1.31) was found to exceed Cohen’s (1988) convention for a large effect (d = .80).

Table 4
Paired Samples t-Test – Robotics Teaching Competency Surveys

PresurveyPostsurveytdfpd
MSDMSD
1.55.522.45.544.3310.001*1.31
Note. Out of 4-point scale.
* Indicates the differences between pretest and posttest is significant p < .05.

Teaching Observations

Teacher-participants were formally observed by the researchers while conducting robotics-integrated lessons. The teacher-participants’ observation scores were evaluated at the total, category, and performance indicator levels. First, descriptive statistics were tabulated with the observation scores from each researcher paired and averaged for each utilized ADEPT item, category, and the total (Table 5).

Table 5
Descriptive Statistics – Teacher Observation Scores

ADEPT IndicatorsFall ObservationSpring ObservationGain
 MSDMSD 
Communicating Learning Objectives and Standards 2.620.743.390.890.77
Aligning Subobjectives3.050.503.620.430.57
Connecting Learning Objectives3.090.493.820.340.73
Student Performance Expectations3.050.843.580.340.53
Student Mastery3.580.542.850.68-0.73
Standards and Objectives Category3.110.563.390.700.28
Motivating Students: Engaging Students3.090.743.500.340.41
Motivating Students: Learning Experiences3.020.813.500.710.48
Presenting Instructional Content2.910.693.291.000.38
Lesson Structure and Pacing: Structure3.430.372.861.00-0.57
Lesson Structure and Pacing: Pacing2.770.753.020.690.25
Lesson Structure and Pacing: Routines, Transitions3.270.852.970.81-0.30
Activities and Materials
3.370.573.580.650.21
Instructional Plans: Activities, Materials, Assessments3.050.502.900.22-0.15
Student Work: Assignments2.880.603.580.700.70
Student Work: Drawing and Supporting Conclusions2.800.683.100.780.30
Student Work: Connecting Learning2.690.883.310.670.62
Student Instruction Category3.070.493.340.680.27
Questioning2.830.152.830.760.00
Academic Feedback: Oral and Written Feedback3.320.093.030.67-0.29
Academic Feedback: Frequency of Feedback3.090.073.330.860.24
Academic Feedback: Monitoring Student Progress3.220.103.580.570.36
Academic Feedback: Student Feedback3.090.183.250.470.16
Thinking: Types of Thinking3.140.243.400.700.26
Problem-Solving3.270.243.670.670.40
Academic Engagement Category3.140.193.260.610.12
Teacher Content Knowledge: Overall2.980.663.110.480.13
Teacher Content Knowledge: Instructional Strategies2.730.743.060.860.33
Teacher Content Knowledge: Connecting Concepts2.680.603.080.710.40
Teacher Content Knowledge Category2.820.623.080.480.26
Total Score3.040.493.260.570.22
Note. Out of 4-point scale. Gains are the differences between fall and spring means.

Next, the means between the fall and spring observations were compared to evaluate teaching performance changes. The researchers ran Shapiro-Wilk tests to determine if the data complied with the assumptions for parametric statistical analysis. The data were normally distributed (p > .05) for each of the categories as well as the total. Thus, parametric paired samples t-tests were used. After testing the assumptions, paired samples t-tests showed that observation scores increased from the fall (M = 3.04, SD = .49) to the spring (M = 3.26, SD = .57), t(10) = 1.02, p = .333. The results of the t-tests showed no statistically significant differences between the observations, suggesting only slight gains in teaching performance.

Student-Participants

Pre/Post MAP Exams

Natural gains in mathematics achievement were expected among students as they learned and progressed throughout the year. Therefore, two control samples were used for comparison to determine if additional growth could be attributed to the robotics PD over natural student gains. The NWEA (2019) publishes anonymous assessment data from over 10 million students from 49 states with which researchers can create demographically aligned and nationally normed control groups. We used this openly published data to create the control groups for this study.

To contextualize student-participants’ growth between the first and second exams in this study, means were used to compare student-participants’ scores with two sets of data: (a) national norms and (b) a demographically matched control sample from current published NWEA MAP datasets. The national norm data were used to contextualize the student-participants’ scores against the mathematics growth of the rest of the country among those grade levels. The control sample data were used to precisely contextualize the student-participants’ mathematics growth against students’ scores from similarly disadvantaged schools with demographically matched backgrounds. The control sample was comprised of randomly selected students who matched this study’s student-participants demographically (grade level, gender, and ethnicity) from similarly disadvantaged schools. In Table 6, the student-participants in this study are referred to as Robotics, the national norm sample is referred to as National Norm, and the demographically matched control sample is referred to as Control Sample.

Table 6
Descriptive Statistics – Student MAP Mathematics Scores

GroupGradePretestPosttestChange
MSDM SD
Robotics6220.21.31225.3[a] [b]1.272.33% [a]
Control Sample6212.72.38215.43.031.26%
National Norm6217.616.59225.316.713.54%
Robotics7217.12.92222.4 [a]2.852.44% [a]
Control Sample7211.43.50212.43.990.45%
National Norm7222.616.59228.617.722.70%
Robotics8220.81.23221.21.700.18% [a]
Control Sample8223.21.88223.21.920.00%
National Norm8226.317.85230.919.112.03%
Note. Out of 300 possible points.
Means end at the tenths place instead of hundredths to align to decimal format of MAP data.
[a] The Robotics group exceeded the Control group.
[b] The Robotics group matched the National Norm group.

To evaluate the impact of the teacher-participants’ robotics integration on their students’ achievement, the student-participants’ average pre and post MAP mathematics scores from each grade level (6, 7, and 8) were aligned into their respective pre and post Robotics, Control Sample, and National Norm groups. The three grade levels for each group of data represented 291 student-participants.

Shapiro-Wilk normality tests (p > .05) were used to determine the distribution of the data. The results for Robotics (p = .069), Control Sample (p = .716), and National Norm (p = .893) groups indicated that the data were normally distributed for each group. Therefore, paired samples t-tests (p < .05) were used to compare the mean pre and post MAP mathematics scores. The Robotics group’s MAP mathematics scores increased from the pretest (M = 219.37, SD = 1.99) to the posttest (M = 222.97, SD = 2.11), t(2) = 2.25, p = .154, but not to a statistically significant level. Correspondingly, the Control Sample group’s scores improved, but not to statistically significant levels.

As shown in Table 7, only the National Norm group’s improvement was statistically significant. The effect size was calculated for the National Norm due to its statistical significance, and the effect size (d = 3.93) was found to exceed Cohen’s (1988) convention for a large effect.

Table 7
Paired Samples t-Tests – MAP Mathematics Scores

GroupPretestPosttesttdfpd
MSDMSD
Robotics219.371.99222.972.112.252.154 
Control Sample215.776.47217.005.571.572.258 
National Norm222.174.37228.272.526.812.021*3.93
Note. Out of 300 possible points.
* Indicates the differences between pretest and posttest is significant p < .05.

Discussion

Teacher-Participants (Research Question 1)

Results of this study indicated that teachers experienced modest gains in their teaching performance. Quantitative results showed a statistically significant increase in teacher-participants’ robotics competencies from the presurvey to the postsurvey. This significant improvement suggests that the robotics PD sessions improved teacher-participants’ perceptions of their teaching abilities.

This significant improvement aligns with findings by Kay et al. (2014), Scaradozzi et al. (2019) and Sullivan and Moriarty (2009), which identified statistically significant robotics knowledge and confidence increases among teacher-participants because of educational robotics PD. However, the teacher-participants’ concluding robotics competencies in this study suggest that, while the teacher-participants felt they had some robotics competency, they did not feel fully comfortable with teaching others.

Furthermore, teaching observation scores indicated that teacher-participants experienced modest growth in their teaching performance. While this growth was not to statistically significant levels, teacher-participants demonstrated competence (>3.0 on a 4-point scale) on 21 of the 26 performance indicators by their final observation. More detailed insights into the teacher-participants teaching performance gains are evidenced in the areas with the highest marks on their final observations: connecting learning objectives, problem-solving, and creating high-quality student assignments.

Moreover, the highest gains from the first to last observation were in communicating STEM learning objectives and standards, connecting STEM learning objectives, and creating high-quality student assignments. We suppose that these gains resulted from teacher-participants becoming more comfortable with integrating robotics into their STEM curricula over time. Teacher-participants’ growth in teaching STEM concepts confirms findings in studies by Kopcha et al. (2017) and Scaradozzi et al. (2019) that found educational robotics STEM PD activities were effective in teaching integrative STEM principles to teachers.

Noticeable performance losses were measured over time for two performance indicators: lesson structure and student mastery. These performance losses may be tied to teacher-participants reaching outside both their own and their students’ comfort zones for their final teaching observations. In total, these data indicated modest teaching performance gains by teacher-participants evidenced by statistically significant robotics competency increases, as well as gains on 81% of observation performance indicators.

In summary, teacher-participants developed modest gains in their teaching performance. The data indicate that they demonstrated improvements in specific teaching practices, as well as statistically significant improvements in robotics teaching competency. The results of this study add to the literature that supports the use of robotics for developing in-service teachers’ teaching competencies (Kay et al., 2014; Kopcha et al., 2017; Scaradozzi et al., 2019; Sullivan & Moriarty, 2009).

Student-Participants (Research Question 2)

Data analyses revealed that the robotics group experienced mathematics growth at a year change rate exceeding their control group counterparts at each grade level (Table 6), although not to a statistically significant level (Table 7). The robotics group’s posttest mathematics scores in this study exceeded those of their control group counterparts in the sixth and seventh grades. This section will discuss the findings from the sixth, seventh, and eighth grade robotics groups and relate them to the existing literature.

In the sixth-grade comparison, the robotics group had a higher average posttest mathematics score than did the control group. In addition, the robotics group improved at a year change rate higher than the control sample to match the national norm mean on the posttest. These results mirror research by Ardito et al. (2014), which showed that sixth-grade students who learned mathematics through educational robotics in their science and mathematics classes exhibited growth on a state standardized mathematics test, but not to a statistically significant level.

These parallel sixth-grade results may be explained through research by Castledine and Chalmers (2011). In their research, Castledine and Chalmers (2011) utilized robotics races and mazes to teach students mathematics, much like the classroom activities experienced by the robotics group. Castledine and Chalmers found students’ problem-solving skills in the robotics challenges related to concepts of speed, distance, and angles translated to growth in solving mathematics problems.

The seventh-grade comparison showed that the robotics group experienced a year change rate and posttest mean far exceeding the control group that approached the national norm. Similarly, research by Casler-Failing (2017) found that educational robotics activities improved seventh-grade students’ mathematics understanding, especially among lower-performing students. The seventh-grade robotics group exhibited the highest percentage change rate out of all the robotics groups.

Eighth-grade data showed that the robotics group improved modestly at a year change rate that exceeded the control group but was beneath the national norm. These findings support research by Williams et al. (2012) that identified learning growth in eighth-grade science students’ understanding of mathematics concepts related to force, velocity, and acceleration after learning with educational robotics. Eighth-grade student performance gains were minimal when compared to the improved scores earned by sixth and seventh graders. In this case, triangulation of teacher-participant data from the teaching observations can be used to explain the low eighth-grade achievement data.

Teacher-participants showed improvement in 21 of the 26 teaching observation indicators. The negative trending by teacher-participants in the remaining five teaching observation indicators could explain the minimal growth by eighth graders. Data confirm that the teacher-participants who scored the lowest on the teaching observation instrument in these indicators were mostly eighth-grade teachers. Thus, this triangulation supports the supposition that eighth-grade students did not improve on the same level as the sixth and seventh-grade groups over the academic year due to weaker eighth-grade teacher instruction.

Altogether, these results are noteworthy because the robotics group experienced learning growth at a higher percentage over the course of the year than the control sample of demographically matched students from similarly disadvantaged schools. Not only did the robotics groups exhibit higher percentage growth than the control groups in all three grade levels, the sixth-grade robotics group matched the national norm average. These findings suggest that robotics PD sessions for middle school science and mathematics teachers can positively impact students’ mathematics achievement.

Implications

Specifically, this study provides perspective on educational robotics PD and ways it can impact science and mathematics teaching performance and mathematics achievement in underserved middle schools. Generally, the findings of this study support the potential of educational robotics PD to impact teaching and learning positively in the science and mathematics classrooms. Therefore, we recommend that (a) schools seek to build upon the PD curriculum outlined in this article, and (b) school districts and partner organizations (i.e., universities and after school programs like FIRST LEGO League) work together to expand their robotics PD initiatives at the middle school level.

Limitations

This study was limited by a few factors. First, it could be logically inferred that teacher-participants’ (N = 11) experiences in the robotics PD sessions positively impacted the standardized test performance of the student-participants (N = 291). However, we must temper such a conclusion, given the limitations of the small treatment group of the teacher-participants. Next, while this study utilized a modified ADEPT instrument to align with state observation protocols, several more specialized valid and reliable instruments have been designed for STEM-specific pedagogy, such as Marshall et al.’s (2010) EQUIP, and Piburn and Sawada’s (2001) Reformed Teaching Observation Protocol. Finally, a more specialized instrument could have been used in this study to evaluate students’ mathematics growth instead of the dynamic computer-based standardized test that included questions aligned to multiple subjects, as protracted student testing could result in assessment apathy (Thompson, 2008).

Future Research

Future inquiry into the burgeoning field of educational robotics builds on these findings in four distinct ways. First, future studies could increase the population size of in-service teachers participating in robotics PD for greater generalizability. Second, STEM-specific observational protocols could be used to evaluate STEM pedagogy. In addition, more specific robotics PD design variables could be examined, including ways STEM concepts and standards can be connected to instruction, ways to present robotics technology concepts, and ways to engage learners in robotics-integrated lessons. Finally, teacher and student performance data could be supplemented with additional qualitative data to aid in triangulation.

Author Note

This study was sponsored by the Improving Teacher Quality Higher Education Grant Program.

References

Ardito, G., Mosley, P., & Scollins, L. (2014). WE, ROBOT: Using robotics to promote collaborative and mathematics learning in a middle school classroom. Middle Grades Research Journal, 9(3), 73.

Barry, C. A., Britten, N. Barber, N., Bradley, C., & Stevensen, F. (1999). Using reflexivity to optimize teamwork in qualitative research. Qualitative Health Research, (1), 26-44. https://journals.sagepub.com/doi/abs/10.1177/104973299129121677

Bers, M. U. (2010). The TangibleK robotics program: Applied computational thinking for young children. Early Childhood Research & Practice, 12(2).

Bers, M. U., Flannery, L., Kazakoff, E. R., & Sullivan, A. (2014). Computational thinking and tinkering: Exploration of an early childhood robotics curriculum. Computers & Education, 72, 145-157.

Casler-Failing, S. L. (2017). The effects of integrating Lego robotics into a mathematics curriculum to promote the development of proportional reasoning. Proceedings of the Interdisciplinary STEM Teaching and Learning Conference (pp. 24-35). Statesboro, GA: Georgia Southern University.

Casler-Failing, S. L. (2018). Robotics and math: Using action research to study growth problems. The Canadian Journal of Action Research, 19(2), 4-25.

Castledine, A.-R., & Chalmers, C. (2011). LEGO robotics: An authentic problem solving tool? Design and Technology Education, 16, 19–27.

Chambers, J. M., & Carbonaro, M. (2003). Designing, developing, and implementing a course on LEGO robotics for technology teacher education. Journal of Technology and Teacher Education, 11(2), 209–242.

Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Nature (2nd ed., Vol. 506). Lawrence Erlbaum Associates. https://doi.org/10.4324/9780203771587

Committee on STEM Education of the National Science & Technology Council (2018). Charting a course for success: America’s strategy for STEM education. https://www.energy.gov/sites/default/files/2019/05/f62/STEM-Education-Strategic-Plan-2018.pdf

Cronbach, L. J. (1950). Further evidence on response sets and test design. Educational and Psychological Measurement, 10, 3-31.

Datteri, E., Zecca, L., Laudisa, F., & Castiglioni, M. (2013). Learning to explain: The role of educational robots in science education. Themes in Science & Technology Education, 6(1), 29–38.

DeVellis, R. F. (2003). Scale development: Theory and applications. Sage.

Eguchi, A. (2013). Educational robotics for promoting 21st century skills. Journal of Automation, Mobile Robotics & Intelligent Systems, 8(1), 1–42. https://doi.org/10.14313/JAMRIS

Falloon, G. (2016). An analysis of young students’ thinking when completing basic coding tasks using Scratch Jnr. on the iPad. Journal of Computer Assisted Learning, 32(6), 576–593.

Fessakis, G., Gouli, E., & Mavroudi, E. (2013). Problem solving by 5–6 years old kindergarten children in a computer programming environment: A case study. Computers & Education, 63, 87-97.

Field, A. (2009). Discovering statistics using SPSS. SAGE Publications.

Guven, G., & Cakir, N.K. (2020). Investigation of the opinions of teachers who received in-service training for Arduino-assisted robotic coding applications. Educational Policy Analysis and Strategic Research, 15(1), 253-274. doi: 10.29329/epasr.2020.236.14

Hall-Lay, S. (2018). Gender effects of robotics programs on STEM-related self-efficacy of high school students (Unpublished doctoral dissertation). Walden University, Minneapolis, MN.

K12 Computer Science Framework Steering Committee. (2016). K–12 computer science framework. http://www.k12cs.org.

Kafai, Y. B., & Resnick, M. (1996). Constructionism in practice: Designing, learning and thinking in a digital world. Lawrence Erlbaum Associates.

Kaloti-Hallak, F., Armoni, M., & Ben-Ari, M. (2019). The effect of robotics activities on learning the engineering design process. Informatics in Education 18(1), 105-129.

Kay, J. S., Moss, J. G., Engelman, S., & McKlin, T. (2014). Sneaking in through the back door: Introducing K-12 teachers to robot programming. SIGCSE 2014 – Proceedings of the 45th ACM Technical Symposium on Computer Science Education, 499–504. https://doi.org/10.1145/2538862.2538972

Kim, C., Kim, D., Yuan, J., Hill, R. B., Doshi, P., & Thai, C. N. (2015). Robotics to promote elementary education pre-service teachers’ STEM engagement, learning, and teaching. Computers and Education, 91, 14–31. https://doi.org/10.1016/j.compedu.2015.08.005

Kim, C., Yuan, J., Vasconcelos, L., Shin, M., & Hill, R. B. (2018). Debugging during block-based programming. Instructional Science, 46(5), 767–787.

Kopcha, T. J., McGregor, J., Shin, S., Qian, Y., Choi, J., Hill, R., Mativo, J., & Choi, I. (2017). Developing an integrative STEM curriculum for robotics education through educational design research. Journal of Formative Design in Learning, 1(1), 31–44. https://doi.org/10.1007/s41686-017-0005-1

Koumoullos, M. (2013). The academic differences between students involved in school-based robotics programs and students not involved in school-based robotics programs (Unpublished doctoral dissertation). St. John’s University, Jamaica, NY.

Leonard, J., Buss, A., Gamboa, R., Mitchell, M., Fashola, O., Hubert, T. & Almughyirah, S. (2016). Using robotics and game design to enhance children’s self-efficacy, STEM attitudes, and computational thinking skills. Journal of Science Education and Technology 25(6), 860-876.

Liang, G., Fu, W., & Wang, K. (2019). Analysis of t-test misuses and SPSS operations in medical research papers. Burns and Trauma, 7(31). https://doi.org/10.1186/s41038-019-0170-3

Marshall, J. C., Smart, J., & Horton, R. M. (2010). The design and validation of EQUIP: An instrument to assess inquiry-based instruction. International Journal of Science and Mathematics Education, 8(2), 299‐321. https://link.springer.com/article/10.1007/s10763-009-9174-y

Martin, F. G., Mikhak, B., Resnick, M., Silverman, B. & Berg, R. (2000). To Mindstorms and beyond: Evolution of a construction kit for magical machines. In A. Druin & J. Hendler (Eds.), Robots for kids: Exploring new technologies for learning (pp. 9–33). Morgan Kaufmann.

Mikropoulos, T., & Bellou, I. (2013). Educational robotics as mindtools. Themes in Science & Technology Education, 6(1), 5–14. http://earthlab.uoi.gr/theste/index.php/theste

Nunnally, J. C. (1967). Psychometric theory. McGraw-Hill.

Northwest Evaluation Association. (2019). Normative data & RIT scores. https://www.nwea.org/normative-data-rit-scores/

Papert, S. (1980). Mindstorms: Children, computers and powerful ideas. Basic Books.

Papert, S. (1993). The children’s machine. Basic Books.

Piburn, M., & Sawada, D. (2001). Reformed Teaching Observation Protocol (RTOP): Reference manual. ACEPT Technical Report No. IN00‐3. https://www.public.asu.edu/~anton1/AssessArticles/Assessments/Science%20Assessments/RTOP%20Reference%20Manual.pdf

Resnick, M., & Silverman, B. (2005). Some reflections on designing construction kits for kids. In Proceedings of Interaction Design and Children Conference (pp. 117-122). https://dl.acm.org/doi/10.1145/1109540.1109556

Saldana, J. (2015). The coding manual for qualitative researchers. Sage.

Scaradozzi, D., Screpanti, L., Cesaretti, L., Storti, M., & Mazzieri, E. (2019). Implementation and assessment methodologies of teachers’ training courses for STEM activities. Technology, Knowledge and Learning, 24(2), 247-268. https://doi.org/10.1007/s10758-018-9356-1

South Carolina Department of Education. (2018). South Carolina teaching standards 4.0 rubric. https://www.ed.sc.gov/educators/educator-effectiveness/south-carolina-teaching-standards-4-0/

Stehlik-Barry, K., & Babinec, A. J. (2017). Data analysis with IBM SPSS statistics. Packt Publishing.

Sullivan, F., & Moriarty, M. (2009). Robotics and discover learning: Pedagogical beliefs, teacher practice, and technology integration. Journal of Technology and Teacher Education, 17, 109–142. http://people.umass.edu/florence/jtate.pdf%5Cnpapers2://publication/uuid/284416E1-4D1B-48FA-8F07-583B7FCCFA47

Thompson, G. (2008). Beneath the apathy. Educational Leadership, 65(6), 50–54.

Williams, K., Igel, I., Poveda, R., Kapila, V., & Iskander, M. (2012). Enriching K-12 science and mathematics education using LEGOs. Advances in Engineering Education,3(2).  https://files.eric.ed.gov/fulltext/EJ1076110.pdf

Yesharim, M., & Ben-Ari, M. (2018). Teaching robotics concepts to elementary school children. In W. Lepuschitz, M. Merdan, G. Koppensteiner, R. Balogh, & D. Obdržálek (Eds.), Robotics in education. RiE 2017. Advances in intelligent systems and computing (Vol. 630). Springer, Cham. https://doi.org/10.1007/978-3-319-62875-2_7

Yolcu, V., & Demirer, V. (2017). A review on the studies about the use of robotic technologies in education. SDU International Journal of Educational Studies, 4(2), 127–139. http://dergipark.gov.tr/sduijes/issue/32846/340897

Yuksel, T., Delen, I., & Ilhan Sen, A. (2020). In-service and pre-service teachers’ views about STEM integration and robotics applications. Eurasian Journal of Educational Research, 90, 243–268. https://doi.org/10.14689/ejer.2020.90.13

Zhang, M., & Wan, Y. (2020). Improving learning experiences using LEGO Mindstorms EV3 robots in control systems course. International Journal of Electrical Engineering. doi: 10.1177/0020720920965873

Loading