Most students who are entering the workforce lack the needed engineering skills to contribute to the 21st-century demands. The US Bureau of Labor Statistics (BLS) proposed that “employment in occupations related to STEM is projected to grow to more than 9 million between 2012 and 2022” (Violorio, 2014, p. 3). More specifically, engineering, architectural, and similar industries are projected to increase by 8% between 2014 and 2024 (Fayer et al., 2017).
American policymakers and educators believe science, technology, engineering, and mathematics (STEM) education must be implemented beginning with the kindergarten curriculum to increase the number of graduates in STEM fields for the future of the nation (National Research Council, 2010, 2011). While the number of STEM programs is increasing across the nation, they have failed to incorporate a rigorous and integrated engineering curriculum in which students learn collaboration, creativity, problem-solving, analytical thinking, communication, and evaluation (Jolly, 2014; Lantz, 2009; National Science Board [NSB], 2014).
To incorporate STEM education properly, specifically engineering, into the K-5 elementary curriculum, teachers require adequate training designed to improve teacher self-efficacy and a strong content-knowledge base (Al-Salami et al., 2017; Hammack & Ivey, 2019; US Department of Education [USDE], 2020; Wong et al., 2008). Researchers have shown, however, that K-5 elementary teachers are not adequately prepared and face multiple barriers in implementing a STEM curriculum (Coppola et al., 2015; Hammack & Ivey, 2019). These barriers include but are not limited to the pressure to focus on teaching to the test (Center for Science, Mathematics, and Engineering Education [CSMEE], 2000; Robinson, 2015); lack of leadership support (Hammack & Ivey, 2019; Yasar et al., 2006), skills, comprehensive level of engineering (Custer & Daugherty, 2009; National Science and Technology Council, 2018), confidence, (Sinclair et al., 2011; Stohlmann et al., 2012), and class time (CSMEE, 2000; Lantz 2009; Rogers & Portsmore, 2004; Vilorio, 2010).
Practitioners at academic institutions are challenged to search for evidence-based, practical instructional strategies and integrated approaches to improve student learning outcomes within the STEM disciplines beginning at the kindergarten level (Drew, 2011; Office of Elementary and Secondary Education [OESE], 2016; Yager & Brunkhorst, 2014).
Incorporating STEM activities into the K-5 classrooms can take many forms. The ideology of the teacher being the storehouse of all knowledge is shifting to an active role in engaging in STEM initiatives. These strategies can be integrated into other content areas to allow students opportunities to participate in programs that immerse students in activities and competitions directly related to STEM education (Moore et al., 2014; Sahin et al., 2014).
As the US faces the current and future challenges of the 21st century, STEM education practices must incorporate effective strategies that would provide equitable access to knowledge and opportunities to learn for all students of all ages in all disciplines (OESE, 2016).
STEM academic achievement affects all aspects of an individual in society, from educational choices and employment to quality of life (Pagani et al. 2001; Pellegrino & Hilton, 2012; Simonds, 2013). Providing professional development (PD) opportunities for teachers to learn how to integrate engineering into the elementary curriculum effectively is not a priority. As a result, students have missed opportunities to enhance their critical thinking and other skills required for the growing number of STEM careers (Al-Salami et al., 2017; CSMEE, 2000; Lachapelle et al., 2013; Sinclair et al., 2011). Changes must occur to meet the demands of the 21st century.
By participating in PD, teachers can receive specific, content-based training to improve their instructional techniques and increase student engagement and performance (Custer & Daugherty, 2009; Morrison, 2006; Utley et al., 2019; Yoon et al., 2013). Another goal is for educators to develop the confidence to teach engineering (Hunzicker, 2010; Quattlebaum, 2012; Stohlmann et al., 2012). Wenner (1995) suggested three essential components for teaching content efficiently and effectively: content comprehension, confidence in teaching ability, and a willingness to take responsibility for student learning (see also Bleicher, 2007; Brand & Wilkins, 2007; Kelly et al., 2017; OESE, 2016).
The No Child Left Behind Act (Congress.Gov, 2002) set specific guidelines for PD to be considered highly effective (Regional Educational Laboratory Southwest [RELS], 2007). Specifically, the PD had to be content-specific, contain intensive instruction aligned and integrated with the state standards and assessments, (see also Wojnowski & Pea, 2014; Yoon et al., 2013), and incorporate both teachers’ and school districts’ needs (Guskey, 1995; Hammack & Ivey, 2019; Hunzicker, 2010).
For educators to benefit most from PD, researchers have also shown that it should be a long-term process that recurs consistently (Guskey, 1995; RELS, 2007; Wojnowski & Pea, 2014). PD designed to occur over weeks or months compared to a few hours or single day gives educators more opportunity to receive feedback, implement strategies, collaborate with other educators, share experiences, and ask questions (Hammack & Ivey, 2019; Hunzicker, 2010; Yoon et al., 2013). Implemented PD should provide regular evaluation, consist of follow-up sessions, and monitor teacher effectiveness, student growth, and achievement (Al-Salami et al., 2017; RELS, 2007; Sinclair et al., 2011; Wojnowski & Pea, 2014).
Another characteristic of effective PD is differentiation according to grade level as STEM content standards, and student abilities differ by age and cognitive level. It is less useful for teachers to participate in PD in which the information presented does not relate to their grade level (De Jesus, 2012; Hunzicker, 2010; Quattlebaum, 2012; Yoon et al., 2013). Mosley and Utley (2006) found that teachers who participated in PD designed to increase understanding of science and mathematics content-based courses, as well as enhanced teaching strategies, significantly increased their self-efficacy regarding the subject area throughout the semester.
Engineering Is Elementary PD
Engineering Is Elementary (EiE; https://www.eie.org/eie-curriculum/about-engineering-is-elementary-EiE) is a PD opportunity for teachers to help provide engaging, inquiry-based, collaborative experiences for students through real-world, problem-solving (EiE, 2016).In 2013, personnel associated with the Next Generation Science Standards (NGSS Lead States, 2013) recommended the integration of engineering design and knowledge into the science standards. Based on this recommendation from the NGSS, the Museum of Science located in Boston, Massachusetts, began the development of EiE curriculum units designed to provide opportunities for students to learn and think as engineers through the engagement of real-world, meaningful, problem-solving challenges that integrate all areas of the curriculum (Cunningham & Kelly, 2017; Hill-Cunningham et al., 2018; Kelly et al., 2017).
According to the EiE Museum of Science (2020) website, “Each EiE curriculum unit is the result of more than 3,000 hours of development, using a process based on principles put forward in the classic design text Understanding by Design” (para 1). The process of developing each of the 20 EiE curriculum units included a 2-year, “thorough review of relevant educational research, multiple assessment steps during testing, and review by experts in the field” (para 1).
Self-Efficacy of Teachers
Self-efficacy refers to one’s belief in the ability to produce the desired outcome. Albert Bandura (1977) described self-efficacy as consisting of two dimensions — efficacy expectation and outcome expectancy. An efficacy expectation definition includes “the conviction that one can successfully execute the behavior required to produce outcomes,” and outcome expectancy is defined as “a person’s estimate that a given behavior will lead to certain outcomes” (p. 193). Self-efficacy is important for teachers as it influences teachers’ persistence when faced with difficult situations (Frost et al., 2018; Tschannen-Moran & Woolfolk-Hoy, 2001; Yoon et al., 2014).
Teacher self-efficacy correlates with increased teacher motivation, teacher support for students such as improving student comprehension, and improved attitudes of teachers and students related to the subject matter being taught (Brand & Wilkins, 2007; Cone, 2009; Joseph, 2010; Kelly et al., 2017; Riggs & Enochs, 1990; Tschannen-Moran & Woolfolk-Hoy, 2001). Teachers with high self-efficacy put forth more effort into planning and instruction (Gunning & Mensah, 2011; Tschannen-Moran & Woolfolk-Hoy, 2001). They are more comfortable and confident in teaching the content area (Hammack & Ivey, 2019; Kelly et al., 2017; Mason & McAllister, 2017; Utley et al., 2019).
Additional benefits of teacher self-efficacy include an increase in student academic scores/achievement, a close in the achievement gap, motivation and interest in math and science, independence, innovation and the creation of problem-solving, critical thinking, reasoning, collaboration, deductive, technological (computer), mathematical, and analytical skills, and the ability to make connections across curriculum areas (Bybee, 2011; Cotabish et al., 2013; Cunningham & Kelly, 2017; EiE, 2016; Lantz, 2009; Mason & McAllister, 2017; Morrison, 2006; NSB, 2012, 2014).
Self-Efficacy in Teaching Engineering
Scholars have researched the implementation of engineering methods in elementary classrooms (Cunningham & Hester, 2007; Diefes-Dux, 2015; Frost et al., 2018). This area of interest is gaining notoriety due to the change in science standards in grades K-12 and the lack of teacher self-efficacy in this area (USDE, 2020; Yoon et al., 2014). Prior researchers have used qualitative and quantitative methods such as surveys, interviews, and focus groups to examine self-efficacy using the following instruments: Teaching Engineering Self-Efficacy Scale (TESS); Engineering Design Self-Efficacy Instrument (EDSI); The Teacher Sense of Self-Efficacy Survey (TSES); and the Teacher Efficacy and Attitudes Toward STEM (T-STEM) within their research (Hammack & Ivey, 2017; Webb & LoFaro, 2020). More specifically, Hammack and Ivey (2017) used these instruments by emailing an online questionnaire to 16,546 Oklahoma K–5 public school teachers. The email addresses of the latter were on file with the Oklahoma State Department of Education.
When teaching engineering, self-efficacy is influenced by the fact that K-5 teachers tend to feel inadequate because “many have had negative experiences with science learning. Due to this fact, it is unlikely they will bring sophisticated understandings of science practices and discourses to their teaching and learning strategies” (Gunning & Mensah, 2011, p. 172).
In particular, K-5 teachers who have a desire to help students learn about engineering and a willingness to improve in their ability to teach engineering are believed to have high self-efficacy related to their beliefs and attitudes about their capacity to teach engineering effectively (Blazar & Kraft, 2017; Margot & Kettler, 2019; Shahzad & Naureen, 2017). If engineering education in the K-5 setting is to improve, teachers’ self-efficacy related to their ability to explain and interpret engineering must also advance (Blazar & Kraft, 2017). Self-efficacy is a critical principle of the self-regulatory practices of the Social Cognitive Theory, which K-5 engineering teachers employ as they exhibit confidence, reflect on their ability to motivate themselves, and self-correct the teaching and learning of science (Bandura, 1977; Cone, 2009).
Self-Efficacy and EiE
According to DiFrancesca et al. (2014), while colleges and universities do not have engineering programs, teacher educators must find solutions to bridge preparation with engineering professionals. A partnership between public schools and local engineers would provide opportunities for preservice teachers to develop engineering skills and improve self-efficacy.
EiE links the knowledge of engineers with teachers by providing PD that addresses the current lack of self-efficacy in teaching engineering content and skills necessary for successful teaching (Coppola, 2018; Cunningham et al., 2012; Cunningham & Kelly, 2017). Adding to self-efficacy, EiE offers a stable framework for the design and redesigning of preservice teacher preparation. Teacher self-efficacy in these approaches corresponds with an increase in teacher motivation and support in areas such as student comprehension and attitudes of both the teachers and students towards the subject matter (see also Fortus et al., 2005).
Notably, the research and evaluation manager at the Museum of Science reported that a series of EiE workshops sponsored by Dell served over 200 K-5 elementary teachers. One hundred eighty-five of the participants completed the workshop evaluation surveys before and after the PD. The survey measured the self-efficacy of the participants, which on average increased 33% between pre- and posttest (C. San Antonio-Tunis, personal communication, February 21, 2020). In general, educators felt unprepared to teach their elementary students about engineering and technology before participating in EiE workshops. Since minimal research about PD and teacher self-efficacy in teaching engineering in elementary schools exists (e.g., Baker et al., 2007; Bybee, 2009; Nadelson et al., 2012; Yoon et al., 2013), further research is necessary.
Purpose of the Study
This study aimed to examine the self-efficacy of K-5 elementary teachers required to teach engineering in a rural school in southeastern North Carolina. The primary research question asked was as follows: What was the influence of EiE PD on teachers’ (a) engineering teaching efficacy and beliefs, (b) engineering teaching outcome expectancy, and (c) engineering instruction.
An embedded mixed methods design examined the influence of EiE PD on K-5 elementary teachers’ self-efficacy. As Creswell and Plano Clark (2018) stated, “Embedded design mixes the different data sets at the design level” (p. 67). This research design gained its popularity from researchers testing an intervention in a school setting (Creswell, 2014). In the current study, we embedded qualitative data within a quantitative methodology. As such, within the overall design, the qualitative data was supplemental (as recommended by Creswell & Plano Clark, 2018).
The quantitative, repeated measure descriptive part of the study involved tracking the influence of the EiE PD on teachers’ self-efficacy over 6 months. Meanwhile, the qualitative data (i.e., interviews) provided in-depth information from the participants related to their perceived self-efficacy. Data were collected sequentially in this order: presurvey (Week 2), postsurvey (Week 10), interviews (Week 16), and delayed-postsurvey (Week 22). See Figure 1.
Figure 1 Embedded Mixed Methods Design
According to Creswell and Plano Clark (2018), since the embedded design consistrf of two methods to answer different research questions, researchers often publish “the two sets of results separately” (pp. 70-71). Hence, the sole focus of this paper is the quantitative aspect of the study.
Description of the Setting
This research occurred at an elementary school in a rural district in southeastern North Carolina. Approximately 740 students were enrolled in prekindergarten through fifth grade at this elementary school. The economically disadvantaged district was evident by the 96% of students receiving free/reduced lunch (North Carolina Department of Public Instruction [NCDPI], 2015). The school population contained mixed demographics with 83% Native American, 6% two or more races, 5% African American, 3% Hispanic, and 2% Caucasian students.
The primary researcher obtained written permission from the rural school district in North Carolina and the school principal, as well as Institutional Review Board approval. At a faculty meeting held at the elementary school, Parker discussed the opportunity for teachers to participate in the study. Laptops were set up, with a Qualtrics survey for teachers to consent to participate in the research and to collect demographic information.
In June 2016, interested teachers employed in one K-5 rural elementary school in southeastern North Carolina were provided an opportunity to volunteer as a participant in this study during a faculty meeting. A total of 51 teachers actively participated in the EiE PD in summer and fall 2016. Of the 51 teachers, 43 participants (84.3%) completed the Institutional Review Board consent and three surveys (pre, post, and delayed-post).
The participants were K-5 certified teachers in the state of North Carolina. They included two males (4.65%) and 41 female teachers (95.35%), and most of the participants were Native American (81.40%). Of the participants, 28 (65.11%) were between 25 and 44 years old, and 20 (46.52%) of the participants had 0-10 years of teaching experience. (See Table 1.)
Most teachers were female (n = 41, 95.34%), and a majority of the teachers (n = 29, 67.44%) were in entirely self-contained classrooms. (See Table 2.) In the self-contained classrooms, all subjects were taught by one teacher; however, in the departmentalized classes, teachers were only responsible for one or two content areas.
Table 1 T-STEM Survey Respondents by Race, Age, and Years of Teaching Experience
|Age Range (Years)|
|Teaching Experience (Years)|
|Note: Survey respondents (N = 43)|
Table 2 Survey Participants by Grade Level, Classroom Structure, and Gender
|Grade Level||Classroom Structure||Teachers (n)||Gender (n)|
|Note: Survey respondents (N = 43)|
Description of Intervention
EiE curriculum incorporates both rigorous and research-based design principles that prompts learners to think like engineers. The EiE curriculum integrated engineering with reading, science, and technology to demonstrate the connectedness of all content areas and promote project-based learning that engages teachers and students in inquiry, real-world problem-solving, critical thinking, and analyzing (EiE, 2016; Hill-Cunningham et al., 2018; Kelly et al., 2017; Mason & McAlister, 2017).
Currently, the curriculum includes built-in, inquiry-based, hands-on experiences that align with grade-level reading, engineering, and science standards. Each of the 20 EiE units provides a storybook to promote literacy and an assessment to determine comprehension. The unit design includes four lessons that scaffold and prepare students for the engineering design challenge found in Lesson 4 (EiE, 2016, 2020b; Hill-Cunningham et al., 2018; Kelly et al., 2017; Mason & McAlister, 2017).
About 1 year before the study, the elementary school and local university established a collaborative partnership (DiFrancesca et al., 2014; Webb & LoFaro, 2020). During the spring 2016 semester, faculty members and university students met with the School Improvement Team (SIT) at the school to plan a Family STEM Night. The teachers identified the yearly academic goals outlined in the School Improvement Plan (SIP). The university students created all materials needed for each activity during Family STEM Night, ensuring each met a goal identified on the SIP.
One week before Family STEM Night, the university students attended the faculty meeting at the school and modeled each activity for the teachers. The teachers mirrored the lessons for their peers with the assistance of university students. The teachers and university students collaboratively led each activity during Family STEM Night in March 2016.
In August 2016, during Week 9 of the study, participants took part in the first intervention, which included 8 hours of EiE PD on a teacher workday. Trained EiE facilitators led participants through the completion of the Thinking Inside the Box: Designing Plant Packages unit (Museum of Science, 2020). This life science unit includes four lessons in which the participants work together to problem solve as they design and create a packaging system that will keep a plant healthy and alive for several days through the shipping process (Engineering Is Elementary Research Team, 2011a). The participants were encouraged to implement engineering strategies learned during the EiE PD and bring recorded evidence to the next professional development opportunity.
During Week 18 of the study, a second 8-hour intervention of EiE professional development occurred using the balance and forces unit titled To Get to the Other Side: Designing Bridges (Museum of Science, 2020). This unit includes four lessons designed to challenge learners to think like civil engineers as they create a bridge with specific features of a beam, arch, and suspension bridge.
The engineering design challenge in lesson four requires learners to plan, build, and test and redesign bridges until they reach the desired outcome (Engineering Is Elementary Research Team, 2011b). Once again, participants were encouraged to implement newly learned strategies within their classrooms and prepare to share the recorded evidence with their peers.
Participants met by grade level during Week 22 of the study to develop activities based on the EiE PD for a future opportunity to engage students, teachers, families, and community members in a fun-filled, engineering experience. Participants worked collaboratively, integrating strategies learned from both interventions to prepare for Family Engineering Night, which was held at the end of the study.
Although two people, both trained by EiE, presented the PD, there was variance in delivery. For instance, one trainer completed the plant activity, and the other completed the building bridges activity. Also, facilitator characteristics differed (e.g., age, years of experience, gender, and education) and facilitation style (i.e., one facilitator exhibited more passion about the material).
The collection of quantitative data was provided by the administration of the Teacher Efficacy and Attitudes Toward STEM (T-STEM) Survey: Engineering Teacherspre, post, and delayed-post. Qualtrics survey software was used to collect the survey data from participants.
Teacher Efficacy and Attitudes Toward STEM Survey
The T-STEM survey contains five content-specific focus area surveys: Science, Technology, Engineering, Mathematics, and one for elementary teachers. Included in each focus area survey are seven constructs:
- Personal Teaching Efficacy & Beliefs
- Teaching Outcome Expectancy Beliefs
- Student Technology Use
- STEM Instruction
- 21st-Century Learning Attitudes
- Teacher Leadership Attitudes
- STEM Career Awareness. (Friday Institute for Educational Innovation, 2012)
For this study, three constructs were selected from the Engineering content-specific focus area survey: Personal Teaching Efficacy and Beliefs, Teaching Outcome Expectancy Beliefs, and STEM Instruction.
Through the initial development of the T-STEM survey, staff members at the Friday Institute for Educational Innovation (2012) established validity and reliability. The survey was “administered to 257 science teachers, 72 technology teachers, 17 engineering teachers, 120 math teachers, and 228 elementary teachers” (p. 3). Accordingly, staff members used exploratory factor analysis to identify survey constructs (see also Riggs & Enochs, 1990).
For each construct of the T-STEM survey, internal consistency reliability was measured using Cronbach’s alpha. The 5-point Likert-scale instrument included 11 questions on the Engineering Teaching Efficacy and Beliefs construct, nine questions on the Engineering Teaching Outcome Expectancy construct, and 14 questions on the Engineering Instruction construct and alpha coefficients ranging from .90 to .94.
Two days after the initial faculty meeting, the primary researcher (first author Parker) emailed the teachers the date for the subsequent meeting. At the meeting, teachers completed the pre-T-STEM survey electronically.
After the first EiE training, the primary researcher returned to the school within 1 week to administer the post-T-STEM survey to participants. This survey was distributed electronically during Week 10 of the study.
Delayed-Post T-STEM Survey
During Week 18 of the study, participants completed the final EiE professional development. Within a week, the primary researcher returned to the school to administer the delayed-post T-STEM survey electronically. According to Banilower et al. (2013), this timeframe may be sufficient to improve teacher’s self-efficacy.
Each participant’s T-STEM score was calculated by averaging their Likert scale response for items in each construct: (a) Efficacy and Beliefs, (b) Engineering Teaching Outcome Expectancy, and (c) Engineering Instruction. Then survey data in an Excel spreadsheet was imported into the Statistical Package for the Social Sciences (version 23) to conduct descriptive and inferential statistics.
The descriptive statistics included frequency data (counts and percentages) per item on the survey. Also, a score for each construct was presurveyed, and respective descriptive statistics were obtained. Then, repeated Measures ANOVA was used to analyze the data. According to Lomax (2001), “Repeated measures designs are used where there is at least one factor where each individual is exposed to all levels of that factor” (p. 395).
For each construct, the assumption for the Mauchly’s Test of Sphericity was examined. In each case, the sphericity assumption was violated (p = .001). Since the condition of sphericity was not met, and Epsilon was .80, .79, and .79, respectively, the Huynh-Feldt correction was used for the results of the repeated measures (Lomax, 2001). A repeated-measures ANOVA source table was provided (Meyers et al., 2006) for the three engineering self-efficacy constructs investigated.
Eta square was reported for effect size. According to Pallant (2007), the suggested norm for interpreting partial eta-squared is small = 0.01, medium = 0.06, and large = 0.14. The Bonferroni post hoc test and a p-value of < .05 were used to determine the statistical significance of the pairwise comparisons (as recommended by Lomax, 2001).
The quantitative results are organized by construct on the T-STEM survey: (a) Engineering Teaching Efficacy and Beliefs, (b) Engineering Teaching Outcome Expectancy, and (c) Engineering Instruction.
Engineering Teaching Efficacy and Beliefs
For the Engineering Teaching Efficacy and Beliefs construct, the participants’ mean increased from the presurvey (2.54, SD = 0.82) to the delayed-postsurvey (3.62, SD = .54). Although there was a minimal mean difference in the post- and delayed-postsurvey (.20), there was a 1.08 difference in the meanof the delayed-postsurvey compared to the presurvey. The means (M) and standard deviations (SD) for the Engineering Teaching Efficacy and Beliefs construct are presented in Table 3.
As indicated in Table 4, the results of the repeated measures were statistically significant for Engineering Teaching Efficacy and Beliefs: F (2,67) = 38.86, p = .001. Across time points, the means were statistically different. Regarding effective size, eta squared was .48, which is considered a large effect (e.g., Pallant, 2007).
The identified pairwise comparisons that were statistically different for Engineering Teaching Efficacy and Beliefs were the presurvey and postsurvey, as well as the presurvey and delayed-postsurvey (i.e., p < .001). There were differences in teachers’ perceptions of teaching efficacy and beliefs between the first survey administration and the other instances when teachers completed the surveys. Regarding pairwise comparisons, no statistically significant differences were found between the post and the delayed-postsurvey for the Engineering Teaching Efficacy and Beliefs. See Table 5.
Table 3 Teacher Self-Efficacy and Attitude Toward STEM Survey: Descriptive Statistics
|Engineering Teaching Efficacy and Beliefs|
|Engineering Teaching Outcome Expectancy|
Table 4 Repeated Measures of Teacher Self-Efficacy and Attitude Toward STEM: Survey Constructs
|Engineering teaching Efficacy and Beliefs||28.55||2||17.95||38.86*|
|Engineering teaching Outcomes Expectancy||2.52||2||1.59||3.91*|
|Note. SS = Sum of Squares, df = Degrees of Freedom, MS = Means Square, F = F Test|
*p < .05.
Table 5 Pairwise Comparisons of Teacher Self-Efficacy and Attitude Toward STEM: Survey Constructs
|Engineering Teaching Efficacy and Beliefs|
|Engineering Teaching Outcome Expectancy|
|Note: SE = Standard Error|
*p < .05
Engineering Teaching Outcome Expectancy
The participants’ presurvey mean for the construct, Engineering Teaching Outcome Expectancy, was 3.34 (SD = .62). By the completion of the delayed-postsurvey, the mean increased to 3.68 (SD = .53), for the means (M) and standard deviations (SD) for the Engineering Teaching Outcome Expectancy in Table 3. As shown in Table 4, the repeated measures result for Engineering Teaching Outcome Expectancy were statistically significant: F (2, 67) = 3.91, p = .001. The effect size (Eta squared) was .09, which is a medium effect (Pallant, 2007).
Mean differences between the presurvey and postsurvey (.21), the postsurvey and the delayed-postsurvey (.13), and the presurvey and the delayed-postsurvey (.34) were small. See Table 5. Since the means of the pre-, post- and delayed-postsurveys were statistically different, for Engineering Teaching Outcome Expectancy, the Bonferroni post hoc test was used for pairwise comparisons. Although one comparison, that between pre and the delayed-postsurvey was close (p = .06), the pairwise differences were not statistically significant using an alpha level of .05. See Table 5.
The descriptive statistics for Engineering Instruction revealed the mean increase from the presurvey (2.66, SD = 0.86) to the delayed-postsurvey (3.04, SD = 0.91). Notably, the standard deviations are consistently higher (i.e., 0.85 to 0.91) within this construct compared to other constructs (e.g., Outcomes). See Table 3.
Upon examination of the repeated measure inferential statistics of the Engineering Instruction construct, the condition of sphericity was not met. As shown in Table 4, the results for the repeated measures were statistically significant: F (2,74) = 2.62, p = .001. Eta squared was .06 for Engineering Instruction, which is a medium effect (Pallant, 2007).
The mean differences over time were small for Engineering Instruction. There was a .14 mean difference between the presurvey and the postsurvey. Results also indicated a mean difference of .38 between the delayed-postsurvey and the presurvey. The Bonferroni post hoc tests revealed no statistically significant pairwise comparisons. Yet, the pre- and delayed-postsurvey results yielded a .08 p-value that is close to the preestablished threshold of .05 for statistical significance. The results for the pairwise comparisons are reported in Table 5.
Investigated in this study was theself-efficacy of elementary teachers in a rural district in North Carolina who experienced EiE PD. Based on the data collected via the T-STEM survey, the means increased over time (i.e., pre-, post-, and delayed-postsurvey) for Engineering Teaching Efficacy and Beliefs, Engineering Teaching Outcome Expectancy, and Engineering Instruction. The increases in self-efficacy corresponded with other studies. For instance, on average, the Dell-sponsored EiE PD yielded a pretest score of 3.22, an average posttest score of 4.29. The scores reported for the Dell-sponsored workshop are reflective of average scores, both pre and post from educators (C. San Antonio-Tunis, personal communication, February 21, 2020).
For each of the three constructs the results of the repeated measures were statistically significant. The EiE PD had a positive influence on teachers’ self-efficacy in all three areas. Yet, when examining pairwise comparisons for the Engineering Teaching Efficacy and Beliefs construct, no statistically significant differences were found between the post- and the delayed-postsurvey. This result is likely because the delayed-postsurvey was administered 4 weeks after the final intervention. Teachers did not have enough time to implement the newly learned content because a holiday was approaching. Also, teachers may have been less focused when completing the T-STEM survey for a third time.
For the constructs Engineering Teaching Outcome Expectancy and Engineering Instruction there were no statistically significant pairwise comparisons. However, the amount of time between the pre-, post-, and the delayed-postsurvey may have influenced these specific outcomes within the study. Providing teachers with additional time to receive feedback and implement engineering strategies into the curriculum may or may not have influenced their response regarding their self-efficacy with teaching engineering on the delayed-postsurvey. Guskey (2002) and Tal et al. (2001) contended that for a significant change in teachers’ content knowledge and pedagogical views, more than 1 year of consistent support and PD is needed, as seen in the results of the Outcome and Instruction construct of this study.
Meanwhile, Banilower et al. (2013) said that while “some involvement in professional development may be better than none, brief exposure of a few hours over several years is not likely to be sufficient to enhance teachers’ knowledge and skills in meaningful ways” (p. 34). As such, this approach is more effective than traditional workshops and conferences (Hammack & Ivey, 2019; Wojnowski & Pea, 2014).
In this study, Native Americans were the predominant group of participants. Hence, the results are not generalizable to other racial groups. Interestingly, over 80% of the students enrolled at the elementary school were Native American (NCDPI, 2015). Several considerations should be taken into account when striving to boost Native Americans’ interest in the STEM content.
For instance, Native Americans prefer that information presented build on culturally responsive conceptual understanding (Kant et al., 2014a, b). The use of concrete items and experiments aid in bridging comprehension gaps (Cajete, 1986; Schindler & Davison, 1985). Traditional and cultural activities that promote STEM careers are more likely to interest Native Americans (Davis & Reid, 1999; Kant et al., 2014a, b; Smith et al., 2014).
The variation of intervention delivery, facilitator, and facilitation characteristics could have affected the study outcome (e.g., Butryn, Rohde, Marti, & Stice, 2014; Higginbotham, & Myler, 2010). Also, without a control group, we are unable to determine whether the EiE PD was the definitive factor for increased self-efficacy. The single research setting, number of survey participants, and their demographic characteristics limited the external validity of the study. This study was completed over a 6-month period of time, which is a limitation (Fullan, 2007; Guskey, 2002; Tal et al., 2001). Self-report bias, inherent in surveys, is another limitation (e.g., Curtis et al., 2020; Hill et al., 2019) of the quantitative research presented in this paper.
This study suggests multiple implications for future research. One area for future research is investigating self-efficacy and engineering integration in schools with varied teacher populations and with a more significant number of teachers (e.g., multiple school districts). A future study can examine changes across teachers’ gender, years of experience, school level, discipline taught, and education level.
In the future, scholars can include a control group as well as measures of student achievement. Sandall et al. (2018) suggested that “a final area for future research would be to replicate this study in different educational environments, such as charter schools, private schools, magnet secondary schools, and elementary settings in which STEM is incorporated into the curriculum” (p. 38). Replication, using research-based programs (e.g., EiE, 2020a, b; STEM NOLA, 2020) could determine if the results are similar across types and levels of educational settings or if variances exist that warrant further research (Sandall et al., 2018).
Across all three intervals, increases were found in the average scores for Engineering Teaching Efficacy and Beliefs, Engineering Teaching Outcome Expectancy, and Engineering Instruction. The repeated measures results provide evidence supporting the assertion that K-5 teachers’ participation in EiE PD can increase teacher self-efficacy in engineering over time. School systems and administrators can use this information to make educationally sound decisions regarding the planning and implementation of teacher PD in engineering. Teachers must be equipped with the skills and knowledge to believe they can facilitate the engineering curriculum in their classrooms.
Al Salami, M. K., Makela, C. J., & de Miranda, M. A. (2017). Assessing changes in teachers’ attitudes toward interdisciplinary STEM teaching. International Journal of Technology and Design Education, 27(1), 63–88. http://dx.doi.org/10.1007/s10798-015-9341-0
Baker, D., Krause, S., Yasar, S., Roberts, C., & Robinson-Kurpius, S. (2007). An intervention to address gender issues in a course on design, engineering, and technology for science educators. Journal of Engineering Education, 96(3), 213– 226. https://doi.org/10.1002/j.2168-9830.2007.tb00931.x
Bandura, A. (1977). Self-efficacy: Toward a unifying theory of behavioral change. Psychological Review, 84, 191-215. https://doi.org/10.1037/0033-295X.84.2.191
Banilower, E. R., Smith, P. S., Weiss, I. R., Malzahn, K. A., Campbell, K. M., & Weis, A. M. (2013). Report of the 2012 National Survey of Science and Mathematics Education. Horizon Research, Inc. http://www.horizon-research.com/2012nssme/wp-content/uploads/2013/02/2012-NSSME-Full-Report1.pdf
Blazar, D, & Kraft, M. (2017). Teacher and teaching effects on students’ attitudes and behaviors. Educational Evaluation and Policy Analysis, 39(1), 146–170. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5602565/
Bleicher, R. E. (2007). Nurturing confidence in preservice elementary science teachers. Journal of Science Teacher Education, 18(6), 841–860.
Brand, B. R. & Wilkins, J. L. (2007). Using self-efficacy as a construct for evaluating science and mathematics methods courses. Journal of Science Teacher Education, 18(2), 297–317. https://doi.org/10.1007/s10972-007-9038-7
Butryn, M. L., Rohde, P., Marti, C. N., & Stice, E. (2014). Do participant, facilitator, or group factors moderate effectiveness of the Body Project? Implications for dissemination. Behavior Research and Therapy, 61, 142–149. https://doi.org/10.1016/j.brat.2014.08.004
Bybee, R. W. (2009). K-12 engineering education standards: Opportunities and barriers. Workshop on Standards for K-12 Engineering Education. National Academies Press.
Bybee, R. W. (2011). Scientific and engineering practices in K-12 classrooms: Understanding a framework for k-12 science education. The Science Teacher, 78(9), 34–40.
Cajete, G. (1986). Science: A Native American perspective: A culturally based science education curriculum. [Unpublished doctoral dissertation]. International College of Los Angeles, California.
Center for Science, Mathematics, and Engineering Education. (2000). Inquiry and the National Science Education Standards: A guide for teaching and learning. National Academy Press.
Cone, N. (2009). Preservice elementary teachers’ self-efficacy beliefs about equitable science teaching: Does service learning make a difference? Journal of Elementary Science Education, 21(2), 25–34. https://doi.org/10.1007/BF03173682
Coppola, S. M., Madariaga, L. A., & Schnedeker, M. H. (2015). Assessing teachers’ experiences with STEM and perceived barriers to teaching engineering. ASEE. https://peer.asee.org/23583
Cotabish, A., Dailey, D., Robinson, A., & Hughes, G. (2013). The effects of a STEM intervention on elementary students’ science knowledge and skills. School of Science and Mathematics, 113(5), 215–226. https://doi.org/10.1111/ssm.12023
Creswell, J. W. (2014). Research design: Qualitative, quantitative, and mixed methods approaches. SAGE Publications.
Creswell, J. W., & Plano Clark, V. (2018). Designing and conducting mixed methods research. SAGE Publications.
Cunningham, C. M., & Hester, K. (2007, March). Engineering is elementary: An engineering and technology curriculum for children. Paper presented at the American Society for Engineering Education Annual Conference & Exposition, Honolulu, HI.
Cunningham, C. M., & Kelly, G. J. (2017). Framing engineering practices in elementary school classrooms. International Journal of Engineering Education, 33(1), 295–307.
Curtis, R. G., Olds, T., Plotnikoff, R., Vandelanotte, C., Edney, S., Ryan, J., & Maher, C. (2020). Validity and bias on the online active Australia survey: Activity level and participant factors associated with self-report bias. BMC Medical Research Methodology, 20(1), 1–8.
Custer, R., & Daugherty. (2009). The bridge: Linking engineering and society. Professional development for teachers of engineering. K-12 Engineering Education, 39(3), 18–25. https://www.nae.edu/File.aspx?id=16147
Davis, S. M., & Reid, R. (1999). Practicing participatory research in American Indian communities. The American Journal of Clinical Nutrition, 69(4), 755–759. https://doi.org/10.1093/ajcn/69.4.755S
De Jesus, O. (2012). Differentiated instruction: Can differentiated instruction provide success for all learners. National Teacher Education Center, 5(3), 5–11.
Diefes-Dux, Heidi. (2015). Introducing engineering in elementary education: A 5-year study of teachers and students. British Journal of Educational Technology. 46. https://doi.org/10.1111/bjet.12319
DiFrancesca, D., Lee, C., & McIntyre, E. (2014). Where is the “e” in STEM for young children? Engineering design education in an elementary teacher preparation program. Issues in Teacher Education, 23(1), 49-64.
Drew, D. E. (2011). STEM the tide: Reforming science, technology, engineering, and math education in America. Johns Hopkins University Press.
Engineering Is Elementary. (2020a). About Engineering is Elementary professional development: STEM empowerment through professional development for preK-8 educators, teacher educators, and STEM coaches. https://eie.org/stem-curricula/engineering-grades-prek-8
Engineering Is Elementary. (2020b). EiE classroom video library. Museum of Science, Boston. http://d7.eie.org/engineering-elementary/eie-classroom-video-library
Engineering Is Elementary Research Team. (2011a). Thinking inside the box: Designing plant packages. Museum of Science, Boston.
Engineering Is Elementary Research Team. (2011b). To get to the other side: Designing bridges. Museum of Science, Boston.
Fayer, S., Lacey, A., & Watson, A. (2017). BLS spotlight on statistics: STEM occupations – past, present, and future. U.S. Department of Labor, Bureau of Labor Statistics.
Fortus, D., Krajcik, J., Dershimer, R. C., Marx, R. W., & Mamlok-Naaman, R. (2005). Design based science and real-world problem-solving. International Journal of Science Education, 27, 855–879.
Friday Institute for Educational Innovation. (2012). Teacher efficacy and beliefs toward STEM survey. NC State University.
Frost, L., Greene, J., Huffman, T., Johnson, B., & Kunberger, T. (2018). SPARCT: A STEM professional academy to reinvigorate the culture of teaching. Journal of STEM Education, 19(1), 62-69.
Fullan, M. (2007). The new meaning of educational change (4th ed.). Teachers College Press.
Gunning, A. M., & Mensah, F. M. (2011). Preservice elementary teachers’ development of self-efficacy and confidence to teach science: A case study. Journal of Science Teacher Education, 22(2), 171–185.
Guskey, T. R. (1995). Professional development in education: In search of the optimal mix. Teachers College Press (ED369181) ERIC. https://eric.ed.gov/?q=ED369181&id=ED369181
Guskey, T. R. (2002). Professional development and teacher change. Teachers and Teaching: Theory and Practice, 8(3), 381–391.
Hammack, R., & Ivey, T. (2019). Elementary teachers’ perceptions of K‐5 engineering education and perceived barriers to implementation. Journal of Engineering Education, 108(4), 503–522.
Higginbotham, B. J. & Myler, C. (2010). The influence of facilitator and facilitation characteristics on participants’ ratings of stepfamily education. Family Relations, 59(1), p. 74–86.
Hill‐Cunningham, P. R., Mott, M. S., & Hunt, A. B. (2018). Facilitating an elementary engineering design process module. School Science and Mathematics, 118(1-2), 53–60.
Hill, N. L., Mogle, J., Whitaker, E. B., Gilmore-Bykovskyi, A., Bhargava, S., Young Bhang, I. Sweeder, L., Tiwari, P. A., & Van Haitsma, K. (2019). Sources of response bias in cognitive self-report items: “Which memory are you talking about?” The Gerontologist, 59(5), 912–924. https://doi-org.liblink.uncw.edu/10.1093/geront/gny087
Hunzicker, J. (2010). Characteristics of effective professional development: A checklist (ED510366). ERIC. http://files.eric.ed.gov/fulltext/ED510366.pdf
Jolly, A. (2014, November 18). STEM vs. STEAM:Do the arts belong. Education Teacher Week. http://www.edweek.org/tm/articles/2014/11/18/ctq-jolly-stem-vs-steam.html
Joseph, J. (2010). Does intention matter? Assessing the science teaching efficacy beliefs of preservice teachers as compared to the general student population. Electronic Journal of Science Education.
Kant, J. M., Tinant, C. J., Burckhard, S. R., & Sawyer, J. F. (2014a, Fall). Partnering with a Native American community in a collaboration between a tribal college and two mainstream universities. International Journal for Service Learning in Engineering, 460-474. http://library.queensu.ca/ojs/index.php/ijsle/article/view/5563/5330
Kant, J. M., Burckhard, S. R., Kilts, W. K., & Min, K. (2014b, October). Increasing diversity in engineering: Capacity building matters. In Proceedings of the ASEE North Midwest Section Conference. http://ir.uiowa.edu/aseenmw2014/diversity/2A/1/
Kelly, G. J., Cunningham, C. M., & Ricketts, A. (2017). Engaging in identity work through engineering practices in elementary classrooms. Linguistics and Education, 39, 48–59. https://doi.org/10.1016/j.linged.2017.05.003
Lachapelle, C. P., Sargianis, K., & Cunningham, C. M. (2013). Engineering encounters: Engineer it, learn it–Science and engineering practices in action. Science and Children, 51(3), 70–76.
Lantz, H. (2009). Science, technology, engineering, and mathematics (STEM) education: What form? What function? https://www.scribd.com/document/216341575/Science-Technology-Engineering-and-Mathematics-STEM-Education-What-Form-What-Function
Lomax, R. G. (2001). An introduction to statistical concepts for education and behavioral sciences. Lawrence Erlbaum Associates.
Margot, K., & Kettler, T. (2019). Teachers’ perception of STEM integration and education: A systematic literature review. International Journal of STEM Education, 6(2). https://doi.org/10.1186/s40594-018-0151-2
Mason, K. & McAlister, B. (2017). Advancing science learning: Teaching life science through engineering. Children’s Technology and Engineering, 21(3),10–14.
Moore, T. J., Glancy, A.W., Tank, K., Kersten, J. A., & Smith, K. A. (2014). A framework for quality K-12 engineering education: Research and development. Journal of Pre-College Engineering Education Research, 4(1), 1–13.
Morrison, J. (2006). TIES STEM education monograph series: Attributes of STEM education. http://www.wytheexcellence.org/media/STEM_Articles.pdf
Moseley, C. & Utley, J. (2006). The effect of an integrated science and mathematics content-based course on science and mathematics teaching efficacy of preservice elementary teachers. Journal of Elementary Science Education, 18(2), 1–12.
Museum of Science, Boston. (2020). EiE. https://www.eie.org/
Meyers, L. S., Gamst, G., & Guarino, A. J. (2006). Applied multivariate research: Design and interpretation. SAGE Publications.
Nadelson, L. S., Seifert, A., Moll, A. M., Coats, B. (2012). I-STEM summer institute: An integrated approach to teacher professional development in STEM. Journal of STEM Education, 13(2), 69-83.
NGSS Lead States. (2013). Next generation science standards: For states, by states, next generation science standards. https://www.nextgenscience.org/
National Research Council. (2010). Standards for K-12 engineering education? The National Academy Press.
National Research Council. (2011). A framework for K-12 science education: Practice, crosscutting concepts, and core ideas. National Academy Press.
National Science Board. (2012). Science and engineering indicators digest 2012. National Science Foundation. http://www.nsf.gov/statistics/digest12/nsb1202.pdf
National Science Board. (2014). Revisiting the STEM workforce: A companion to science and engineering indicators. National Science Foundation. http://www.nsf.gov/nsb/publications/2015/nsb201510.pdf
National Science and Technology Council. (2018). Charting a course for success: America’s Strategy for STEM education. The Whitehouse. https://www.whitehouse.gov/wp-content/uploads/2018/12/STEM-Education-Strategic-Plan-2018.pdf
North Carolina Department of Public Instruction. (2015). NC school report cards. https://ncreportcards.ondemand.sas.com/src/school?school=780364&year=2015&lng=e
Office of Elementary and Secondary Education. (2016). What are considerations when building a strong relationship between teacher preparation programs and districts that are focused on STEM teacher leadership? Building STEM Teacher Leadership. https://oese.ed.gov/stem/building-stem-teacher-leadership/building-a-strong-relationship-between-teacher-preparation-programs-and-districts-that-are-focused-on-stem-teacher-leadership/
Pallant, J. (2007). SPSS survival manual: A step by step guide to data analysis using SPSS for windows (3rd ed.). Open University Press.
Pagani, L., Tremblay, R. E., Vitaro, F., Boulerice, B., & McDuff, P. (2001). Effects of grade retention on academic performance and behavioral development. Development and Psychopathology, 13(2), 297–315.
Pellegrino, H. W., & Hilton, M. L. (Eds.). (2012). Education for life and work: Developing transferable knowledge and skills in the 21st century. National Research Council/National Academies Press.
Quattlebaum, S. (2012). Why professional development for teachers is critical. The EvoLLLution. http://evolllution.com/opinions/why-professional-development-for-teachers-is-critical/
Regional Educational Laboratory Southwest. (2007). Reviewing the evidence on how teacher professional development affects student achievement. http://ies.ed.gov/ncee/edlabs/regions/southwest/pdf/REL_2007033.pdf
Riggs, I. M., & Enochs, L. G. (1990). Toward the development of an elementary teacher’s science teaching efficacy belief instrument. Science Education, 74(6), 625–637.
Robinson, K. (2015). Creative schools: The grassroots revolution that’s transforming education. Viking Press.
Rogers, C., & Portsmore, M. (2004). Bringing engineering to elementary school. Journal of STEM Education, 3(4), 17–28. http://integratingengineering.org/stem/research/item1_ bring_engr_elem021505.pdf
Sandall, B. K., Sandall, D. L., & Walton, A, L. J. (2018). Educators’ perceptions of integrated STEM: A phenomenological study. Journal of STEM Teacher Education, 53(1), 27-42.
Sahin, A., Ayar, M. C., & Adiguzel, T. (2014). STEM related after-school program activities and associated outcomes on student learning. Educational Sciences: Theory and Practices, 14(1), 309–322.
Schindler, D., & Davison, D. (1985). Language, culture and mathematical concepts of American Indian learners. Journal of American Indian Education, 24(3), 27-34. https://www.jstor.org/stable/24398021
Shahzad, K., & Naureen, S. (2017). Impact of teacher self-efficacy on secondary school
students’ academic achievement. Journal of Education and Educational Development, 4(1), 48-72.
Sinclair, B., Naizer, G., & Ledbetter, C. (2011). Observed implementation of a science professional development program for k-8 classrooms. The Association for Science Teacher Education, 22(7), 579–594. https://doi.org/10.1007/s10972-010-9206-z
Smith, J., Cech, E., Metz, A., Huntoon, M., & Moyer, C. (2014). Giving back or giving up: Native American student experiences in science and engineering. Cultural Diversity and Ethnic Minority Psychology, 20(3), 413–429. https://doi.org/10.1037/a0036945
STEM NOLA. (2020). Growing the innovators, creators, and makers of the future. https://www.stemnola.com/
Stohlmann, M., Moore, T., & Roehrig, G. (2012). Considerations for teaching integrated STEM education. Journal of Pre-College Engineering Education Research, 2(1), 29–34.
Symonds, W. C. (2013). What contributes to youth workforce success and how states are responding. Madison, WI: Policy Institute for Family Impact Seminars. http://wisfamilyimpact.org/wp-content/uploads/2014/10/s_wifis31c01
Tal, R. T., Dori, Y. J. & Keiny, S. (2001). Assessing conceptual change of teachers involved in STES education and curriculum development-The STEMS project approach. International Journal of Science Education, 23(3), 247–262.
Tschannen-Moran, M., & Woolfolk-Hoy, A. (2001). Teacher efficacy: Capturing an elusive construct. Teaching and Teacher Education, 17(7), 783–805.
U. S. Department of Education. (2020). Science, technology, engineering, and math, including computer science. https://www.ed.gov/stem
Utley, J., Ivey, T., Hammack, R., & High, K. (2019). Enhancing engineering education in the elementary school. School Science and Mathematics, 119(4), 203–212.
Vilorio, D. (2014). STEM 101: Intro to tomorrow’s jobs. Occupational Outlook Quarterly, 58(1), 2–12. https://www.bls.gov/careeroutlook/2014/spring/art01.pdf
Webb, D.L., & LoFaro, K.P. (2020) Sources of engineering teaching self-efficacy in a STEAM methods course for elementary preservice teachers. School Science and Mathematics, 120(4), 209–219. https://doi.org/10.1111/ssm.12403
Wenner, G. (1995). Science knowledge and efficacy beliefs among preservice elementary teachers: A follow-up study. Journal of Science Education and Technology, 4(4), 307–315. https://doi.org/10.1007/BF02211262
Wojnowski, B., & Pea, C. H. (2014). Models and approaches to STEM professional development. National Science Teachers Association.
Wong, A. F., Chong, S., Choy, D., Wong, I. Y., & Goh, K. C. (2008). A comparison of perceptions of knowledge and skills held by primary and secondary teachers: From the entry to exit of their preservice programme. Australian Journal of Teacher Education, 33(3), 77–93. http://dx.doi.org/10.14221/ajte.2008v33n3.6
Yager, Y. E., & Brunkhorst, H. (2014). Exemplary STEM programs: Designs for success. National Science Teachers Association Press.
Yasar, S., Baker, D., Robinson-Kurpius, S., Krause, S., & Roberts, C. (2006). Development of a survey to assess K-12 teachers’ perceptions of engineers and familiarity with teaching design, engineering, and technology. Journal of Engineering Education, 95(3), 205–216.
Yoon, S. Y., Diefes-Dux, H., & Strobel, J. (2013). First-year effects of an engineering professional development program for elementary teachers. American Journal of Engineering Education, 4(1), 67–84. https://doi.org/10.19030/ajee.v4i1.7859
Yoon, S., Evans, M. G., & Strobel, J. (2014). Validation of the teaching engineering self-efficacy scale for K-12 teachers: A structural equation modeling approach. Journal of Engineering Education, 103(3), 463-485. https://doi.org/10.1002/jee.20049
35 total views, 1 views today