YouDemo enables the continuous assessment of two video-creator-defined metrics. For the remainder of this article, \u201cvideo creators\u201d include users who create or upload videos, while \u201cvideo evaluators\u201d or \u201cassessors\u201d are those who provide feedback for the videos. Creators can view the results of aggregated quantitative metric assessment as well as qualitative feedback provided by evaluators. Creators can then evaluate, reflect, and compare their own self-assessment with an aggregate of their peers\u2019 anonymous assessment of their work. This process allows video creators to gain authentic summative and formative feedback on their videos, which promotes reflection and pedagogical questioning.<\/p>\n
YouDemo provides a teaching mechanism for both formative and summative assessment that can support and enable learning at all levels of education. Additionally, the validity and reliability of tools or assignments used in the classroom are important assessment aspects, and YouDemo underwent this scrutiny. As stated by Mertler (2003),<\/p>\n
Evidence must be continually gathered and examined in order to determine the degree of validity possessed by decisions. Three formal sources of evidence that support the existence of validity include content, criterion, and construct evidence. Content evidence relies on professional judgment; whereas, criterion and construct evidence rely on statistical analyses. Content evidence of validity is the most important source of evidence for classroom assessments. As with validity, reliability addresses assessment scores and their ensuing use. (p. 66)<\/p><\/blockquote>\n
Over the course of 5 years, we trialed the continuous evaluation and video data aggregation at three universities in North America. In order to assess the impact of the tool, we conducted a mixed methods study, where a subset of the trial participants\u2019 feedback of their own videos and that of their peers was captured before and after its use.<\/p>\n
Although continuous rating and evaluation of a target source is not a new concept, having been used in election debates (Yang & Park, 2014), behavior coding practices (Messinger, Mattson, Mahoor, & Cohn, 2012), and even emotional response to music videos (Soleymani, Pantic, & Pun, 2012), we found no teaching connection. Thus, the new technology used during our study enabled preservice teachers with the means of collecting peer-assessment of any two instructor-selected video content qualities (such as content clarity, sound-level, humor, evidence of data collection, evidence of data analysis, and others).<\/p>\n
Other potential use cases include K-20 teachers collecting critique feedback on student work from a class of students, K-20 students collecting critique feedback on their work or a group’s work from a class or panel of teachers, or administrators collecting feedback on their own work, teacher work, or student work.<\/p>\n
To the best of our knowledge, no other online or free tool exists that allows continuous assessment of videos. Furthermore, no tools exist that allow users to specify and enforce the metric, or criteria, that they wish to have evaluated. The tool presented in this study, YouDemo, is a free tool for continuous, metric-focused evaluation of videos enabling formative anonymous, peer-assessment as well as experience in self-reflective practice.<\/p>\n
Theoretical Framework and Literature Review<\/p>\n
In using the video assessment technology, we embraced a social constructivist view (Vygotsky, 1978) as a theoretical framework. Focusing on the social process of learning while the preservice teachers critiqued the videos of themselves and their peers rather than only on the final product produced (the video itself) was paramount. Since STEM education is currently in the US national spotlight (Air Force Studies Board National Research Council, 2010; Bush, Karp, Popelka, & Bennett, 2012; National Governors Association Center for Best Practices and the Council of Chief State School Officers, 2010; National Science Board, 2012; National Council of Teachers of Mathematics, 2012; NGSS Lead States, 2013; National Science Teachers Association, 2012), gaining insights into STEM education video production and critique using a social constructivist perspective is important to consider while emphasizing critical content. Additionally, it is important to include the perspectives and assessments of currently underrepresented groups in STEM fields, such as minorities, students with low socioeconomic backgrounds, and women (Lehming, Gawalt, Cohen, & Bell, 2013), and how a technology implementation (like the tool presented here) might engage these groups with STEM content and purpose.<\/p>\n
Partnership building and sustained collaboration are extremely important for mutually beneficial interaction between STEM and educational partners (Borowczak, 2015; Burrows, 2011, 2015). Through the use of video technology that provides explicit feedback, teacher-to-student and student-to-student dyads can strengthen their collaboration efforts and partnerships through directed and focused reflection. Over the years, video has been used to assess pre- and in-service teachers (Hannafin, Shepherd, & Polly, 2009), enhance learning (Clarke, Flaherty, & Mottner, 2001; Williams, Farmer, & Manwaring 2008), build technical skills for careers (Clarke et al., 2001; Hunt, Eagle, & Kitchen, 2004), promote more efficient teaching and better learning (Hunt et al., 2004; Kpanja, 2001), increase student understanding (Dillon & Gabbard, 1998), and increase student participation and teamwork (Sweeney & Ingram, 2001; Ueltschy, 2001), amongst other outcomes.<\/p>\n
Thus, the whole scene of learning, or the process that leads to the product as expressed in sociocultural theory, is embraced. The individual parts in isolation do not create the scene. Using the whole scene within context will sharpen the understanding of how STEM education videos and their peer and instructor critiques can affect learning and understanding for the K-20 student audience.<\/p>\n
Building partnerships and collaborations through interactions are not limited to face-to-face meetings, as technology interactions can build partnerships and learning as well. McCabe and Meuter (2011) looked at the seven principles for good relationship practices that included (a) encouraging contact between faculty and students, (b) encouraging reciprocity and cooperation among students, (c) encouraging active learning, (d) giving prompt feedback, (e) emphasizing time on task, (f) communicating high expectations, and (g) respecting diverse talents and ways of learning (Chickering & Gamson, 1987).<\/p>\n
Determining if a tool enhances one or more of the seven principles is vital, as technology is one method to augment learning (McCabe & Meuter, 2011). Looking at technologies, and choosing the right one enables instructors to differentiate student instruction (Jones & Cuthrell, 2011).<\/p>\n
With 83% of young adults using social networking sites (McCabe & Meuter, 2011; Taylor & Keeter, 2010; Zickuhr, 2010), video is already a part of the daily life of most in-service teachers. \u201cVideo adds a new dimension to the ways in which teaching and learning can be viewed, described, and interpreted. In particular, the literature emphasizes that video footage enables data collection and analysis to be an ongoing and iterative process\u201d (Fitzgerald, Hackling, & Dawson, 2013, p. 61). Web 2.0 technologies are infiltrating schools of every level (Jones & Cuthrell, 2011). \u201cThe 21st century science classroom now contains nontraditional teaching tools, including laptops, personal digital assistants, and digital measuring devices\u201d (Bang & Luft, 2013, p. 118).<\/p>\n
University faculty members are utilizing YouTube and other social networking sites to distribute details of events and ideas (Haase, 2009). \u201cYouTube can be used as a tool to inform and display and as a forum for critical analysis and commentary\u201d (Jones & Cuthrell, 2011, p. 76). K-20 students are producing YouTube videos and displaying their own work in various settings, such as art and science classrooms (Sweeney & Ingram, 2001).<\/p>\n
As Liberatore (2010) stated, \u201cIt is clear that the tech-savvy students of the net generation enjoy finding and sharing the videos\u201d (p. 215). Acknowledging, then, that students would also like sharing self-produced videos is not a huge leap, and those self-produced projects allow for an authentic learning experience (Kearney & Schuck, 2006).<\/p>\n
Preservice teachers can benefit from recording and analyzing their own lessons (Friend & Millitello, 2014; Star, Lynch, & Perova, 2011, Van Es & Sherin, 2008). However, preservice teachers who are new to video self-observation tend to hyperfocus on their teaching methods (Fadde & Sullivan, 2013b). While coding videos can be daunting (de Mesquita, Dean, & Young, 2010), peer critique with classroom partners using video sharing and Web 2.0 technologies can generate discussion and learning with preservice teachers (Fadde & Sullivan, 2013b; Heintz, Borsheim, Caughlan, & Juzwik, 2010; Star et al., 2011).<\/p>\n
Providing preservice teachers with opportunities to practice analyzing videos of other peer preservice teachers may help the video creators eventually to evaluate video recordings of themselves (Fadde & Sullivan, 2013b). Research shows that well-defined and challenging but achievable tasks with immediate feedback are critical for skill improvement. The opportunity to correct errors and repeat the process until skills become more routine is also vital (Williams et al., 2008).<\/p>\n
There are limitations to technology use such as video assessment, and some tools will work better than others in different situations (McCabe & Meuter, 2011).\u00a0 Preservice teachers who are new to video self-observation tend to notice only their teaching delivery (Fadde & Sullivan, 2013a; Kagan & Tippins, 1991; Wang & Hartley, 2003). Using peer critique first is beneficial, since the focus is on peer teaching and the delivery is only one piece to assess (Kagan & Tippins, 1991).<\/p>\n
Methods<\/p>\n
To determine the usefulness of YouDemo in real-world preservice teacher applications, we tracked responses and solicited feedback on the tool itself. YouDemo, used in the study presented here as well as in prior studies (Borowczak & Burrows 2011; Burrows & Borowczak, 2014), enabled the assessment of online videos.\u00a0 To date, YouDemo has been utilized in over 900 assessments of 170 videos, with over 131 unique users. The tool does not edit, store, or manipulate videos in any way, rather it links to video already hosted on the Internet (e.g., YouTube).<\/p>\n
YouDemo targets three main users\u2014a video creator (e.g., a preservice student), peer evaluators (e.g., a student’s peers), and an expert assessor (e.g., a student’s instructor). Each of these users plays a different role in any assessment cycle. Figure 1 shows the five main stages within the continuous video assessment cycle, as well as the user associated with that stage: video linkage (video creator), assessment requests (video creator and expert assessor), peer-assessment (peer evaluators), aggregate assessment review (video creator and expert assessor), and sharing of results (video creator).<\/p>\nFigure 1.<\/strong> The continuous cycle of video assessment: Creation, sharing, assessment, assessment aggregation, and sharing of assessment results.<\/figcaption><\/figure>\nStage A: Video Linkage for Assessment<\/p>\n
YouDemo does not store any videos, rather it relies on an existing video sharing site such as YouTube to store and play back videos. Users wishing to add a video to YouDemo simply link to an existing online video. During this process, the user has the opportunity to provide additional details about the video, the class it pertains to, summary details and, most importantly, two metrics that they want continuously assessed throughout the video playback. Figure 2 shows the linking process.<\/p>\nFigure 2.<\/strong> The linking process consists of five required fields including a YouTube address, a video name, two metrics and a video summary. Optionally, the video creator can select a course and enter a course PIN (personal identification number) as defined by the course instructor.<\/figcaption><\/figure>\n <\/p>\n
Stage B: Disseminating Assessment Request Using Social Media<\/p>\n
Recognizing that today’s students enjoy sharing online videos (Liberatore, 2010), the YouDemo implementation connects to several popular social media platforms, including Facebook, Twitter, and Google+ (see Figure 3). This feature allows both creators and evaluators the ability to share, promote, and comment on the videos that they have added or the videos they have previously assessed. This type of propagation allows for an increased assessment population sample beyond the traditional confines of the typical classroom.<\/p>\nFigure 3.<\/strong> An example of the social media integration available to video creators.<\/figcaption><\/figure>\nStage C: Video Assessment<\/p>\n
The video assessment portion of YouDemo consists of three main areas, including the video playback panel, a live assessment stream panel, and an information panel with video details and statistics. The video playback occurs using an interface similar to other online video sites with a play\/pause button. As seen in Figure 4, the assessment stream (and the data collected from it) is controlled by the evaluator throughout the entire video using either the four directional keyboard arrows or, while on mobile devices, the four onscreen arrows. Evaluators see both a historical summary of their ratings and an instantaneous qualitative mapping of the current rating using the mappings in Table 1. Since the primary objective of the tool is to gather information in real time, an evaluator can pause the video without restarting the entire video and rating process. Upon completion of the video playback, the collected live assessment scores are stored by the YouDemo tool.<\/p>\nFigure 4.<\/strong> The assessment page containing three separate panels:\u00a0 the video panel, the assessment streams, and information panel.<\/figcaption><\/figure>\nTable 1<\/strong> \nThe Likert-Scale to Qualitative Text Mapping in the Current Implementation<\/p>\n\n
\n\n\n Likert-Value <\/strong><\/td>\nQualitative Text<\/strong><\/td>\n<\/tr>\n\n0-1<\/td>\n Non-Existent<\/td>\n<\/tr>\n \n2-3<\/td>\n Lacking<\/td>\n<\/tr>\n \n3-6<\/td>\n Average<\/td>\n<\/tr>\n \n7-8<\/td>\n Good<\/td>\n<\/tr>\n \n9-10<\/td>\n Excellent<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\nStage D: Aggregated Video Assessment Results<\/p>\n
To collect meaningful and useful peer-assessment data a process should guarantee assessor anonymity while providing aggregated summative assessment. This process is at the core of YouDemo. YouDemo enables the collection of assessment data on any two video metrics, as defined by the video creator. The tool allows video creators to view assessment results only across all assessors. Figure 5 shows an example of the aggregation process, where the average of all the individual evaluator ratings form an aggregate rating, ultimately shown to the video creator. This convention fundamentally handles the key hurdles of anonymity and aggregation.<\/p>\nFigure 5<\/strong>. Aggregation of evaluators\u2019 assessment scores anonymizes individual assessment scores.<\/figcaption><\/figure>\nFigure 6 shows the aggregated results as presented to video creators in YouDemo. In this current implementation of the technology, the video creator has access to a graphical representation of the metric score over time, as well what was \u201cliked\u201d and what needs \u201cpotential change.\u201dOnce a video is assessed, the collected data is stored, processed, and used to derive a new aggregated assessment summary, which includes both the two continuous quantitative metrics and several qualitative open-response questions that follow the video. This mixed (quantitative and qualitative) data provide the creator insight to how the video is perceived by others in both the context of the metrics selected and the evaluator\u2019s personal lens.<\/p>\n
Stage E: Sharing Video Assessment Results<\/p>\n
The ability to share aggregated evaluations allows video creators such as preservice teachers to disseminate results to an instructor, an interviewer, a mentor teacher, or even their peers in order to understand more global trends. While the ability to disseminate results is not central to the scope of this work, it may be of particular interest in the context of classroom and online instruction when the number of students makes individual assessment infeasible.<\/p>\n
The continuous video rating technology allows for peer critique of teaching videos. While the focus of this discussion is on its use in university level secondary science methods courses, implementation of this technology in other K-20 classrooms might require modification[a<\/a>] of the ways video creators add videos and metrics.<\/p>\nWhile the tool has been presented as a peer-to-peer assessment tool, another expected use is as an instructor-to-student tool in which a classroom instructor could upload a video for a flipped classroom and have metrics of \u201cDoes this make sense?\u201d and \u201cAre you learning?\u201d Students would be required not only to watch the video before class but to engage actively in rating the video. A teacher could easily view the students\u2019 overall self-assessment of the material as well as how engaging the material was in its presentation, before meeting in class with the students.<\/p>\n
Study<\/p>\n
While we have been using YouDemo for 5 years with 76 preservice science teachers, this study focused on 27 preservice science teachers\u2019 use of YouDemo over 2 years as they provided feedback to us in written and electronic forms.\u00a0 The preservice science teachers were a mixed group, with undergraduates and graduates obtaining degrees in both a STEM subject and science education. As part of their degree requirements, they took a course on how to teach science within the context of STEM integration. The course required them to create two videos per class and post them to YouTube. The videos took the form of STEM demonstrations directed at a K-12 student audience, STEM hot topic commercials, and practice teaching sessions (micro-teaches). The instructor (second author Burrows) provided guidelines that the videos should run between 2 and 10 minutes in length, highlight specific STEM content, and relate to real world STEM applications in an engaging manner.<\/p>\n
The study relies on three datasets: (a) participant self-assessment before and after their use of YouDemo (pre\/post self-assessment), (2) written peer assessments of participant videos, and (c) YouDemo assessment data of participant videos. The three datasets contained both quantitative data and qualitative data.<\/p>\n
The participant self-assessment consisted of summative assessment of the participant\u2019s own video with respect to two metrics.\u00a0 The self-assessment also contained open-response questions asking why the participant choose that self-assessment score. The written peer assessment asked the same questions (summative assessment of two metrics per video and open response) as did the self-assessment\u2014each video was peer assessed by two fellow students. Finally, the YouDemo assessment data contained formative assessment data\u2014tracking two assessment metrics throughout the entirety of the video\u2014and open-response data concerning the specific qualities of the video. The questions were as follows:<\/p>\n
\nWhat specifically do you remember about the video you just watched?<\/li>\n How did the tool affect your viewing of this video?<\/li>\n How do you think the tool could or should be used?<\/li>\n What did you like best about this video?<\/li>\n What would you change about this video?<\/li>\n<\/ul>\nWith YouDemo explained from our perspectives and grounded in the literature of video use, we explored the usefulness, interactions, and peer-to-tool assessments with preservice science teachers. The research questions we investigated were as follows:<\/p>\n
\nHow do preservice teachers assess themselves and their peers\u2014and how do these assessments compare using the tool?<\/li>\n How do preservice teachers interact with the tool and what video characteristics are most important to them?<\/li>\n<\/ul>\nAnalysis<\/p>\n
In addition to traditional statistical analysis, the quantitative data was subjected to validity and reliability testing. Validity of the YouDemo assessment tool was established by performing a correlation analysis between the per-metric and average scores computed by YouDemo and those reported by peers. The overall Pearson\u2019s correlation coefficient between the YouDemo scores and the peer reported scores was 0.71. Additionally, a similar analysis was conducted between the pre- and postself-assessment of the same metrics, and the correlation between the two samples was 0.61.<\/p>\n
Reliability of the quantitative data collected was assessed using both McDonald\u2019s omega (\u03c9) and Cronbach\u2019s alpha (\u03b1) (Zinbarg, Revelle, Yovel, & Li), and were found to be \u03c9 = 0.96 and \u03b1 = 0.89, respectively.\u00a0 The qualitative data analysis was completed using coding for themes following Tesch\u2019s (1990) eight steps. Ultimately, data collected during the study showed that the YouDemo tool is a valid and reliable method to collect data and answer the research questions.<\/p>\n
Findings<\/p>\n
Overall, the data sets indicated four main trends regarding participant interactions and comparability of the peer and self-assessments: (a) simultaneously rating two metrics had a non-zero bias or relationship between the two metrics; (b) the preservice teachers found continuous video rating beneficial in enabling video assessment, promoting critical thinking and increasing engagement; (c) the preservice teachers\u2019 self-assessments were uncorrelated with their peers\u2019 assessments; and (d) students with lower self-assessment were rated higher by their peers.<\/p>\n
The quantitative data, derived from the actual video assessment tool, as well as pre- and postsurveys, revealed interesting patterns and relationships between the preservice teachers\u2019 self-assessments and their peers\u2019 assessments. Additionally, the relationships between the formative peer-assessments and the computed formative peer-assessments using the continuous summative assessment provided by the YouDemo technology showed trends in the data. This data enabled us to answer the second research question, \u201cHow do preservice teachers assess themselves and their peers, and how do these assessments compare using the tool?\u201d<\/p>\n
Figure 7 links to a Plotly graph [b<\/a>] online showing the relationship between the two assessment metrics for all videos. A slight bias existed between the metrics. Over a large data set, an unbiased relationship between metrics would produce a linear regression through the line Y<\/em> = X<\/em>. In our data, the relationship between the metrics was skewed such that Y<\/em> = 0.78*X<\/em> + 1.25. When the first metric\u2019s average was below 5, the second metric\u2019s average tended to be less than the first, and when the first metric\u2019s average was above 5 the second metric\u2019s average tended to be greater than the first.<\/p>\n