A Deconstructed Example of a Type 2 Study: Research to Improve Implementation Strategies

Abstract

This article explores an exemplar of a Type 2 (Research to Improve Implementation Strategies) study. As the introductory article in this series described (Roblyer, 2005), our increasing reliance on technology-based communications has put many technology-based strategies into common use. These have often become strategies of choice not necessarily because they result in higher achievement or savings of time (for which there are few Type 1 studies to confirm), but because they use technologies that have become the automobiles to replace yesterday’s horses and carriages. Because implementations of these technology-based strategies vary widely and situations in which they are used have infinite permutations, we need many studies that examine thoroughly why certain implementations of a given technology can work well while others do not. While we cannot answer all questions about all possible implementations, we can discover trends that yield guidelines for how technologies should be implemented for maximum impact in many or most environments. The published study reviewed in this article offers such guidelines for the design and use of multimedia materials.

Introduction: An Exemplary Technology Research Study

Educational Research: Many Restaurants But Few Gourmet Meals

Though historians tend to put the dawn of the Information Age around the appearance of the first computers in the 1950’s and 1960’s, the current digital era of this Age began when the first Internet browser went live, circa 1994. As throngs of spectators signed up and logged on to view the first Web sites, the graphical Internet also gave would-be authors and publishers a glimpse of the power of information sharing. A scattered few at first, then a whole host of information venues sprang up all along the Information Highway – and not just Web-based ones. Online forums, Web sites, blogs, and email were joined by hundreds of new print journals, magazines, newspapers, and newsletters to allow unprecedented opportunities for offering both facts and opinions: a smorgasbord of information restaurants to visit on the Internet and beyond.

Perhaps it is not surprising, though, that an increased number of places to publish does not result in a throng of high-quality research studies; it is so much easier to write about research than to do it, so much faster to offer light snacks of studies than to prepare the genuine gourmet meal of substantial research. Thus, when one looks for exemplary studies to use as models for students and others interested in doing high-quality work of this kind, it is difficult to find published examples; educators remain hungry for substantial research results.

The introductory article to this series outlined four kinds of studies that could move the educational technology field forward and that are lacking in the current published research base. These include

• Type 1: Research to Establish Relative Advantage – Studies that show that a given technology-based strategy is better than other strategies in common use because it has unique features that help bring about improved achievement, better attitudes, greater time on task, and/or more efficient learning on a topic (e.g., increasing reading comprehension through use of interactive technologies such as electronic storybooks)

• Type 2: Research to Improve Implementation Strategies – Studies on how to implement technology-based strategies that are already in common use so that they have greater instructional impact and benefits (e.g., implementing use of word processing for writing instruction)

• Type 3: Research to Monitor Impact on Important Societal Goals – Studies to indicate that technology’s impact on society is positive and that society-wide goals for technology are being met as originally envisioned (e.g., the goal of more equitable access to learning opportunities for underserved students)

• Type 4: Studies That Monitor and Report on Common Uses and Shape Desired Directions – Studies to predict and prevent negative sociological side effects of technology uses and bring about appropriate adjustments to make its overall impact on education more positive (e.g., how to address the issues and problems inherent in the current practice of students bringing handheld devices to school)

Background on Type 2 Studies

The current article explores an exemplar of a Type 2 study. As the introductory article in this series described (Roblyer, 2005), our increasing reliance on technology-based communications has put many technology-based strategies into common use. These have often become strategies of choice not necessarily because they result in higher achievement or savings of time (for which there are few Type 1 studies to confirm), but because they use technologies that have become the automobiles to replace yesterday’s horses and carriages. For example, online distance learning coincides with a society-wide need for on-demand educational opportunities and seems destined to take a position of equal importance with face-to-face teaching. Word processing has replaced handwritten or typed communications both at home and in the world of work and, thus, has begun to permeate writing instruction in schools. Multimedia communications are a ubiquitous feature of modern communications and have begun to appear with increasingly frequency in distance instruction.

Because implementations of these technology-based strategies vary widely and situations in which they are used have infinite permutations, we need many studies that examine thoroughly why certain implementations of a given technology can work well while others do not. While we cannot answer all questions about all possible implementations, we can discover trends that yield guidelines for how technologies should be implemented for maximum impact in many or most environments. The published study reviewed in this article offers such guidelines for the design and use of multimedia materials.

A Review of a Type 2 Exemplar: Moreno and Mayer’s Verbal Redundancy Studies

It is difficult to find a professional team currently working on educational research whose work is as consistently well conceived and reported and whose findings are as useful as that of Richard Mayer, Roxana Moreno, and their colleagues. Mayer et al.’s studies on the uses of multimedia tend to focus not so much on the technology itself but on the instructional strategies it makes possible. It is useful more from the stance of appropriate strategies for multimedia use in specific content areas than from justifying technology’s choice in a general situation. (See reference list at the end of this article.) This makes several of their articles good examples of Type 2 studies.
The study selected for deconstruction here is “Verbal Redundancy in Multimedia Learning: When Reading Helps Listening” (Moreno & Mayer, 2002). Copyright permissions of the American Psychological Association (APA) do not allow republishing Moreno and Mayer’s article – even as an exemplar. However, CITE did receive permission to republish the article’s abstract. We encourage readers to obtain and read the full text of this article, either from your campus library or directly from APA’s Journal of Educational Psychology (http://www.apa.org/psycarticles). This useful study clearly meets each of the “pillars of high quality research,” that is, criteria for good research studies referred to in the introductory article (Roblyer, 2005).

Pillar 1: The Significance Criterion

The significance criterion holds that an educational research study should make a clear and compelling case for its existence. Authors should explain why they felt the study was worth spending time and resources to pursue. Though researchers rarely focus on the expenses involved in carrying out a given study, the costs of research are real and of practical concern. Thus, research should be more than an interesting scientific diversion; researchers must begin from the premise that the study has real potential for findings that can further the field.

Verbal redundancy refers to “the simultaneous presentation of text and narration with identical words” (Moreno & Mayer, 2002, p. 156). The need to study such phenomenon in instructional multimedia environments should be somewhat self-evident to anyone who has watched young people using several communications technologies at one time, apparently relishing the deluge of information and sensory inputs, and ostensibly paying attention to each one. Are they really attending to and remembering all this simultaneous input? If so, perhaps multimedia instruction can take advantage of these cacophonous environments in which students seem to thrive.

Moreno and Mayer explain that multimedia learning has been proposed as beneficial for conceptually difficult instructional materials, for example, those often seen in the study of scientific principles. They want to know under what conditions multimedia-based instruction can, indeed, help students learn. In other words, how should multiple inputs of multimedia instruction be configured for maximum impact on learning? They point out that the findings have significance for the design of multimedia instruction.

Pillar 2: The Rationale Criterion

As a basis for their study, researchers should have some findings from previous research for the studies they propose, and they should use these findings to generate research questions on predicted impact for their own study. The literature review part of the research report should show that the current study has a solid theory base and builds on and adds important information to past findings. Researchers should not have only reviewed the past research in the area, but also analyzed the findings in light of an underlying theory and synthesized them into a statement on why we might expect certain results.

Moreno and Mayer’s introduction and literature review is a stellar example of this criterion. It is noteworthy that most of the studies they summarize are from outside educational technology, drawing instead on research from educational psychology and studies from content areas and even those outside education (e.g., training for job tasks). In a review of past research on verbal redundancy, they found several studies whose findings indicated that material presented through more than one sensory channel at the same time made possible better comprehension, increased recall, and faster response times on job tasks. The theoretical explanation for this benefit is based in the “redundant signals effect” (p. 156), which has its roots in information processing theory. It proposes that presenting information in dual modes (e.g., written and spoken) makes use of the two strands that comprise working memory: visual and auditory. This dual presentation makes it more likely that information will get transferred into long-term memory.

However, Moreno and Mayer also found other studies with opposite findings when using a different combination of input channels. They found that students’ learning can be negatively affected by dual presentation of nonverbal (a graphic diagram) and verbal (written text) modes. These researchers also explained their results in terms of information processing theory, this time from cognitive load principles. They proposed that certain kinds of redundancy (e.g., presenting information in the verbal and nonverbal modes of text-plus-graphics) can force learners to split their attention, which “overburdens their limited working memory capacity” (p 157). Other studies found that presenting information in diagrams and auditory explanations is more efficient than in diagrams-plus-text or in the combination of with diagrams-plus-auditory and visual explanations.

Moreno and Mayer proposed that what matters is that “relevant information in each mode is
selected, organized into a coherent representation, and connected with each other” (p. 157). This led to the two central questions in the series of studies reported in this article: “(1) Does the addition of on-screen text to an otherwise intelligible spoken explanation promote the deep understanding of a complex scientific system?” and (2) “How is the processing of verbal information affected by the presentation of additional nonverbal information, such as graphics and sounds?” (p. 157).

Pillar 3: The Design Criterion

The design criterion holds that the methods researchers use to study their topic must be well suited to capturing and measuring impact. This is the most challenging of the five criteria and, due to ongoing debates about what constitutes “evidence-based” approaches and effective design, the least likely to meet with unanimous agreement among a given group of researchers. However, Moreno and Mayer seem to do an outstanding job with experimental methods. To test their two central research questions, they set up three different experiments around the learning of “how lightning works.”

• To test Question 1 – To find out whether adding onscreen text to a (well-designed) spoken explanation helped students learn better, they compared the learning outcomes of a group of students who were given only a verbal/auditory explanation (i.e., a narration) of how lightning works with those of other groups of students who received the same verbal/auditory explanation plus other types of explanations (see Table 1).

• To test Question 2 – To find out how adding graphics and sounds to a spoken explanation affects students’ processing of verbal explanations, groups were also compared as to their learning outcomes after they received various treatments (see Table 1).

Table 1
Summary of Treatments and Outcomes in the Three Experiments

Experiments/Groups	Verbal (words)/ Nonverbal (images)	Modality(ies)	Outcomes
*Experiment 1: Adding animation before* explanation(s)**
Narrated explanation alone	Verbal only	Auditory only	Better results when any two different modes were used
Narrated explanation withonscreen text	Both verbal	Auditory + visual
Animation before narrated explanation	Nonverbal + verbal	Visual + auditory
Animation before narrated explanation + onscreen text	Nonverbal + verbal	Visual + auditory
*Experiment 2: Adding animation before vs. after* explanation(s)**
Animation with narrated explanation	Both verbal	Auditory + visual	Better results when verbal and nonverbal modes were used in sequence, rather than at once
Animation with narrated explanation + onscreen text	Nonverbal + verbal	Auditory + visual
Animation before narrated explanation	Both verbal	Auditory + visual
Animation before narrated explanation + onscreen text	Nonverbal + verbal	Auditory + visual
Experiment 3: Adding sounds to explanation(s)
Narrated explanation	Verbal only	Auditory only	Better results without addition of sounds
Narrated explanation withonscreen text	Both verbal	Auditory + visual
Narrated explanation withenvironmental sounds	Verbal + nonverbal	Auditory only
Narrated explanation withonscreen text and environmental sounds	Verbal + nonverbal	Auditory + visual

The design for these experiments was appropriate – even ingenious. It was based on what they expected would happen in light of a combination of verbal redundancy and split attention findings of previous research. “Because the auditory and visual processing channels are independent … students can hold both representations in working memory at the same time and build referential connections between them … When narration and on-screen text are processed simultaneously, students are able to integrate both inputs and build a coherent verbal representation” (p. 157). Therefore, they predicted better learning with redundant messages – but only if verbal explanations and nonverbal messages (such as animations or diagrams) are not presented simultaneously, but sequentially.

The only aspect about the design that is not made clear in this report is whether students were randomly assigned to groups. One can assume they were, but it is never explicitly stated. Thus, it is difficult to tell if these were true experimental designs or quasi-experimental ones.

Pillar 4: The Comprehensive Reporting Criterion

The comprehensive reporting criterion says that a research article must offer sufficiently detailed information to allow others to analyze and build on previous work. With the exception about randomization noted above, the authors certainly met this criterion; they gave detailed descriptions of experimental subjects, treatments, measures of learning and retention, and results, as well as a thorough analysis of what the results meant.

This was an educational psychology journal, so perhaps typical readers of this article would follow effortlessly the explanation of how groups were comprised. However, to this reviewer’s less-experienced eye, the “verbal vs. nonverbal,” “visual vs. auditory,” and “redundant vs. nonredundant” designations began to swim like a guppy screensaver before my eyes. It would have been helpful to have a little table like the one included in this article to keep the various treatments straight.

Pillar 5: The Cumulativity Criterion

This criterion addresses the need for building a body of evidence over time, rather than doing studies on a “one shot” basis. Moreno and Mayer also clearly meet this criterion. As was noted in the opening paragraph of this section, the study reported here is part of a long series of related work they have done in this area. Clearly, much more work of similar quality needs to be done to replicate and build on the work reported in this series of experiments. Although the studies were well-designed, it may be that the subject matter confounds the results. Similar strategies should be used in other science or mathematics areas to see if the guidelines offered here hold true for other topics and content areas with similarly complex concepts.

In Conclusion: Invitation to Nominate Exemplary Studies

The article reported here contributes in important ways to the research foundation that is sorely needed in educational technology. It offers benefits of several kinds. First, the clearly articulated theoretical and research foundation, which lays the groundwork for both its methods and for predicted results, does much to validate the potential for technology’s benefits to instruction. Second, it models the kind of coherent, thoughtful design and reporting that others can emulate and that is much needed to provide defensible justification for using technology in education. Finally, its findings offer substantial, if tentative, guidelines for an important, expanding area: multimedia-based instruction.

As did the introductory article, this article ends with an invitation to all educators in the field of educational technology and in the content areas to nominate studies of similar high quality to serve as exemplars of the criteria described here. We would like to include examples of each one of the four types of studies as reflected in content-area research. Nominations may be submitted to CITE editors for inclusion in this series.

References

Moreno, R., & Mayer, R. (2002). Verbal redundancy in multimedia learning: When reading helps listening. Journal of Educational Psychology, 94(1), 156–163.

Roblyer, M. D. (2005). Educational technology research that makes a difference: Series introduction. Contemporary Issues in Technology and Teacher Education [Online serial], 5(2). Retrieved July 5, 2006, from https://citejournal.org/vol5/iss2/seminal/article1.cfm