The Need for a Series of How-to Articles<\/p>\n
In a field as complex as educational technology, conflict is common; consensus is not. Yet there is widespread agreement on at least one point: We need a more organized and persuasive body of evidence on technology’s benefits to classroom practice. Criticism of educational methods has accelerated in recent years, and the cost and complexity of implementing technology-based teaching strategies make them a favorite target. In the last decade, a diverse and growing collection of critics (e.g., Cordes & Miller, 1999; Cuban, 2001; Oppenheimer, 1997, 2003; Stoll, 1995) has been steadily making the case for reducing or eliminating the use of electronic technologies in teaching. At the same time, technology\u2019s advocates have acknowledged an increasing need for research that could address these concerns and make a strong case for technology’s pedagogical contributions.<\/p>\n
The current climate calling for accountability through evidence-based research has been building for some time and seems unlikely to diminish. Yet many authors confirm that there are weaknesses in research designs and reports that have rendered past educational studies less than helpful (Berliner, 2002; Burkhardt & Schoenfled, 2003; Kaestle, 1993; Lederman, 2003<\/a>). Other writers have focused on the special problems reflected in studies of educational technology topics (Clark, 1983, 1985, 1991, 1994; Cradler, 2003; Roblyer & Knezek, 2003; Roblyer, 2004). These weaknesses include fragmented and uncoordinated approaches to studying technology resources and strategies, methods that lack rigor or are ill-matched to the research questions at hand, and poorly written reports that render problematic subsequent attempts at replication and follow-up. We in the field of educational technology have a clear and imminent challenge. We must design and carry out research that will both address past concerns about methods and findings and clarify the directions we should take in the future.<\/p>\n With this article, CITE Journal introduces a series of examples designed to address one of the central problems of providing useful educational technology research: quality assurance. The entries in this series will serve as exemplars of each of several types of technology research study. Each will be an article re-published from a research journal, accompanied by a discussion that deconstructs the characteristics that make it a model for other researchers. This introductory article describes the challenges that must be met in order to do high quality research, discusses the criteria that will be used to select published models, and outlines the types of research studies represented in the series.<\/p>\n Problems and Challenges of Educational Technology Research<\/p>\n In one of several reaction pieces published in the Educational Researcher<\/em> shortly after the passage of the No Child Left Behind Act of 2001 <\/em>(NCLB),<\/em> Berliner (2002) made an interesting distinction between what he termed \u201ceasy-to-do science\u201d and \u201chard-to-do science.\u201d He said,<\/p>\n Easy-to-do science is what those in physics, chemistry, geology, and some other fields do. Hard-to-do science is what the social scientists do and, in particular, it is what we educational researchers do. In my estimation, we have the hardest to do science of them all….We face particular problems and must deal with local conditions that limit generalizations and theory-building\u2014problems that are different from those faced by the easier-to-do sciences. (p. 18)<\/p><\/blockquote>\n Berliner and others like him (Lederman, 2003<\/a>) offer these descriptions not as excuses for ill-conceived research, but rather by way of explanation for why even good research has had so little impact on shaping teaching methods. He cites in particular the problem of \u201cubiquitous interactions,\u201d or the sheer number of variables in any given education study that increases the difficulty of both isolating effects and of combining results across studies. For example, in any study of teaching behaviors, variables of student IQ, socioeconomic status, motivation to learn, and other factors come into play. A set of practical and logistical problems such as Institutional Review Board (IRB) approvals and permissions from schools and parents, and long-term access to busy classrooms have added yet another layer of difficulty.<\/p>\n As Roblyer (2004) observed, several conditions common to educational technology research add to the already considerable problems inherent in all behavioral research. Technologies change so quickly that it is difficult to build a body of findings over time on any given application. The flaws in the educational technology research paradigm described so convincingly by Clark (1983, 1985, 1991, 1994) have never been satisfactorily resolved. Clark insisted that comparisons of technology-based strategies with non-technology-based ones were confounded by variables such as instructional design and teacher effects. Despite Kozma’s (1991) insightful proposal that research should focus on technology-enhanced instructional designs, rather than the technologies themselves, research of the kind he proposed has been almost nonexistent.<\/p>\n It is tempting to conclude that researchers in educational technology have become convinced that \u201cchallenging\u201d means \u201cimpossible\u201d and have eschewed studies of technology’s impact. Yet as Burkhardt and Schoenfeld (2003) noted, despite the acknowledged barriers, there are precedents for even more difficult areas of study that have found the means to carry out meaningful research. They note that prior to the Flexner Report in 1910, funding for medical research was almost nonexistent and the field itself was unproven. Yet \u201c\u2026once balkanized and disputatious, medicine began slowly to cohere into a discipline,\u201d thanks to the work of a few pioneering institutions. As research became more coherent and reliable, funding followed. Though some educators object that the study of classroom learning is different from studies of physiological systems (Lederman, 2003)<\/a>, Burkhardt and Schoenfeld and others feel that education should use medical education research as its model and begin as it did, with the establishment of standards for quality.<\/p>\n Instructional Purposes the Series Will Serve<\/p>\n There are indications that the field of educational technology is gearing up to begin foundation work of the kind Burkhardt and Schoenfeld say is so essential. Recent articles (Roblyer, 2004; Roblyer & Knezek, 2003) and editorials (Bull, Knezek, Roblyer, Schrum, & Thompson, 2005; Cradler, 2003; Thompson, Bull, & Bell, 2005) serve both as an acknowledgement of past failures and a rallying cry for new attacks on the problem of providing a sound research base for technology in teaching. The field is beginning to resound with the call for a new educational technology research agenda\u2014one that focuses on capturing the unique impact of technology-enhanced instructional designs, rather than of the digital technologies, themselves.<\/p>\n This article introduces a series of educational technology research studies that can serve as models for others who wish to do meaningful research. This series proposes to build on and extend the foundation work begun in the aforementioned articles and editorials. By modeling and explicating some of the criteria required for good research, the articles in this series will show that useful research, while not easy, is doable. While a \u201cFlexner Report\u201d for educational technology remains a goal for the future, perhaps an essential first step is the recognition that it is an achievable goal. The series also should prove useful as a set of \u201chow to\u201d directions for those just beginning their research efforts (e.g., dissertation studies). To set the stage for these examples, we present major categories of criteria that future research must reflect.<\/p>\n The Pillars of Good Educational Research<\/p>\n The current demands for \u201cscientifically-based research\u201d (read \u201crandomized experiments\u201d) in education notwithstanding, studies that can move the field forward have much less to do with the type of method used in a particular study than with certain features that lend structural support to its findings. Studies that will be helpful are those that meet the following five criteria or \u201cpillars\u201d of good research.<\/p>\n Pillar 1: The Significance Criterion<\/p>\n Every educational research study should make a clear and compelling case for its existence. While significance is a subjective criterion, it seems especially important in technology research that would-be researchers recognize what makes a study significant enough to undertake in today’s educational climate.<\/p>\n In technology research, studies frequently focus on issues such as low use or non-adoption of a technology or perceptions of its current or potential usefulness. However, compared to traditional teaching methods, electronic resources are nearly always more expensive to purchase and maintain over time and technology-based methods more complex to learn and to implement. According to diffusion of innovations expert Everett Rogers (2003), expense and complexity are two of the key concerns people have about adopting innovations, so even more justification is needed for recommending them.<\/p>\n Education is replete with recognized problems related to the \u201cbottom-line indicators\u201d of quality in educational experiences: achievement, school attendance, course and degree completion, and learning time. The fact that the NCLB Act ties funding to scientific evidence of benefits (i.e., quality indicators such as achievement) has given even greater impetus for research on the impact of technology-enhanced strategies on these quality indicators. The legislation itself may change with the administration; the clamor for such evidence seems unlikely to subside. Articles that report research studies with technology-based teaching strategies should begin by making it clear that they address a significant educational problem, as opposed to a proposed technology solution.<\/p>\n Pillar 2: The Rationale Criterion<\/p>\n Burkhardt and Schoenfeld said that many barriers to progress in educational research are, in large part, \u201cself-inflicted wounds\u201d (p. 13). The most damaging of these, they say, is the failure to attend to the need for a solid theory base in research.<\/p>\n Lack of attention to coherent theory building leaves us looking balkanized and incoherent….It also leaves us vulnerable to attack from outside\u2014powerful politicians, and some academics, who understand little of what educational research is all about feel empowered to tell us how to go about our business. (p. 13)<\/p><\/blockquote>\n Researchers who wish to counter this problem have not only reviewed the past research in the area, but also analyzed the findings in light of an underlying theory of why we might expect certain results.<\/p>\n For example, studies of interactive\/electronic storybook uses in reading might propose, as did Matthew (1997) that auditory cues provided by electronic storybooks reduce the cognitive load required for comprehension by assisting with decoding words. This leads to higher comprehension, especially among students who have decoding difficulties to begin with, since it scaffolds their acquisition of better language skills. Evidence of the impact of cognitive load and its relationship to language learning has been established with past studies.<\/p>\n If we are to make progress in this field or any other, new research must carefully consider previous lines of research, and each study must be built on a foundation of theory about expected effects derived from past work. Many researchers seem to use \u201ctheory\u201d in the popular sense. However, as Kerlinger (1973) said, \u201cWhile the man in the street uses \u2018theories\u2019 and concepts\u2026in a loose fashion….The scientist…systematically builds his theoretical structures, tests them for internal consistency, and subjects them to empirical tests\u201d (p. 3).<\/p>\n After reading and scrutinizing past work that has been done on a topic, researchers ask questions like, \u201cWhy might a technology-based strategy have impact?\u201d \u201cWhat evidence do we have that a strategy has potential for impact?\u201d and \u201cWhat remains to be done to explicate past findings?\u201d This kind of analysis leads to sound research questions on predicted impact. The literature review part of the research report should show that the current study has a solid theory base and builds on and adds important information to past findings.<\/p>\n Pillar 3: The Design Criterion<\/p>\n After establishing research questions, researchers must decide on a research approach and methods that are well-suited to capturing and measuring impact on the variables of interest. The most challenging of the five criteria, this area reflects ongoing debates about what constitutes \u201cevidence-based\u201d approaches and effective design.<\/p>\n Although the NCLB Act equates \u201cgood design\u201d with \u201crandomized experiments,\u201d the research community has responded with uncharacteristic unanimity that true experimental1<\/a> or quasi-experimental methods are neither always required nor possible. The method selected should depend on the type of problem being studied and the type of information desired. Researchers seem in agreement that selecting the research approach (i.e., experimental and quasi-experimental designs) before identifying the problem is poor practice that will not advance our understanding of what works. What is essential is a design that is a logical choice for the questions under study.<\/p>\n Choice of design falls into two broad areas: (a) “objective” (sometimes referred to as “scientific”), which is usually associated with experimental, quasi-experimental, or other quantitative methods; and (b) \u201cnaturalistic inquiry,\u201d which is usually associated with qualitative methods such as narratives, phenomenologies, ethnographies, grounded theory studies, or case studies.<\/p>\n In experimental studies the researcher usually seeks to generalize results to a population, and this means ensuring as much internal and external validity as is possible by including adequate numbers of subjects, subjects selected for good representation of a larger population, methods that attempt to control or describe alternative explanations of impact, data collection that clearly measures the variables under consideration, instrumentation with reported reliability and validity, and statistical analyses that are clearly suited for analyzing the data and that are correctly applied.<\/p>\n Usually, qualitative studies aim to study impact at a given site, rather than generalize findings to other similar sites. While the focus in these studies is on thorough observation and analysis rather than controlled measurement, it is especially important that descriptions of methods in qualitative studies are detailed enough so that they may be understood and interpreted. As with all research, readers should be able to ascertain from the descriptions that methods of gathering and analyzing data were appropriate to the questions being explored and that conclusions were reasonable in light of the findings.<\/p>\n With research on educational technology, some design problems seem to be more frequent than others. Cradler (2003) found, for example, that there is often an overreliance on self-report data in methods that purport to be quantitative and measures of impact that are not objective or validated.<\/p>\n Although studies that employ experimental or qualitative approaches, as well as mixed-methods ones (Chatterji, 2004), can help researchers understand the effects of technology on student learning, an article reporting technology research should have a methods section that shows a good match between the questions being addressed in the study and the designs and methods used to carry it out.<\/p>\n Pillar 4: The Comprehensive Reporting Criterion<\/p>\n Burkhardt and Schoenfeld (2003) emphasized that educational research is in desperate need of \u201c\u2026cumulativity through studies that build on previous work\u201d (p. 8). However, due to what Mosteller, Nave, and Miech (2004) called the \u201csprawling nature of education research\u201d (p. 29), this objective is difficult. They said that cumulative studies are possible only to the extent that each one reports enough sufficiently detailed information to allow others to analyze and build on previous work. To help meet this need, Mosteller, Nave, and Miech made a good case for a \u201cstructured abstract\u201d to be included with every research report. They stated that a structured abstract will be one way of assuring that completed research is useful to those who can use and build on it. The abstract they described and provided an example for includes an original APA-style abstract, background on the study, purpose, setting, subjects, intervention, research design, data collection and analysis, findings, and conclusion.<\/p>\n Naturally, all these components should be well articulated, and conclusions should flow from them logically in light of the study findings. An article reporting technology research should include all of this information in as detailed a form as possible.<\/p>\n Pillar 5: The Cumulativity Criterion<\/p>\n Studies may be well-designed and beautifully reported, meeting criteria 1-4 in all respects, and still not be as helpful as they might to moving the field forward. Even in the so-called “hard sciences,” no single study can address all relevant issues. Therefore, the final quality each study should reflect is a clear indication of where the current study fits in providing the required information, what kinds of studies must follow, and why.<\/p>\n Technology research, especially, suffers from what might be called the single study syndrome. Perhaps the goal was to get the doctoral degree or to achieve tenure. The study is done; case closed. The best, most well-bred research has a lineage that looks to the future as well as builds on the past. In studies of human behavior, where variables are numerous and complex, it takes many studies over time to build a case for causation and impact. Consequently, articles reporting research ideally should make it clear that the study is part of a current or proposed line of research, along with proposed next steps in the line.<\/p>\n Types of Studies That Move the Field Forward<\/p>\n As Clark (1983) argued, there is a need for a new paradigm for what he called \u201cmedia research.\u201d Although Kozma (1991) answered Clark by proposing that subsequent research should focus on technology-enhanced instructional designs, rather than the technologies themselves, this kind of research has not happened to any great degree. The current climate for accountability asks educators to produce a smoking gun: clear, irrefutable, scientific evidence to justify that their choice of given technology-based designs work better than any other choice in any situation. However, it may be more realistic to seek evidence that technology-based designs lead to improved learning. One way to do this is to establish potential for relative advantage.<\/p>\n Everett Rogers (2003) said that people are more likely to adopt an innovation if they see the advantage of the new strategy relative to what they currently use. When there is a clear need for a better instructional method than those used in the past, researchers can propose that a given technology-based method is the best choice because it offers the combination of relevant symbol systems, processing capabilities, and logistical feasibility to address the need\u2014and then do research to support that it has this relative advantage and clarify the conditions under which it works best. Roblyer (2004) proposed four kinds of impact research to support this paradigm: establishing relative advantage, improving implementation strategies, monitoring impact on societal goals, and reporting on common practices in order to measure sociological impact and shape directions accordingly.<\/p>\n Study Type 1: Research to Establish Relative Advantage<\/p>\n Some supposed benefits of technology-based strategies have been talked about and discussed so frequently and for so long that educational technology professionals often assume their benefits to be well-established, yet descriptions of impact are often anecdotal and few collections of empirical studies (scientific or otherwise) exist to support them. Research lines are urgently needed to confirm and clarify oft-discussed benefits, such as prevention of inert knowledge (Brown, Collins, & Duguid, 1989) with visual technologies like simulations and video-based scenarios; increasing cultural awareness and acceptance (e.g., with distance collaboration projects); increased reading comprehension with interactive technologies (e.g., interactive\/electronic storybooks) and increased comprehension of abstract concepts (e.g., with spreadsheets, geometry software) to clarify concepts that students traditionally find complex and difficult to understand.<\/p>\n Study Type 2: Research to Improve Implementation Strategies<\/p>\n Many technology-based strategies are already in common use (e.g., online distance learning, word processing in writing instruction) and only seem likely to expand in the future. Yet, implementations of these technology-based strategies vary widely, and relatively little is known about what has the greatest impact on educational outcomes and why it works well in given situations. For example, word processing is often used in writing assessments, and online chats and conferences are much used in distance courses. Since turning back to precomputer ways of doing the same thing does not seem a good option, much more needs to be known about how to implement the current technology-based strategies to achieve the desired impact.<\/p>\n Sometimes what is not known about implementation strategies can make all the difference, as when we found that students tended to get higher scores on handwritten compositions than on word-processed ones (Roblyer, 1997). Raters seemed to have higher expectations for typed essays and were able to spot errors more easily. Once raters were trained to guard against this unintended bias, the rating differences disappeared. If educators know what causes effects like this, they can adjust their methods. If implementations are not studied over time, effects like these may never be known.<\/p>\n Study Type 3: Research to Monitor Impact on Important Societal Goals<\/p>\n Some technologies have been much promoted as ways of supporting and providing more equitable access to learning opportunities. Educators need to know whether or not students are gaining access as expected to the benefits that technology makes possible or if technologies are instead widening still further the Digital Divide. For example, are underserved students benefiting as much from access to Virtual High Schools as are their privileged counterparts? Are computer-based remedial strategies that are often viewed as the strategies of choice for struggling math students having the desired impact on improved achievement?<\/p>\n Study Type 4: Studies That Monitor and Report on Common Uses and Shape Desired Directions<\/p>\n Many new technologies (e.g., distance learning, handheld devices) are already in such common use that what we need now is clear evidence about what sociological impact they are having on school life and whether they are meeting their own ostensible goals. Do we have theories from sociology, psychology, or elsewhere that can help predict negative side effects and shape technology uses to make their impact more positive? For example:<\/p>\n Invitation to Nominate Exemplary Studies<\/p>\n This introductory article ends with an invitation to all educators in the field of educational technology and in the content areas to nominate studies to serve as exemplars of the criteria described here. We would like to include examples of the four types of studies as reflected in content-area research. Nominations may be submitted to CITE Journal editors for inclusion in this series.<\/p>\n As Roblyer (2004) noted,<\/p>\n There is much to be said for starting small and building results. If we are able to offer strong, consistent evidence of benefits for even a handful of technology uses, it will make it more likely that they would become methods of choice for given instructional situations.\u2026If we can make a strong case for even a few technology-based methods as best practices, we can justify the costs of technology resources and, consequently, even more applications will come into common use. Positive results will drive future practice and research.<\/p><\/blockquote>\n The need for this research foundation can hardly be overstated. If technology is to be viewed as having a clear and essential role to play in education, it must have a clearly articulated research agenda and high quality studies that both document and shape its impact. The evidence these studies yield will help drive effective classroom practice, preparation of educators of the future, and funding for technology-based instructional methods. The criteria described in this article and the model studies that will appear in later CITE Journal issues constitute one small but critically important step on the road to providing educational research that makes a difference.<\/p>\n\n