High quality research is the foundation from which effective and impactful EdTech is built on. Educators, institutions of learning, and companies depend on research to make informed decisions about what products and interventions to implement. There is, of course, some responsibility on those choosing products to do their due diligence on the research backing such products. But, on the other hand, the greater responsibility should fall on the companies and researchers conducting studies.
Why is research so important? Because we want to know what actually helps students and the impact that EdTech products are having on our populations of focus. If EdTech is interested in building long-term, effective, and sustainable solutions to drive innovation, then credible and rigorous research is essential to accomplish these goals.
Our research practices and policies at WGU Labs were designed to ensure that the research we’re doing on EdTech products and educational interventions is rigorous, transparent, and impactful. There are three problems that EdTech focused companies, especially those like WGU Labs that also work with third-party EdTech products, must solve to produce rigorous research:
- The Research Problem
- The Credibility Problem
- The Accessibility Problem
The Research Problem
The Research Problem is characterized by the broader “replication crisis” in psychology, which became mainstream in 2015 when the first large-scale replication attempt of classic psychology findings failed to reproduce effects of nearly two-thirds of studies. Several follow-up replication efforts, including the several Many Labs efforts, have also failed to replicate many studies. As a field, these efforts have replicated fewer than 50% of the findings, on average.
The reasons why psychology studies fail to replicate are complex and varied, but some big reasons stand out. The first is that psychology studies are often underpowered, with fewer than 50% of educational psychology studies being adequately powered. What this means is that psychology studies typically have sample sizes that are too small to accurately detect an effect of, for example, an education intervention on student performance. The estimated effect sizes from underpowered studies—the size of the difference between two groups (i.e., a treatment and control group) on an outcome of interest—are highly variable, and false positives are more likely. Put differently, as sample sizes increase, the true effect of an intervention will become more stable and less error prone.
Second, researcher degrees of freedom or questionable research practices which can include a broad range of behaviors such as reporting only certain outcomes that a study measured, dropping intervention conditions that didn’t work, massaging data until a positive finding is found (known as “p-hacking”), or changing the study hypothesis after the results are known (known as “HARKing”), are more likely to produce false positive findings that won’t replicate.
Why does psychology matter for EdTech research? Because psychology research is often the basis for educational and technological interventions in the classroom. As Annie Brookman-Byrne said, “If an educational trial is based on a psychological finding that has not been replicated and shown to be reliable, the chances of the trial working are slim.” How are educational trials doing? A recent analysis of more than 1.2 million students in K-12 educational randomized control trials across the US and UK showed an average effect size of .06. There is a lot of room for improvement.
The Credibility Problem
The Credibility Problem is characterized by conflicts of (financial) interest that arise when (EdTech) companies hire, or become financially intertwined with, other companies to conduct product efficacy research. When evaluating the credibility of any research, an important aspect to look at is who did the research. Often EdTech companies will have investors, and some of those investors will provide research services in the form of user testing and/or efficacy research to evaluate the impact of the EdTech product on student outcome metrics.
Efficacy research is where the credibility problem can arise. Studies conducted by a company, such as an accelerator like WGU Labs, for example, that is financially intertwined with an early stage EdTech company, may have, or may be perceived to have, financial interests in producing positive results—of course the product works! Because there is money involved, company funded research on EdTech products is perceived as less credible than studies conducted without financial conflicts of interest. The credibility problem is real, and the conflict of interest that necessarily arises when finances become involved in research can undermine the trustworthiness of the results to the public.
The Accessibility Problem
Finally, the Accessibility Problem is characterized by the difficulty the public—and many scientists—have when trying to access research. If educators, institutions of learning, and companies are expected to evaluate the research of EdTech products, then that research should be easily accessible and presented clearly. The problem is that research reports often don’t include all the information evaluated in the study (e.g., selective reporting), the report is not written in accessible language for non-researchers, negative or non-favorable outcomes are suppressed, and published reports are locked behind expensive paywalls.
Open Science as a Solution
So, what is a solution to these three problems? By practicing open science EdTech research can be more rigorous, transparent, and impactful. The Center for Open Science (COS) is a non-profit company that is supported by a variety of federal and private institutions with the goal of making science open and transparent for researchers and the public. The COS, and its Open Science Framework (OSF), facilitates open science practices that can help solve these problems in several ways.
The first important component of open science practices is pre-registering research studies, especially when third-party companies are involved. Pre-registration is a transparent science service that is offered by the OSF (and other organizations). Pre-registration is a timestamped electronic document containing detailed study information, including hypotheses, measures, and analysis plans. Pre-registering a study timestamps and archives the research plan and hypotheses agreed upon prior to the start of data collection. Pre-registration of a research plan holds researchers accountable for their decisions when analyzing and reporting the results of a research study. Pre-registration helps protect against p-hacking and HARKing, leading to more credible and accurate research results. This directly helps solve important aspects of the Research Problem.
Pre-registration also makes findings more trustworthy, especially for EdTech research where a conflict of interest (COI) may be present. Because of the COI that arises when two companies are financially involved, positive research results are perceived with extra skepticism. A pre-registered research study can help mitigate the skepticism arising from COIs. A research study that produces positive results and follows a pre-registered research plan (e.g., hypotheses, measures, analysis plan) will be more credible to the public and other scientists.
Finally, open science is transparent and accessible—to everyone. Pre-registrations and study plans are time stamped and available either immediately or after a set embargo period (which has an upper limit on the OSF, so everything is public eventually). A single project link can contain the study registrations, supplementary materials, and final research reports and papers that are generated from the study. There are no paywalls, no logins, and no special institutional status required to access the research. When used correctly, OSF is essentially a one stop place on the internet for research information.
Here at WGU Labs, we’re committed to rigorous, transparent, and impactful research (our research values). Our research processes involve creating a research plan for all EdTech and intervention research and striving to pre-register those research plans. WGU Labs has an OSF profile where the public can see what kinds of work we’re doing. We’ve pre-registered our first EdTech intervention, the TimeCoach™ Efficacy Study on the OSF. Keep an eye on this space for more projects in the coming months!
In our research, we strive for rigorous design, theory-driven interventions and products, and accessible results. Open science is a solution for rigorous and impactful research, but it is not the only solution. Open science can improve credibility and accessibility, but it does not replace good theory, educational experience, and well-designed research studies.
Follow us on Twitter at @WGULabs and Linkedin for more articles and interviews.
