Measuring the Impact of EdTech Solutions
June 22, 2021
ESSA Levels of Evidence (U.S. Department of Education)
The impact of educational technology is a key question for both the startups and traditional educational publishers when looking at how to sell to schools. Teachers need to know what exactly the learning solution is teaching, why it is effective and how it can be used. The traditional educational material publishers have the advantage of a long history of collaboration with schools, and a plenty of practical experience on how to approach such clients.
Therefore, gathering real evidence on how your product affects the students, teachers and the school’s performance overall, is crucial for smaller companies as well. It makes everything easier: getting funding from EdTech investors, selling your services to organizations and convincing EdTech influencers to recommend your product to their followers.
We claim that the best and most successful EdTech companies put a lot of effort on measuring and communicating what is the true impact of their solutions. Here I’d like to present some examples of how to gather evidence on EdTech efficacy.
Pearson & Imagine Learning Show the Way
Pearson is one of the leading educational publishers in the world. It regularly conducts research on the efficacy of its products and publishes the results online for everyone to see.
The studies are designed to be highly rigorous - the experiments include a large number of students and they are followed for a relatively long period of time. For example, with their product MyPedia India they observed 557 lessons in 74 schools. The learning outcomes were measured in standardized tests and compared to national averages.
Compared to Pearson, Imagine Learning is a newcomer in educational publishing. Founded in 2004 and currently employing several hundred people, they offer digital math and language learning solutions. As a way to build trust towards their product's efficacy, Imagine Learning is publishing their research papers and case studies on their website.
ESSA Levels of Evidence - Guideline for EdTech Impact Evaluation
To prove the validity of their product research, Imagine Learning refers to Every Student Succeeds Act (ESSA) which is a non-regulatory guidance framework by the U.S. Department of education, and provides a guideline of what counts as relevant and strong research evidence of educational impact.
ESSA has four tiers of evidence, from Strong (Tier 1) to Demonstrating rationale (Tier 4). An important point to note is that the tiers are not mutually exclusive but rather serve a different purpose. When designing a whole new product or coming up with new features to an existing one, ESSA tiers can be followed step-by-step to build up the efficacy of the product, or included in a way that is most helpful for the development.
Tier 4: When designing new features, make sure they align with learning science principles. Just like if you'd build an airplane, you'd first make sure its design aligns with laws of physics, increasing the odds it'll stay in the air.
When you have a rationale why your solution enhances learning, then it's time to move on to testing with real users. A great way to conduct Tier 4. efficacy evaluation is to apply for Education Alliance Finland's EdTech Certification. In the product certification process teacher-evaluators conduct a very thorough product review using a science-based evaluation method, developed by university researchers. If the product passes the certification criteria, it will be granted the EdTech Product Certificate by Education Alliance Finland.
Tier 3: The goal for the research is to identify who are the specific groups of users your solution is designed for. You can define a certain age group you need answers from or focus on students with a specific background. Targeting only certain groups, usually leads to better findings in a reasonable amount of time.
Tier 2: Testing with larger groups and then comparing the results, builds up credibility. However, such a study requires a lot of time and is better to do after you've used Tiers’ 4 and 3 approaches to refine the product’s design. You don't want to spend 12 months gathering data only to find that e.g. your lesson plan activities are not challenging enough for the target age or don't align with the curriculum.
Tier 1: This approach is otherwise the same as tier 2, but uses randomized control groups to find out what difference the use of your solution makes - and expects the results to be published. The method is widely used in medical science. When it comes to education though, this approach is challenging because it is not possible to replicate the same learning experience while maintaining the same parameters. The students’ background, motivation and how teachers use your solution are hard to control. Minimizing the effect of these kinds of parameters in research takes a lot of work, and the study has no validity if not done correctly. So, make sure you've done the previous steps well before you move on to this stage!
Testing EdTech Product's Efficacy From the Day One
A similar kind of viewpoint on educational impact is presented by Jennifer Carolan and Molly B. Zielezinski in their Edsurge article - an Efficacy portfolio. They see efficacy as something that should be considered from the beginning and tested as early as possible in the design process - with suitable means for each company. After all, research is a powerful aid for many of the stages in a product’s life cycle and greatly affects which EdTech innovations actually continue to live on!