How to Tell if Your Edtech Integration is Successful

A few years ago, I was talking with a school Superintendent at the International Society for Technology in Education (ISTE) annual conference. He expressed concern about a new EdTech integration. The district recently implemented a 1:1 initiative (one computing device per student). He commented that he passed several classrooms where students weren’t using their new devices. He assessed this as “bad.” My question to him was, how did he know it was bad? Assessing an edtech implementation as “bad” because students aren’t using a device constantly is misguided.  It was as if he said an implementation is successful because all students are on a device. 

The Goals of EdTech Integration

The goal of any edtech integration should be to advance student learning outcomes. A secondary goal is to increase time for learning. This often comes through increased efficiency achieved from the technology. Dixon, Cassady, and Cross (2005) provide a great example of achieving both goals. They wanted to determine if using computers in writing impacted the critical thinking skills displayed in high school student essays. They also wanted to determine whether gender played any effect.

Example of Assessment Framework

Students were required to handwrite an essay at the onset of their junior year. During the fall of their senior year, the students were required to write a second essay. In the second essay, half of the students of each gender were asked to compose their essays using a computer. Essays were evaluated using a modified AP English Composition rubric. This helped reviewers identify displays of critical thinking. Boys in the study who used the computer to compose their essays increased their word counts by 83%. The boy’s rubric scores increased from 3.1 to 4.1 over the handwritten group.

Increasing word counts is an example of increased efficiency gained by the edtech implementation. On the other hand, increasing rubric scores exemplify advancing student learning outcomes. I.E. critical thinking. Interestingly, girls in the study showed no statistically significant difference between the two writing modalities. In this case, the edtech didn’t help or harm this group of students. 

The Power of Ongoing Assessment on EdTech Integration

Carly Britt, a Learning Experience Designer at Clarity Innovations, believes that schools and districts need an ongoing assessment process to gauge the success of an edtech integration. She notes that “to integrate edtech successfully, schools must regularly evaluate the success of their edtech implementation efforts through various measures aligned to a common vision. Leaders and educator teams can evaluate edtech success through collaborative classroom observations and learning cycles. Before observing teaching and learning in a classroom, educators collaboratively discuss best practices and edtech integration with one another to highlight areas to watch.”

The key here is that the discussion should be an actual dialogue where classroom-based educators have an authentic opportunity to have their voices heard. This can’t be a top-down initiative if it’s meant to succeed and foster growth. Carly continues that after classroom observations, “educators then discuss effective practices used, review formative and summative data gathered, and consider potential next steps for instruction and professional development.

Fostering a Culture of Learning that Supports Your EdTech Integration

This process continues to cycle to foster a culture of learning and growing collectively. The culture of learning enhances both technology implementation and instructional practices to meet all learners’ needs best.” Importantly, Carly highlights one key point about effective assessments – it’s a cycle, not an event. 

You need to discuss, design, and budget assessments in the planning stages of the edtech integration. Educators don’t implement a curriculum and then design the assessments after the students have completed the curriculum. Therefore, a well-designed curriculum uses a backward-design process where you design assessments first. The curriculum is created to ensure students have the best chance of demonstrating mastery of the assessments.

Edtech integrations should be designed using a similar approach. Furthermore, think first about the integration’s goal, then how the team will know they have met it. Therefore, having these discussions early in the process will lead to long-term success for the edtech integration. 

Measuring the Success of Integrating Tools

Measuring the success of edtech integrations requires schools to reevaluate what they are assessing. Too often, schools assess inputs (number of devices or apps implemented) or outputs (number of educators using x or the number of students who earned x certifications) (Burns, 2013). Although input/output assessments can provide valuable data, they do nothing to help educators understand the impact of the edtech integration.

What schools need to know is why and how edtech integrations had an impact. Burns notes that “impact is distal” and focuses on long-term changes at the school or district level. She notes that assessing an integration too early is akin to a doctor giving a ten-day dose of antibiotics and assessing effectiveness after three days. 

The idea that edtech integrations are human change endeavors was highlighted in Professional Development for Edtech Change. Human change doesn’t happen overnight. Additionally, human change is even more difficult when the people expected to implement the change are afraid of the outcomes. Assessment must be non-threatening and designed to gather data that improves outcomes.

An Example of a Successful Assessment

A great example of this is an online survey (see p. 10-11) that Davidson Academy Online uses to gather student feedback on the effectiveness of its online courses each semester. The leadership team administers these surveys and has been for over five years. The survey results are analyzed each semester to look for patterns and areas of opportunity. These surveys are non-evaluative for the teaching staff. They have allowed the team to improve outcomes each year and have a long-term impact on student learning. The entire team buys into the value of this assessment to improve every aspect of this highly successful online school. 

Assessment is a critical part of every successful edtech integration. Assessment should measure outcomes and impact, not inputs.  Therefore, outcomes are the immediate effects on students, and impacts are the longer-term changes at the school and district levels (Burns, 2013). Assessment needs to be a cyclical process, not a singular event. Overall, this cyclical process allows for continuous improvement and emphasizes that edtech integration is a key way the school or district is advancing student learning. In summation, as the John Doerr book title reminds us, we need to “Measure What Matters.” Above all else, respect the ideas and inputs of the people implementing the change. Successful edtech assessments are collaborative.