--- **_Quantifying the Learning Benefits of AI-Enhanced Video Instruction_** *Kelly Puzio, PhD* ## Abstract This study investigated the impact of AI-enhanced interactive elements in video-based instruction on learning outcomes and engagement. In a between-subjects experiment, 220 teachers were randomly assigned to either video-only instruction or video instruction with interactive elements (AI created drag-and-drop activities and AI-simulated video conversations) focused on personality types and communication strategies. Results revealed that participants in the interactive condition demonstrated significantly higher post-test scores (*p* = 0.008, Hedge's *g* = +0.36) compared to the video-only condition, with particularly strong benefits for novice learners (*g* = +0.67). Notably, this improvement is comparable to the learning gains typically observed in tutoring programs, underscoring the practical significance of integrating interactive AI elements into video-based instruction. While self-reported engagement did not differ significantly between conditions, the AI interactive group showed substantially higher Net Promoter Scores (+19 vs. -4), indicating greater willingness to recommend the learning solution. These findings suggest that AI-enhanced interactive elements can significantly improve learning outcomes, particularly for individuals with lower prior knowledge, while also enhancing the perceived value of educational content. Educational designers should consider incorporating such interactive elements into digital learning experiences, especially when targeting learners with limited domain knowledge. ## Background The landscape of education continues to evolve as advances in technology transform both teaching and learning. This evolution is particularly evident in the growing emphasis on active approaches to learning, which have consistently demonstrated superiority over passive learning methods across multiple disciplines. Active learning refers to instructional approaches that engage students through meaningful activities, conversations, and negotiations of meaning rather than passive (sit and get) consumption of information (Chi & Wylie, 2014; Theobald et al., 2020). The effectiveness of these approaches is well-documented, with research showing that active learning can reduce course failure rates by 55% and improve achievement by an average of 0.47 standard deviations (Freeman et al., 2014). The cognitive science underlying active learning reveals why these approaches are so effective. When students actively engage with learning materials, multiple brain regions activate simultaneously, including areas associated with memory formation, emotional processing, and executive function (Brod et al., 2016). This simultaneous activation creates stronger neural pathways, facilitating better long-term retention and knowledge transfer to new contexts (Zhou et al., 2019). In contrast, passive learning primarily activates basic sensory processing regions, resulting in weaker neural engagement and less durable learning outcomes. Active learning operates through both private (cognitive) and public (social) processes. Private meaning-making occurs through activities like retrieval practice, self-assessment, and metacognitive reflection, which have been shown to significantly improve learning outcomes in a wide variety of disciplines (Pan & Rickard, 2018; Zingaro & Porter, 2016). Equally important are the social aspects of learning, where discussions and collaborative activities enhance both technical competency and professional skill development (Chan & Blikstein, 2018). The synergy between private and public forms of active learning creates a powerful learning environment that consistently outperforms traditional methods. When embedded inside digital learning experiences, interactive elements represent a promising approach to implementing active learning principles at scale. These interactive components can range from adaptive quizzes and concept mapping activities to simulated conversations and collaborative problem-solving tasks. By categorizing such elements, educators can create opportunities for both private cognitive engagement and public meaning-making. Research in computer-supported collaborative learning has shown significant improvements in knowledge acquisition (*d* = +0.42) and problem-solving (*d* = +0.64) skills (Chen et al., 2018). Research consistently demonstrates that interactive learning environments provide particularly strong benefits for novice learners compared to more experienced ones (Kalyuga, 2007; Sweller et al., 2019). This “expertise reversal effect” occurs because novices lack well-developed mental models and benefit from the scaffolded guidance and immediate feedback that interactive elements provide. Novices show significantly improved outcomes when learning with interactive components that manage cognitive load through worked examples (Renkl, 2014), visualization aids (McElhaney et al., 2015), and adaptive feedback systems (VanLehn, 2011). These approaches effectively bridge the gap between novices’ limited prior knowledge and complex learning demands, making interactive learning especially valuable for beginners in any domain. Despite the compelling evidence supporting active learning in general, there remains a gap in understanding how AI tools can support private and public forms of active learning. While artificial intelligence technologies offer unprecedented opportunities for creating interactive, adaptive, and personalized learning experiences, empirical research examining their specific impacts on learning outcomes and engagement remains limited. Although research has established the general effectiveness of active learning principles, we lack sufficient evidence about specific AI-powered interactive features that effectively promote learning and engagement for different types of content and learner populations. Additionally, most studies have focused on traditional classroom implementations of active learning rather than AI-enhanced digital learning experiences, leaving crucial questions about how AI can best be leveraged to implement active learning principles in online instructional materials. This study aims to address these gaps by examining how different interactive elements affect learning outcomes and engagement when teaching about personality types and communication strategies. By comparing video-only instruction to video instruction enhanced with one cognitive form of active learning and one social form of active learning, we seek to contribute to the growing literature on effective implementation of AI-supported active learning principles in digital learning environments. ## Research Methods --- This investigation focused on examining the effectiveness of AI-interactive elements in video-based instruction on learning outcomes and learner engagement. We sought to understand how incorporating various interactive components affects knowledge acquisition and self-reported engagement when learning about interpersonal communication strategies. Specifically, the following questions guided this investigation: 1. To what extent does the addition of interactive elements (drag and drop activities; AI-simulated real-time video interactions) to video-based instruction affect learning outcomes compared to video-only instruction? 2. To what extent does the inclusion of interactive elements in instructional videos influence learner engagement or promotion compared to traditional video-only instruction? 3. To what extent do participant variables (e.g., pretest) influence learning, engagement, or Net Promoter outcomes? ## Study Design and Participants This study employed a between-subjects experimental design with 220 teachers recruited through Prolific, randomly assigned to either a control group (video-only instruction) or experimental group (video with AI interactive elements). All participants were U.S.-born English-speaking teachers residing in the United States, with one self-reporting birth in Jamaica and two in Canada. The data for 24 users was excluded from the analytic sample due to incomplete assessment data, failing attention checks, or the user stating that they did not complete the learning module. Table 1 presents the demographic characteristics of participants by condition: ## Table 1 Video | Characteristic | Video n ( = 102) | Video with AI Interactives ( n = 94) | |-----------------------------------|-------------------|--------------------------------------| | Age (Mean, SD) | 39.20 (10.40) | 40.20 (11.70) | | Sex (% Female) | 71% | 76% | | Ethnicity (% White) | 75% | 72% | | Student Status (% Yes) | 30% | 23% | | Employment (% Full-Time) | 80% | 82% | As shown in Table 1, the two experimental groups were comparable across demographic variables. Participants in both conditions were predominantly female (70-76%), primarily White (72-75%), and employed full-time (80-82%), with a mean age of approximately 40 years. A small percentage of participants were also current students, with slightly more student participants in the video-only condition (20%) compared to the interactive condition (18%). ## Task and Conditions Participants in both conditions received instructional content on interacting with different personality types and strategies for effective communication across personality differences. The experiment consisted of two conditions. In the Video-Only Condition (Control), participants received instructional videos presenting information about personality types and communication strategies. In the Video with Interactives Condition (Experimental), participants viewed the same instructional video content, supplemented with two types of interactive learning elements: two interactive drag and drop activities and a video talk/text role simulation where participants could interact in real-time with simulated individuals representing different personality types. The interactive elements were designed to reinforce the concepts presented in the videos and provide opportunities for applied practice of key communication strategies. ## Data Collection All data was collected through a survey instrument hosted by Survey Monkey platform. Prior to instruction, all participants completed a multiple-choice pre-test measuring their baseline knowledge of personality types and communication strategies. Following the instructional intervention, participants completed: (1) a different multiple-choice post-test to measure their knowledge of personality types and communications strategies, (2) an engagement measure, and (3) a Net Promoter Score (NPS) measure. Prior to this experiment, both knowledge assessments were pilot tested with different participants and aligned with the content presented in the videos and interactive learning resources. Test scores were calculated as the proportion of correct responses (0-1). ## Analysis To analyze the results, we first conducted preliminary analyses to compare groups at baseline. For the primary research question, the primary analysis employed was an Analysis of Covariance (ANCOVA) to examine differences between conditions while controlling for pre-test scores. Before conducting the ANCOVA, we verified that the data met the assumption of homogeneity of variance using Levene's test. Effect sizes were calculated using partial eta squared (η²) for the ANCOVA results and Hedge's g to provide a standardized measure of the difference between conditions, with appropriate 95% confidence intervals. ## Results ## Descriptive Statistics Table 2 presents the means and standard deviations for all measures across both experimental conditions, providing a foundation for understanding the distribution of scores. **Table 2** Video | Measure | Video ( = 102) n | VideowithInteractives ( n = 94) | |-----------------------|-------------------|----------------------------------| | Pre-Test | 48.41 (21.66) | 49.73 (18.87) | | Post-Test | 76.27 (24.53) | 84.15 (18.80) | | Engagement | 3.50 (1.33) | 3.46 (1.22) | | Net Promoter | 5.39 (2.65) | 6.73 (2.14) | As shown in Table 2, participants in both conditions demonstrated similar pre-test performance. An independent samples t-test was conducted to determine if there were significant differences in pretest knowledge between participants assigned to the video-only and interactive video conditions. The results indicate no statistically significant difference between the groups at pre-test (*t* = -0.74, *p* = 0.47). As show above, post-test scores were notably higher in the interactive condition compared to the video-only condition, suggesting potential benefits of the interactive elements. In relation to engagement and promotion, teachers reported similar engagement ratings across both conditions yet higher promotion of the interactive learning experiences. ## The Impact of AI Interactions on Learning Outcomes First, Levene's test was conducted to assess the assumption of homogeneity of variance for post-test scores between the video-only and interactive video conditions. The non-significant result (*p* = 0.44) indicates that the assumption of equal variances was not violated. This confirms that the variances of post-test scores are sufficiently similar across the two experimental conditions, thus satisfying a key assumption for proceeding with the planned ANCOVA analysis. Second, we conducted an ANCOVA with pre-test scores as the covariate. The results revealed a significant main effect of condition on post-test performance after controlling for pre-test scores, F(3, 192) = 8.20, *p* = 0.008, partial η² = 0.114. This indicates that the interactive video condition produced significantly higher post-test scores compared to the standard video condition, even after accounting for initial performance differences. To quantify the magnitude of this overall effect in standardized terms, Hedge’s *g* was calculated, yielding a value of +0.36 (95% CI [0.12, 0.60]). This represents a moderate effect size, indicating a substantial practical difference between the two instructional approaches and confirming that incorporating interactive elements into video-based instruction has a meaningful, positive impact on learning outcomes. $$ \eta^{2}=0.114 $$ Additionally, in the ANCOVA model, pre-test scores were a significant predictor of post-test performance (p < 0.001), demonstrating that prior knowledge strongly influences learning outcomes. The interaction between condition and pre-test approached significance (*p* = 0.06), suggesting that interactive elements may provide greater benefits for participants with lower prior knowledge. ## Table 3 | Prior Knowledge | Hedge’s g | n | |---------------------|-----------|---| | Low (<50%) | +0.67 | 79| | Moderate (50%) | +0.06 | 58| | High (>50%) | +0.33 | 59| Table 3 demonstrates the differential impact of interactive elements across prior knowledge levels. The effect size analysis reveals that participants with low prior knowledge experienced the largest benefit from interactive elements (*g* = +0.67), representing a moderate-to-large effect. High prior knowledge participants also showed meaningful benefits (*g* = +0.33), while those with moderate prior knowledge demonstrated minimal advantage from interactivity (*g* = +0.06). This pattern aligns with the nearly significant interaction observed in the ANCOVA model, confirming that interactive video elements provide the greatest learning benefits for participants with lower baseline knowledge. ## The Impact of AI Interactions on Engagement After confirming that the assumption of homogeneity of variance was not violated (Levene's test: *p* = 0.359), we conducted an ANCOVA with pretest scores as the covariate to examine the effect of interactive video elements on learner engagement. The results revealed no significant main effect of condition on engagement scores, F(3, 192) = 1.311, *p* = 0.41, partial η² = 0.02, indicating that participants in the interactive video condition did not report significantly different engagement levels compared to those in the video-only condition. Pre-test scores were not a significant predictor of engagement (*p* = 0.34), and the interaction between condition and pre-test was also non-significant (*p* = 0.76), suggesting that the relationship between interactive elements and engagement does not vary based on prior knowledge. $$ \eta^{2}=0.02 $$ $$ (p=0.76) $$ ## The Impact of AI Interactions on Net Promotion Scores After confirming that the assumption of homogeneity of variance was not violated (Levene's test: *p* = 0.52), an ANCOVA was conducted with pre-test scores as the covariate to examine the effect of interactive video elements on Net Promoter Scores. The results revealed a marginally significant main effect of condition (*p* = 0.076), with participants in the interactive condition rating their likelihood to recommend the content 1.34 points higher on average than those in the video-only condition. Pre-test scores significantly predicted Net Promoter Scores (*p* = 0.036), indicating that participants with higher prior knowledge tended to provide more positive recommendations. ## Table 4 Promoter | Condition | Promoters (9-10) | Passives (7-8) | Detractors (0-6) | NPS Score | |--------------------------|-------------------|-----------------|------------------|-----------| | Video-only | 19% | 58% | 23% | -4 | | Interactive | 32% | 55% | 13% | 19 | | Difference | 13% | -3% | -10% | 23 | From an NPS outcome perspective, the interactive condition achieved a substantially higher NPS score (+19) compared to the video-only condition (-4), representing a 23-point difference. This improvement was driven primarily by both an increase in Promoters (+13%) and a decrease in Detractors (-10%). The medium effect size (Hedge's *g* = 0.54) confirms the practical significance of this difference. Notably, the effect sizes for Net Promoter Score were similar across all levels of prior knowledge, with Hedge’s *g* values ranging from 0.49 to 0.63. This means that whether learners started with low, moderate, or high prior knowledge, those in the interactive condition were consistently more likely to recommend the instructional content than those in the video-only condition. This pattern aligns with the non-significant interaction term in the ANCOVA model, indicating that the positive impact of interactive elements on learners’ willingness to recommend the content was robust across different knowledge backgrounds. ## Limitations Several limitations warrant consideration when interpreting this study's results. The moderate sample size (*N* = 220) limited our ability to detect subtle effects, particularly for our NPS measures that approached but didn't reach statistical significance. In addition, the focus on personality types with predominantly White female teachers restricts generalizability. Additionally, by measuring only immediate learning outcomes without longitudinal follow-up, we cannot determine whether benefits persist over time or transfer to real-world applications. Finally, our design examined a combination of interactive elements (drag-and-drop activities and AI video-talk simulations) rather than isolating their individual effects, preventing identification of which specific features contributed most to learning gains. Future research should address these limitations through larger, more diverse samples, longitudinal designs, and factorial experiments that separate interactive components. ## Discussion This study investigated the efficacy of incorporating AI-enhanced interactive elements into video-based instruction, with a particular focus on learning outcomes and engagement. The findings provide empirical support for the integration of AI interactive components in digital learning environments and offer several important insights for both researchers and practitioners. Our first research question examined how interactive elements affected learning outcomes compared to video-only instruction. The results demonstrated a significant positive effect of interactive elements on post-test performance, with participants in the interactive condition scoring approximately 8 percentage points higher than those in the video-only condition after controlling for pre-test knowledge. The moderate effect size (Hedge's g = 0.36) indicates not only statistical significance but practical importance as well. These findings align with the theoretical principles outlined in our background, supporting the cognitive science literature suggesting that active engagement creates stronger neural pathways and more durable learning (Brod et al., 2016; Zhou et al., 2019). From a practical perspective, the overall effect size of +0.36 is particularly noteworthy when compared to the effects achieved by tutoring interventions, which are widely regarded as one of the most impactful educational strategies. Recent meta-analyses of tutoring programs have reported pooled effect sizes ranging from +0.3 to +0.4 for small-scale, high-dosage interventions (Dietrichson et al., 2017; Nickow et al., 2020). These effects are considered substantial, equivalent to several months of additional learning. The similarity in magnitude between the effect size observed in this study and those associated with tutoring underscores the potential of AI-enhanced interactive video instruction to rival traditional tutoring in improving learning outcomes, especially in scalable digital formats. Particularly noteworthy was the differential impact of interactive elements based on prior knowledge levels. Participants with low prior knowledge experienced substantially greater benefits (*g* = +0.67) compared to those with moderate prior knowledge. This pattern aligns with the expertise reversal effect described by Kalyuga (2007) and Sweller et al. (2019), where novices benefit more from structured interactive guidance than do more knowledgeable learners. The interactive elements likely provided scaffolding that helped novices manage cognitive load while building initial mental models of the content—precisely the support they needed most. Interestingly, while the interactive elements significantly improved learning outcomes, they did not produce corresponding increases in self-reported engagement. This unexpected finding suggests that subjective perceptions of engagement may not always align with objective learning benefits. However, participants in the interactive condition were substantially more likely to recommend the learning experience to others, as evidenced by a 23-point improvement in Net Promoter Score. This discrepancy between reported engagement and recommendation likelihood merits further investigation, as it suggests that interactive elements may enhance perceived value even when participants don't explicitly recognize higher engagement. The robust NPS improvements across all knowledge levels indicate that interactive elements enhanced the perceived value of the learning experience regardless of prior familiarity with the content. This finding has important implications for instructional design, suggesting that interactive elements can improve learner satisfaction and potentially increase voluntary participation in educational opportunities even when their learning impact varies. From a practical perspective, these results provide compelling evidence that incorporating AI-enhanced interactive elements into instructional videos represents an effective strategy for improving learning outcomes, particularly for novice learners. The specific interactive components used in this study—drag and drop activities and simulated video talk interactions—provided opportunities for both private cognitive engagement (through self-assessment and application) and public meaning-making (through simulated social interaction). This combination addresses both key aspects of active learning identified in the literature (Chi & Wylie, 2014; Pan & Rickard, 2018). Educational designers and content developers should consider the significant advantages of incorporating similar interactive elements into digital learning experiences, with particular attention to supporting novice learners. The differential impact based on prior knowledge also suggests potential value in adaptive systems that could adjust the level of interactivity based on learner expertise. Future research should explore a wider variety of interactive element types, examine long-term retention of knowledge, and investigate how interactivity might be optimally tailored to different content domains and learner characteristics. Additionally, more nuanced measures of gagement could help clarify the relationship between subjective gagement perceptions and objective learning benefits. Nevertheless, this study provides robust evidence that AI-enhanced interactive elements can significantly improve learning outcomes in digital educational environments, particularly for those with limited prior knowledge. ## References - Brod, G., Werkle-Bergner, M., & Shing, Y. L. (2016). The influence of prior knowledge on memory: A developmental cognitive neuroscience perspective. *Frontiers in Behavioral Neuroscience, 10*, 111. - Chan, K. Y. G., & Blikstein, P. (2018). Exploring problem-based collaborative learning in undergraduate engineering education: Understanding student benefits and challenges. *Journal of Engineering* *Education, 107*(4), 604-630. - Chen, J., Wang, M., Kirschner, P. A., & Tsai, C. C. (2018). The role of collaboration, computer use, learning environments, and supporting strategies in CSCL: A meta-analysis. *Review of Educational Research,* *88*(6), 799-843. - Chi, M. T., & Wylie, R. (2014). The ICAP framework: Linking cognitive engagement to active learning outcomes. *Educational* *Psychologist*, 49(4), 219-243. - Dietrichson, J., Bøg, M., Filges, T., & Jørgensen, A. M. K. (2017). Academic interventions for elementary and middle school students with low socioeconomic status: A systematic review and meta-analysis. *Review of Educational Research, 87*(2), 243–282. - Freeman, S., Eddy, S. L., McDonough, M., Smith, M. K., Okoroafor, N., Jordt, H., & Wenderoth, M. P. (2014). Active learning increases student performance in science, engineering, and mathematics. *Proceedings of the National Academy of Sciences*, 111(23), 8410-8415. - Kalyuga, S. (2007). Expertise reversal effect and its implications for learner-tailored instruction. *Educational Psychology Review*, 19(4), 509-539. - McElhaney, K. W., Chang, H. Y., Chiu, J. L., & Linn, M. C. (2015). Evidence for effective uses of dynamic visualisations in science curriculum materials. *Studies in Science Education*, 51(1), 49-85. - Nickow, A. J., Oreopoulos, P., & Quan, V. (2020). The impressive effects of tutoring on PreK-12 learning: A systematic review and meta-analysis of the experimental evidence. *EdWorkingPaper No.* *20-267*. Annenberg Institute at Brown University. - Pan, S. C., & Rickard, T. C. (2018). Transfer of test-enhanced learning: Meta-analytic review and synthesis. *Psychological Bulletin*, 144(7), 710-756. - Renkl, A. (2014). Toward an instructionally oriented theory of example‐based learning. *Cognitive Science*, 38(1), 1-37. - Sweller, J., van Merriënboer, J. J. G., & Paas, F. (2019). Cognitive architecture and instructional design: 20 years later. *Educational* *Psychology Review*, 31(2), 261-292. - Theobald, E. J., Hill, M. J., Tran, E., Agrawal, S., Arroyo, E. N., Behling, S., ... & Freeman, S. (2020). Active learning narrows achievement gaps for underrepresented students in undergraduate science, technology, engineering, and math. *Proceedings of the National* *Academy of Sciences*, 117(12), 6476-6483. - VanLehn, K. (2011). The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. *Educational* *Psychologist*, 46(4), 197-221. - Zingaro, D., & Porter, L. (2016). Impact of student achievement goals on CS1 outcomes. In *Proceedings of the 47th ACM Technical* *Symposium on Computing Science Education (SIGCSE '16)*, 279-284. ## Author’s Background Dr. Puzio is the Director of Research at Creatium. He is a learning scientist who received a Ph.D. in Learning, Teaching, and Diversity from Vanderbilt University (2012), a Master's in Education from DePaul University (2002), and a bachelor's degree from the University of Notre Dame. He was a certified Language Arts teacher in New Zealand and the United States (2001-2007). He was an Assistant Professor and then Associate Professor with tenure at Washington State University in the Department of Teaching and Learning. His current research interests include personalized learning and artificial intelligence. Dr. Puzio has served as Principal Investigator on over $5 million in grants from the Institutes of Health and the National Science Foundation. He was a recipient of the Young Scholar Award (2014) from the International Literacy Association.