Updated Research_learning_study_5.0.pdf

Quantifying the Learning Benefits of AI-Enhanced Video Instruction
Kelly Puzio, PhD

Abstract

This study investigated the impact of AI-enhanced interactive elements
in video-based instruction on learning outcomes and engagement. In a
between-subjects experiment, 220 teachers were randomly assigned to
either video-only instruction or video instruction with interactive
elements (AI created drag-and-drop activities and AI-simulated video
conversations) focused on personality types and communication
strategies. Results revealed that participants in the interactive condition
demonstrated significantly higher post-test scores (p = 0.008, Hedge's g
= +0.36) compared to the video-only condition, with particularly strong
benefits for novice learners (g = +0.67). Notably, this improvement is
comparable to the learning gains typically observed in tutoring
programs, underscoring the practical significance of integrating
interactive AI elements into video-based instruction. While
self-reported engagement did not differ significantly between
conditions, the AI interactive group showed substantially higher Net
Promoter Scores (+19 vs. -4), indicating greater willingness to
recommend the learning solution. These findings suggest that
AI-enhanced interactive elements can significantly improve learning
outcomes, particularly for individuals with lower prior knowledge,
while also enhancing the perceived value of educational content. Educational designers should consider incorporating such interactive
elements into digital learning experiences, especially when targeting
learners with limited domain knowledge.

Background

The landscape of education continues to evolve as advances in
technology transform both teaching and learning. This evolution is
particularly evident in the growing emphasis on active approaches to
learning, which have consistently demonstrated superiority over passive
learning methods across multiple disciplines. Active learning refers to
instructional approaches that engage students through meaningful
activities, conversations, and negotiations of meaning rather than
passive (sit and get) consumption of information (Chi & Wylie, 2014;
Theobald et al., 2020). The effectiveness of these approaches is
well-documented, with research showing that active learning can reduce
course failure rates by 55% and improve achievement by an average of
0.47 standard deviations (Freeman et al., 2014).

The cognitive science underlying active learning reveals why these
approaches are so effective. When students actively engage with
learning materials, multiple brain regions activate simultaneously,
including areas associated with memory formation, emotional
processing, and executive function (Brod et al., 2016). This
simultaneous activation creates stronger neural pathways, facilitating
better long-term retention and knowledge transfer to new contexts
(Zhou et al., 2019). In contrast, passive learning primarily activates
basic sensory processing regions, resulting in weaker neural
engagement and less durable learning outcomes.

Active learning operates through both private (cognitive) and public
(social) processes. Private meaning-making occurs through activities
like retrieval practice, self-assessment, and metacognitive reflection,
which have been shown to significantly improve learning outcomes in
a wide variety of disciplines (Pan & Rickard, 2018; Zingaro & Porter,
2016). Equally important are the social aspects of learning, where
discussions and collaborative activities enhance both technical
competency and professional skill development (Chan & Blikstein,
2018). The synergy between private and public forms of active learning
creates a powerful learning environment that consistently outperforms
traditional methods.

When embedded inside digital learning experiences, interactive
elements represent a promising approach to implementing active
learning principles at scale. These interactive components can range
from adaptive quizzes and concept mapping activities to simulated
conversations and collaborative problem-solving tasks. By
categorizing such elements, educators can create opportunities for both
private cognitive engagement and public meaning-making. Research in
computer-supported collaborative learning has shown significant
improvements in knowledge acquisition (d = +0.42) and
problem-solving (d = +0.64) skills (Chen et al., 2018).

Research consistently demonstrates that interactive learning
environments provide particularly strong benefits for novice learners
compared to more experienced ones (Kalyuga, 2007; Sweller et al.,
2019). This “expertise reversal effect” occurs because novices lack
well-developed mental models and benefit from the scaffolded guidance
and immediate feedback that interactive elements provide. Novices
show significantly improved outcomes when learning with interactive
components that manage cognitive load through worked examples
(Renkl, 2014), visualization aids (McElhaney et al., 2015), and adaptive
feedback systems (VanLehn, 2011). These approaches effectively bridge
the gap between novices’ limited prior knowledge and complex learning
demands, making interactive learning especially valuable for beginners
in any domain.

Despite the compelling evidence supporting active learning in general,
there remains a gap in understanding how AI tools can support private
and public forms of active learning. While artificial intelligence
technologies offer unprecedented opportunities for creating interactive,
adaptive, and personalized learning experiences, empirical research
examining their specific impacts on learning outcomes and engagement
remains limited. Although research has established the general
effectiveness of active learning principles, we lack sufficient evidence
about specific AI-powered interactive features that effectively promote
learning and engagement for different types of content and learner
populations. Additionally, most studies have focused on traditional
classroom implementations of active learning rather than AI-enhanced
digital learning experiences, leaving crucial questions about how AI can
best be leveraged to implement active learning principles in online
instructional materials.

This study aims to address these gaps by examining how different
interactive elements affect learning outcomes and engagement when
teaching about personality types and communication strategies. By
comparing video-only instruction to video instruction enhanced with
one cognitive form of active learning and one social form of active
learning, we seek to contribute to the growing literature on effective
implementation of AI-supported active learning principles in digital
learning environments.

Research Methods

This investigation focused on examining the effectiveness of
AI-interactive elements in video-based instruction on learning outcomes
and learner engagement. We sought to understand how incorporating
various interactive components affects knowledge acquisition and
self-reported engagement when learning about interpersonal
communication strategies. Specifically, the following questions guided
this investigation:

1. To what extent does the addition of interactive elements (drag and
   drop activities; AI-simulated real-time video interactions) to
   video-based instruction affect learning outcomes compared to
   video-only instruction?

2. To what extent does the inclusion of interactive elements in
   instructional videos influence learner engagement or promotion
   compared to traditional video-only instruction?

3. To what extent do participant variables (e.g., pretest) influence
   learning, engagement, or Net Promoter outcomes?

Study Design and Participants

This study employed a between-subjects experimental design with 220
teachers recruited through Prolific, randomly assigned to either a
control group (video-only instruction) or experimental group (video
with AI interactive elements). All participants were U.S.-born
English-speaking teachers residing in the United States, with one
self-reporting birth in Jamaica and two in Canada. The data for 24 users
was excluded from the analytic sample due to incomplete assessment
data, failing attention checks, or the user stating that they did not
complete the learning module. Table 1 presents the demographic
characteristics of participants by condition:

Table 1

Video

As shown in Table 1, the two experimental groups were comparable
across demographic variables. Participants in both conditions were
predominantly female (70-76%), primarily White (72-75%), and
employed full-time (80-82%), with a mean age of approximately 40
years. A small percentage of participants were also current students,
with slightly more student participants in the video-only condition
(20%) compared to the interactive condition (18%).

Task and Conditions

Participants in both conditions received instructional content on
interacting with different personality types and strategies for effective
communication across personality differences. The experiment
consisted of two conditions. In the Video-Only Condition (Control),
participants received instructional videos presenting information about
personality types and communication strategies. In the Video with
Interactives Condition (Experimental), participants viewed the same
instructional video content, supplemented with two types of interactive
learning elements: two interactive drag and drop activities and a video
talk/text role simulation where participants could interact in real-time
with simulated individuals representing different personality types. The
interactive elements were designed to reinforce the concepts presented
in the videos and provide opportunities for applied practice of key
communication strategies.

Data Collection

All data was collected through a survey instrument hosted by Survey
Monkey platform. Prior to instruction, all participants completed a
multiple-choice pre-test measuring their baseline knowledge of
personality types and communication strategies. Following the
instructional intervention, participants completed: (1) a different
multiple-choice post-test to measure their knowledge of personality
types and communications strategies, (2) an engagement measure, and
(3) a Net Promoter Score (NPS) measure. Prior to this experiment, both
knowledge assessments were pilot tested with different participants and
aligned with the content presented in the videos and interactive learning
resources. Test scores were calculated as the proportion of correct
responses (0-1).

Analysis

To analyze the results, we first conducted preliminary analyses to
compare groups at baseline. For the primary research question, the
primary analysis employed was an Analysis of Covariance (ANCOVA)
to examine differences between conditions while controlling for pre-test
scores. Before conducting the ANCOVA, we verified that the data met
the assumption of homogeneity of variance using Levene's test. Effect
sizes were calculated using partial eta squared (η²) for the ANCOVA
results and Hedge's g to provide a standardized measure of the
difference between conditions, with appropriate 95% confidence
intervals.

Results

Descriptive Statistics

Table 2 presents the means and standard deviations for all measures
across both experimental conditions, providing a foundation for
understanding the distribution of scores.

Table 2

Video

As shown in Table 2, participants in both conditions demonstrated
similar pre-test performance. An independent samples t-test was
conducted to determine if there were significant differences in pretest
knowledge between participants assigned to the video-only and
interactive video conditions. The results indicate no statistically
significant difference between the groups at pre-test (t = -0.74, p = 0.47). As show above, post-test scores were notably higher in the
interactive condition compared to the video-only condition, suggesting
potential benefits of the interactive elements. In relation to
engagement and promotion, teachers reported similar engagement
ratings across both conditions yet higher promotion of the interactive
learning experiences.

The Impact of AI Interactions on Learning Outcomes

First, Levene's test was conducted to assess the assumption of
homogeneity of variance for post-test scores between the video-only
and interactive video conditions. The non-significant result (p = 0.44)
indicates that the assumption of equal variances was not violated. This
confirms that the variances of post-test scores are sufficiently similar
across the two experimental conditions, thus satisfying a key
assumption for proceeding with the planned ANCOVA analysis.

Second, we conducted an ANCOVA with pre-test scores as the
covariate. The results revealed a significant main effect of condition on
post-test performance after controlling for pre-test scores, F(3, 192) =
8.20, p = 0.008, partial η² = 0.114. This indicates that the interactive
video condition produced significantly higher post-test scores compared
to the standard video condition, even after accounting for initial
performance differences. To quantify the magnitude of this overall
effect in standardized terms, Hedge’s g was calculated, yielding a value
of +0.36 (95% CI [0.12, 0.60]). This represents a moderate effect size,
indicating a substantial practical difference between the two
instructional approaches and confirming that incorporating interactive
elements into video-based instruction has a meaningful, positive impact
on learning outcomes.

$$ \eta^{2}=0.114 $$

Additionally, in the ANCOVA model, pre-test scores were a significant
predictor of post-test performance (p < 0.001), demonstrating that prior
knowledge strongly influences learning outcomes. The interaction
between condition and pre-test approached significance (p = 0.06),
suggesting that interactive elements may provide greater benefits for
participants with lower prior knowledge.

Table 3

Table 3 demonstrates the differential impact of interactive elements
across prior knowledge levels. The effect size analysis reveals that
participants with low prior knowledge experienced the largest benefit
from interactive elements (g = +0.67), representing a moderate-to-large
effect. High prior knowledge participants also showed meaningful
benefits (g = +0.33), while those with moderate prior knowledge
demonstrated minimal advantage from interactivity (g = +0.06). This
pattern aligns with the nearly significant interaction observed in the
ANCOVA model, confirming that interactive video elements provide
the greatest learning benefits for participants with lower baseline
knowledge.

The Impact of AI Interactions on Engagement

After confirming that the assumption of homogeneity of variance was
not violated (Levene's test: p = 0.359), we conducted an ANCOVA with
pretest scores as the covariate to examine the effect of interactive
video elements on learner engagement. The results revealed no significant
main effect of condition on engagement scores, F(3, 192) = 1.311, p =
0.41, partial η² = 0.02, indicating that participants in the
interactive video condition did not report significantly different
engagement levels compared to those in the video-only condition. Pre-test
scores were not a significant predictor of engagement (p = 0.34),
and the interaction between condition and pre-test was also non-significant
(p = 0.76), suggesting that the relationship between interactive
elements and engagement does not vary based on prior knowledge.

$$ \eta^{2}=0.02 $$

$$ (p=0.76) $$

The Impact of AI Interactions on Net Promotion Scores

After confirming that the assumption of homogeneity of variance was
not violated (Levene's test: p = 0.52), an ANCOVA was conducted with
pre-test scores as the covariate to examine the effect of interactive
video elements on Net Promoter Scores. The results revealed a marginally
significant main effect of condition (p = 0.076), with participants in the
interactive condition rating their likelihood to recommend the content
1.34 points higher on average than those in the video-only condition.
Pre-test scores significantly predicted Net Promoter Scores (p = 0.036),
indicating that participants with higher prior knowledge tended to
provide more positive recommendations.

Table 4

Promoter

From an NPS outcome perspective, the interactive condition achieved a
substantially higher NPS score (+19) compared to the video-only
condition (-4), representing a 23-point difference. This improvement
was driven primarily by both an increase in Promoters (+13%) and a
decrease in Detractors (-10%). The medium effect size (Hedge's g =
0.54) confirms the practical significance of this difference. Notably, the
effect sizes for Net Promoter Score were similar across all levels of
prior knowledge, with Hedge’s g values ranging from 0.49 to 0.63. This
means that whether learners started with low, moderate, or high prior
knowledge, those in the interactive condition were consistently more
likely to recommend the instructional content than those in the
video-only condition. This pattern aligns with the non-significant
interaction term in the ANCOVA model, indicating that the positive
impact of interactive elements on learners’ willingness to recommend
the content was robust across different knowledge backgrounds.

Limitations

Several limitations warrant consideration when interpreting this study's
results. The moderate sample size (N = 220) limited our ability to
detect subtle effects, particularly for our NPS measures that approached but
didn't reach statistical significance. In addition, the focus on personality
types with predominantly White female teachers restricts
generalizability. Additionally, by measuring only immediate learning
outcomes without longitudinal follow-up, we cannot determine whether
benefits persist over time or transfer to real-world applications. Finally,
our design examined a combination of interactive elements
(drag-and-drop activities and AI video-talk simulations) rather than
isolating their individual effects, preventing identification of which
specific features contributed most to learning gains. Future research
should address these limitations through larger, more diverse samples,
longitudinal designs, and factorial experiments that separate interactive
components.

Discussion

This study investigated the efficacy of incorporating AI-enhanced
interactive elements into video-based instruction, with a particular focus
on learning outcomes and engagement. The findings provide empirical
support for the integration of AI interactive components in digital
learning environments and offer several important insights for both
researchers and practitioners.

Our first research question examined how interactive elements affected
learning outcomes compared to video-only instruction. The results
demonstrated a significant positive effect of interactive elements on
post-test performance, with participants in the interactive condition
scoring approximately 8 percentage points higher than those in the
video-only condition after controlling for pre-test knowledge. The
moderate effect size (Hedge's g = 0.36) indicates not only statistical
significance but practical importance as well. These findings align with
the theoretical principles outlined in our background, supporting the
cognitive science literature suggesting that active engagement creates
stronger neural pathways and more durable learning (Brod et al., 2016;
Zhou et al., 2019).

From a practical perspective, the overall effect size of +0.36 is
particularly noteworthy when compared to the effects achieved by
tutoring interventions, which are widely regarded as one of the most
impactful educational strategies. Recent meta-analyses of tutoring
programs have reported pooled effect sizes ranging from +0.3 to +0.4
for small-scale, high-dosage interventions (Dietrichson et al., 2017;
Nickow et al., 2020). These effects are considered substantial,
equivalent to several months of additional learning. The similarity in
magnitude between the effect size observed in this study and those
associated with tutoring underscores the potential of AI-enhanced
interactive video instruction to rival traditional tutoring in improving
learning outcomes, especially in scalable digital formats.

Particularly noteworthy was the differential impact of interactive
elements based on prior knowledge levels. Participants with low prior
knowledge experienced substantially greater benefits (g = +0.67)
compared to those with moderate prior knowledge. This pattern aligns
with the expertise reversal effect described by Kalyuga (2007) and
Sweller et al. (2019), where novices benefit more from structured
interactive guidance than do more knowledgeable learners. The
interactive elements likely provided scaffolding that helped novices
manage cognitive load while building initial mental models of the
content—precisely the support they needed most.

Interestingly, while the interactive elements significantly improved
learning outcomes, they did not produce corresponding increases in
self-reported engagement. This unexpected finding suggests that
subjective perceptions of engagement may not always align with
objective learning benefits. However, participants in the interactive
condition were substantially more likely to recommend the learning
experience to others, as evidenced by a 23-point improvement in
Net Promoter Score. This discrepancy between reported engagement and
recommendation likelihood merits further investigation, as it suggests
that interactive elements may enhance perceived value even when
participants don't explicitly recognize higher engagement.

The robust NPS improvements across all knowledge levels indicate that
interactive elements enhanced the perceived value of the learning
experience regardless of prior familiarity with the content. This finding
has important implications for instructional design, suggesting that
interactive elements can improve learner satisfaction and potentially
increase voluntary participation in educational opportunities even when
their learning impact varies.

From a practical perspective, these results provide compelling evidence
that incorporating AI-enhanced interactive elements into instructional
videos represents an effective strategy for improving learning outcomes,
particularly for novice learners. The specific interactive components
used in this study—drag and drop activities and simulated video talk
interactions—provided opportunities for both private cognitive
engagement (through self-assessment and application) and public
meaning-making (through simulated social interaction). This
combination addresses both key aspects of active learning identified in
the literature (Chi & Wylie, 2014; Pan & Rickard, 2018).

Educational designers and content developers should consider the
significant advantages of incorporating similar interactive elements into
digital learning experiences, with particular attention to supporting
novice learners. The differential impact based on prior knowledge also
suggests potential value in adaptive systems that could adjust the
level of interactivity based on learner expertise.

Future research should explore a wider variety of interactive element
types, examine long-term retention of knowledge, and investigate how
interactivity might be optimally tailored to different content domains
and learner characteristics. Additionally, more nuanced measures of
gagement could help clarify the relationship between subjective
gagement perceptions and objective learning benefits. Nevertheless,
this study provides robust evidence that AI-enhanced interactive
elements can significantly improve learning outcomes in digital
educational environments, particularly for those with limited prior
knowledge.

References

• Brod, G., Werkle-Bergner, M., & Shing, Y. L. (2016). The influence of
  prior knowledge on memory: A developmental cognitive neuroscience
  perspective. Frontiers in Behavioral Neuroscience, 10, 111.
• Chan, K. Y. G., & Blikstein, P. (2018). Exploring problem-based
  collaborative learning in undergraduate engineering education:
  Understanding student benefits and challenges. Journal of Engineering
  Education, 107(4), 604-630.
• Chen, J., Wang, M., Kirschner, P. A., & Tsai, C. C. (2018). The role of
  collaboration, computer use, learning environments, and supporting
  strategies in CSCL: A meta-analysis. Review of Educational Research,
  88(6), 799-843.
• Chi, M. T., & Wylie, R. (2014). The ICAP framework: Linking
  cognitive engagement to active learning outcomes. Educational
  Psychologist, 49(4), 219-243.
• Dietrichson, J., Bøg, M., Filges, T., & Jørgensen, A. M. K. (2017).
  Academic interventions for elementary and middle school students with
  low socioeconomic status: A systematic review and
  meta-analysis. Review of Educational Research, 87(2), 243–282.
• Freeman, S., Eddy, S. L., McDonough, M., Smith, M. K., Okoroafor,
  N., Jordt, H., & Wenderoth, M. P. (2014). Active learning increases
  student performance in science, engineering, and mathematics.
  Proceedings of the National Academy of Sciences, 111(23), 8410-8415.
• Kalyuga, S. (2007). Expertise reversal effect and its implications for
  learner-tailored instruction. Educational Psychology Review, 19(4),
  509-539.
• McElhaney, K. W., Chang, H. Y., Chiu, J. L., & Linn, M. C. (2015).
  Evidence for effective uses of dynamic visualisations in science
  curriculum materials. Studies in Science Education, 51(1), 49-85.
• Nickow, A. J., Oreopoulos, P., & Quan, V. (2020). The impressive
  effects of tutoring on PreK-12 learning: A systematic review and
  meta-analysis of the experimental evidence. EdWorkingPaper No.
  20-267. Annenberg Institute at Brown University.
• Pan, S. C., & Rickard, T. C. (2018). Transfer of test-enhanced learning:
  Meta-analytic review and synthesis. Psychological Bulletin, 144(7),
  710-756.
• Renkl, A. (2014). Toward an instructionally oriented theory of
  example‐based learning. Cognitive Science, 38(1), 1-37.
• Sweller, J., van Merriënboer, J. J. G., & Paas, F. (2019). Cognitive
  architecture and instructional design: 20 years later. Educational
  Psychology Review, 31(2), 261-292.
• Theobald, E. J., Hill, M. J., Tran, E., Agrawal, S., Arroyo, E. N.,
  Behling, S., ... & Freeman, S. (2020). Active learning narrows
  achievement gaps for underrepresented students in undergraduate
  science, technology, engineering, and math. Proceedings of the National
  Academy of Sciences, 117(12), 6476-6483.
• VanLehn, K. (2011). The relative effectiveness of human tutoring,
  intelligent tutoring systems, and other tutoring systems. Educational
  Psychologist, 46(4), 197-221.
• Zingaro, D., & Porter, L. (2016). Impact of student achievement goals
  on CS1 outcomes. In Proceedings of the 47th ACM Technical
  Symposium on Computing Science Education (SIGCSE '16), 279-284.

Author’s Background

Dr. Puzio is the Director of Research at Creatium. He is a
learning scientist who received a Ph.D. in Learning, Teaching, and
Diversity from Vanderbilt University (2012), a Master's in Education
from DePaul University (2002), and a bachelor's degree from the
University of Notre Dame. He was a certified Language Arts teacher in
New Zealand and the United States (2001-2007). He was an Assistant
Professor and then Associate Professor with tenure at Washington State
University in the Department of Teaching and Learning. His current
research interests include personalized learning and artificial intelligence.

Dr. Puzio has served as Principal Investigator on over $5 million in
grants from the Institutes of Health and the National Science
Foundation. He was a recipient of the Young Scholar Award (2014) from
the International Literacy Association.