This study explores the viewing patterns of 17 instructional videos in both a traditional and flipped classroom environment by 732 business students taking an IT-related module. While previous work has concentrated mainly on outputs (e.g. student satisfaction/results), this study focuses on how the nature of students’ interactions with videos can be determined through a deep analysis of analytics data. The main findings show that there were less interactions with the instructional videos in the flipped classroom environment compared to the traditional environment, and that videos were used more as a revision aid prior to exams (in both environments) than as an ongoing support to develop skills during term. Implications of this study include the need for regular monitoring of how instructional videos are being used during term and the importance of undertaking a deeper analysis of analytics data as the initial summary data may be misleading.
Este estudio explora los patrones de visualización de 17 videos instruccionales en un contexto tradicional de aprendizaje y en un contexto de clase invertida por 723 estudiantes de negocios cursando un módulo relacionado con IT. Mientras investigaciones anteriores se centraban en resultados (por ejemplo, satisfacción/resultados de los alumnos), en este estudio el foco recae en cómo la naturaleza de las interacciones de los estudiantes con los videos puede determinarse a través de un análisis profundo de los datos analíticos. Nuestros resultados muestran menor interacción con los videos instruccionales en el contexto de clase invertida que en el de clase tradicional. Asimismo, muestran un mayor uso de los videos como material de revisión antes del examen en ambos contextos de aprendizaje que un uso como soporte pedagógico para el desarrollo de habilidades a lo largo del semestre. Las implicaciones de este estudio apuntan hacia la necesidad de una monitorización habitual del uso de los videos instruccionales durante el semestre. Del mismo modo, se constata la importancia de llevar a cabo un análisis más profundo de los datos analísticos puesto que el sumario de datos iniciales en muchos casos puede ser engañoso.
Aquest estudi explora els patrons de visualització de 17 vídeos instruccionals en un context tradicional d´aprenentatge i en un context de classe invertida per 723 estudiants de negocis cursant un mòdul relacionat amb IT. Mentre recerques anteriors s’han centrat principalment en resultats (per exemple, satisfacció/resultats dels alumnes), en aquest estudi l’enfocament recau en com la natura de les interaccions del estudiants amb el vídeos por determinar-se mitjançant un anàlisi profund de les dades analítiques. Els nostres resultats mostren menys interacció amb els vídeos instruccionals en el context de la classe invertida que en el de la classe tradicional. De la mateixa manera, mostren un ús més elevat dels vídeos com a material de revisió abans de l’examen durant el semestre en ambdós contexts d’aprenentatge que no pas un ús com a suport pedagògic pel desenvolupament d’habilitats durant el semestre. Les implicacions d’aquest estudi suggereixen la necessitat de monitoritzar de forma regular l’ús de vídeos instruccionals durant el semestre, així com la importància de realitzar un anàlisi més profund de les dades analítiques ja que el sumari de dades inicial en molts casos pot resultar enganyós.
Educational videos are becoming increasing used as learning objects, embedded in VLE courses and increase students’ perceptions of efficacy. They (educational videos) provide the ability to flip classroom activities so that content is transmitted prior to class allowing more time for interactions within the classroom. Existing research on flipped classrooms has typically focused on outputs and perceptions of the process, yielding mixed results; rather than on the process itself.
Rather than examine student perceptions of the flipped classroom process, this paper examined YouTubeAnalytics data in order to better understand differences in students' engagement with videos in flipped and non-flipped contexts. Students accessed videos more in the non-flipped context where videos were supported by a traditional lecture. In both flipped and non-flipped contexts the videos were used more as a revision aid later in the course than for initial learning. More difficult material was viewed most during the initial learning period and was watched for longer.
While the flipped classroom is seen as a way of moving content delivery away from the classroom, there is value in traditional lectures to support and contextualise online material. There is a need to go beyond the number of times videos are viewed to examine when and for how long such material is accessed, and the relative degree of difficulty involved, to gain a better understanding of how students use such online resources.
In recent years, there has been a growing interest and awareness of learning analytics and educational data mining in the higher education sector (
The literature on learning analytics in higher education predominantly focuses on a single teaching treatment (e.g. in a flipped classroom or traditional classroom environment). Studies carried out to date are somewhat limited in their usage of data analytics, particularly with regards to exploiting data available from sources such as
This section presents a brief overview of some research on the use of video instruction in higher education. It then explores student satisfaction and performance in the ‘flipped’ classroom and concludes with a short discussion on the role of analytics in identifying usage patterns relating to online videos.
In recent years, media, and video in particular, has dramatically changed the educational landscape. The effective use of video can transform the way we teach, learn, study, communicate, and work. The use of video as a learning medium has a long tradition in education, but its availability has radically changed during the last years (
Thanks to Virtual Learning Environments (VLEs), disseminating videos or URL links to videos has become much simpler. Links to
Buzzetto-More’s (
Flipped classroom approach has become a popular pedagogy in many education institutes around the world. The basic notion of flipped classroom approach is to deliver the instructor’s lectures before class through online videos, in order to free-up the in-class time for active learning and problem-solving activities (
The concept of a flipped classroom is in stark contrast to the notion of requiring students to attend so that material can be disseminated. In a flipped environment, teachers typically record short screencast videos of their lectures or record podcasts and then upload them to a virtual learning environment (VLE). In some cases, massive open online course (MOOC) materials have been used to support flipped classroom environments (
Much of the literature carried out to date has focused more on outputs (such as student performance) as well as perceptions (such as students’ satisfaction with a flipped classroom approach) and studies have reported mixed results. For example, (
However, some scholars have found more mixed results. For example, (
With the rise in online and blended learning, massive open and online courses and flipped classroom approaches, the use of video has seen a steady increase (
Several studies have explored when and how students use online instructional videos. (
(
Analysing video-watching patterns provides a unique opportunity to appreciate how, and if, students learn more effectively via video. (
Despite the growing literature on the benefits of video-based instruction and the flipped classroom approach, there is limited research on how instructors can use
Do students view videos more when used in a flipped classroom, or when they are a supplementary resource to a traditional lecture?
To what extent do students use videos as a tool for (a) initial learning (b) revision?
Does the difficulty of the material affect video viewing patterns?
How much [what proportion] of the videos are watched?
Does the difficulty of the material affect how much [what proportion] of the videos are watched?
To what extent do students watch a series of videos on a topic?
This section describes in detail the two teaching treatments that were used, in successive years, for a module. It also describes the characteristics of the instructional videos, the data collected, and the rationale for the data analysis methods employed. Expanding on studies carried out to date and leveraging the growing body of data that can be captured via analytics, this study seeks to investigate and probe a milieu of behaviours and usage patterns – for example, exploring the significance of the difficulty of the video content.
A series of instructional videos were developed specifically for a module entitled ‘Business Information Management’ that covered both Information Systems theory and practical spreadsheet skills using Microsoft Excel. It was a mandatory module for first-year undergraduate students taking a Bachelor of Business Studies degree. The module assumed no prior knowledge of the material though some students had previously used spreadsheets. Students attended two one-hour lectures per week and five one-hour computer labs over the semester. In addition to an end-of-term theory exam, there was also a one-hour end-of-term practical spreadsheet exam.
Before each lab session, students were provided with the following resources:
links to the relevant YouTube videos
a spreadsheet containing data required for the weekly topic
a worksheet with questions that required students to apply concepts covered by the video material, by modifying the spreadsheet data
Additional support was provided in the five weekly computer labs during which students could work on their worksheet, ask questions of the teaching assistant, and receive individual feedback. Teaching assistants did not provide an outline of the concepts required, as students were expected to have engaged with this material prior to the lab. Students were not permitted to watch the videos during the labs. At the end of each week, an ‘answer’ spreadsheet was posted online. No marks for attendance were allocated and the practical component was assessed using only an end-of-term practical exam.
While the same resources were issued to all students for both years of this study, a different teaching treatment was used. In the traditional teaching environment (referred to as Treatment 1 in the tables), students attended a lecture the week before they attended the lab and the supplementary videos covered material that related to the lecture material. The following year, a flipped classroom approach was used, whereby students only had access to the videos and no lecture was provided before the lab (referred to as Treatment 2 in the tables).
The videos covered five topics and demonstrated how certain tasks were implemented using
The videos were intentionally unlisted on YouTube with links provided only through the university virtual learning environment (VLE) to ensure only course participants accessed the videos. The number of hits verified that views were only logged while the courses were running. As outlined in
Data on video usage is collected automatically by
Number of views
Estimated number of minutes watched
Average view duration (in minutes)
Average percentage viewed
Unique views (Web only)
The weekly reports were downloaded, and the data fields were then summarised for each video based on three time periods:
Lab-related period: As links to the videos were made available in the week prior to the practical laboratory classes, data was included for both weeks.
Exam-related period: As we were also interested in the use of videos as a revision aid, we included the week the exams took place as well as the week preceding the exam.
Interval-related period: this data summarised the period between the lab period and the exam period.
The data for each of these time periods, for each separate video, and each separate teaching treatment, were then collated in one data file and used for the analysis presented in the next section.
As outlined in
Weekly Video Topic |
Series of Videos in Each Topic |
Video Topic Number |
Video Length (min:sec) |
Sum of Unique Views* |
||
---|---|---|---|---|---|---|
Treatment 1 (Traditional) |
Treatment 2 (Flipped Classroom) |
Total |
||||
Topic 1: Data Filtering and Totalling |
Data Filtering |
1 |
14:10 |
504 |
409 |
913 |
Data Sorting |
2 |
13:27 |
607 |
450 |
1057 |
|
Data Outlining & Subtotaling (Part 1) |
3 |
10:14 |
511 |
409 |
920 |
|
Data Outlining & Subtotaling (Part 2) |
4 |
5:26 |
457 |
360 |
817 |
|
Topic 2: Data Reporting and Visualisation |
Pivot Tables |
1 |
10:58 |
519 |
445 |
964 |
Conditional Formatting |
2 |
12:50 |
454 |
401 |
855 |
|
Topic 3: Logical Statements |
AND/OR |
1 |
14:49 |
459 |
354 |
813 |
IF |
2 |
6:15 |
522 |
334 |
856 |
|
Nested Statements (Part 1) |
3 |
8:37 |
438 |
329 |
767 |
|
Nested Statements (Part 2) |
4 |
8:35 |
388 |
296 |
684 |
|
Topic 4: Cashflow Statements for Businesses |
Cashflow (Part 1) |
1 |
12:40 |
411 |
370 |
781 |
Cashflow (Part 2) |
2 |
15:00 |
350 |
310 |
660 |
|
Cashflow-Goalseek |
3 |
8:14 |
340 |
280 |
620 |
|
Cashflow- Scenarios |
4 |
7:53 |
317 |
268 |
585 |
|
Topic 5: Linear Programming |
Solver (Part 1) |
1 |
7:53 |
416 |
317 |
733 |
Solver (Part 2) |
2 |
14:27 |
397 |
289 |
686 |
|
Solver (Part 3) |
3 |
8:49 |
370 |
274 |
644 |
|
Total |
|
|
|
|
|
|
The remainder of this section discusses the findings for each of the above presented questions:
We expected that in Treatment 2 (flipped classroom without a traditional lecture), there would be more student views of videos as these were the primary resource available to students.
Examining only the total number of views would suggest that the flipped classroom approach was not as effective in getting students to engage with the video material (5895 total views in Treatment 2 [384 students]vs. 7460 total views in Treatment 1 [348 students])(see
While the total number of views suggests that treatment 1 was more effective than treatment 2 in terms of total number of views, we then examined the number of unique views per student to account for different class sizes (N=348 and 384 respectively) (
When examining views per student by time period the videos were used more as a revision aid prior to the exam when a lecture had been given (11.84), as opposed to when the class was flipped (8.72) (see
Unique views per student |
Treatment 1 Traditional |
Treatment 2 Flipped |
---|---|---|
For the semester |
21.44 |
15.31 |
In the Lab period |
6.35 (31%) |
5.81 (38%) |
In the Exam period |
11.84 (55%) |
8.72 (57%) |
It should be noted that, given the available data, it was not possible to relate individual students to specific views. Therefore, it was possible that some students viewed videos during the lab period and during the exam period, while others did not view any videos. It was not possible to identify the degree to which this occurred.
As the number of students in each treatment, and the number of videos per topic varied, the average views per student per video (
Lab Period |
Interval Period |
Exam Period |
||||
---|---|---|---|---|---|---|
Treatment 1 |
Treatment 2 |
Treatment 1 |
Treatment 2 |
Treatment 1 |
Treatment 2 |
|
Topic 1: Data Filtering and Totalling |
0.34 |
|
0.44 |
0.04 |
0.71 |
|
Topic 2: Data Reporting and Visualisation |
0.32 |
0.32 |
0.36 |
0.16 |
0.72 |
|
Topic 3: Logical Statements |
|
|
0.20 |
0.02 |
|
|
Topic 4: Cashflow Statements for Businesses |
|
0.35 |
0.00 |
0.00 |
0.82 |
0.45 |
Topic 5: Linear Programming |
|
0.36 |
0.00 |
0.00 |
|
0.41 |
When students were given a lecture on the topic (Treatment 1), the more difficult topics (topics 3 & 5) were viewed most during the lab period (0.47 and 0.55) while the easiest topics (topics 1, 2, and 4) were viewed least. One possible explanation could be that the lecture alerted students to the relative difficulty of material; consequently, students were more likely to watch videos on topics that they knew to be difficult, during the lab period. In the flipped year (Treatment 2), however, students were not aware of the difficulty of the various topics in advance. While the first topic was viewed most during the lab period (0.41), and one of the difficult topics (topic 3) was viewed least (0.28), there was no other discernible pattern.
In treatment 1, when it came to the use of videos during the exam period, students viewed the more difficult topics (0.63 and 0.59) least. In the same exam period for treatment 2, students tended to watch the earlier videos more than the later videos, suggesting that perhaps students lost interest or needed a supplementary lecture to instil in them the importance of the content and topic difficulty.
The number of minutes each video was watched was also examined. This was useful as such data has not been explored in related studies in this field. Studies that rely on the number of views as a measure of engagement have limited value as an additional view is counted regardless of whether one second or the entire video was subsequently watched. By analysing the number of minutes watched, we can also determine the extent of the viewing.
Topic |
Treatment 1 (Traditional) |
Treatment 2 (Flipped Classroom) |
---|---|---|
Topic 1: Data Filtering and Totalling |
17,403 |
14,265 |
Topic 2: Data Reporting and Visualisation |
8,661 |
7,705 |
Topic 3: Logical Statements |
12,224 |
9,516 |
Topic 4: Cashflow Statements for Businesses (Easy) |
10,827 |
9,889 |
Topic 5: Linear Programming (Difficult) |
8,777 |
7,124 |
Total |
57,892 |
48,499 |
To make the figures comparable given different class sizes,
Topic |
Treatment 1 |
Treatment 2 |
---|---|---|
Topic 1: Data Filtering and Totalling |
50.01 |
37.15 |
Topic 2: Data Reporting and Visualisation |
24.89 |
20.07 |
Topic 3: Logical Statements |
35.13 |
24.78 |
Topic 4: Cashflow Statements for Businesses |
31.11 |
25.75 |
Topic 5: Linear Programming |
25.22 |
18.55 |
Not only were there more unique views for every topic in Treatment 1 compared to Treatment 2 (
Delving deeper, the average number of minutes watched per student per time period was also examined.
Lab Period |
Interval Period |
Exam Period |
||||
---|---|---|---|---|---|---|
Topic |
Treatment 1 |
Treatment 2 |
Treatment 1 |
Treatment 2 |
Treatment 1 |
Treatment 2 |
Topic 1: Data Filtering and Totaling |
9.95 |
|
14.92 |
1.57 |
25.14 |
|
Topic 2: Data Reporting and Visualisation |
5.17 |
|
6.45 |
2.88 |
13.27 |
|
Topic 3: Logical Statements |
|
|
6.21 |
1.59 |
|
|
Topic 4: Cashflow Statements for Businesses |
|
12.17 |
0.00 |
0.00 |
|
|
Topic 5: Linear Programming |
|
|
0.00 |
0.00 |
|
|
The relatively short length of the videos is possibly a contributing factor in the viewing rates, which agrees with the findings in (
In
Furthermore, despite receiving a lecture on the topic, students in treatment 1 watched the more difficult videos for a longer period (12.49 and 12.41) than the Treatment 2 students (8.64 and 10.10 minutes on average). For some students it seems the videos were used to reinforce the lecture content but, in treatment 2 when no lecture was provided, students were less likely to view the available videos.
During the exam period the more difficult topics (3 and 5) were watched for longer when there was also a lecture (16.43 and 12.81 minutes on average) compared to the same videos under treatment 2 (14.56 and 8.45). This again suggests that the lecture in treatment 1 alerted students to more difficult topics. Interestingly, students in treatment 2 watched easy videos 1, 2, and 4 for longer during the exam period (20.96, 11.45, and 13.58) than during the lab period (14.63, 5.74, and 12.17 minutes on average), possibly because this was their only source of instructional content in the period leading up to the exam.
Another way of analysing the data was to look at the proportion of minutes viewed in the lab period compared to the exam period.
In treatment 2, it would have been reasonable to expect that more minutes would be viewed during the lab period, as no lectures were provided; however, the videos were also used more often as a revision aid during the exam period (54.10% of minutes viewed). Nonetheless, proportionately more minutes were viewed during the lab period (40.91%) for treatment 2 than during the lab period for treatment 1 (29.14%).
Treatment 1 |
Treatment 2 |
|||||
---|---|---|---|---|---|---|
Topic |
Lab Period |
Interval Period |
Exam Period |
Lab Period |
Interval Period |
Exam Period |
Topic 1: Data Filtering and Totalling |
19.90% |
28.83% |
50.26% |
39.37% |
4.22% |
56.41% |
Topic 2: Data Reporting and Visualisation |
20.77% |
25.91% |
53.32% |
28.62% |
14.33% |
57.05% |
Topic 3: Logical Statements |
35.57% |
17.67% |
46.76% |
34.85% |
6.40% |
58.75% |
Topic 4: Cashflow Statements for Businesses |
28.28% |
0.00% |
79.72% |
47.26% |
0.00% |
52.74% |
Topic 5: Linear Programming |
49.20% |
0.00% |
50.80% |
54.44% |
0.00% |
45.56% |
Overall |
29.14% |
14.68% |
56.17% |
40.91% |
4.99% |
54.10% |
Given the number of views and the number of minutes watched per video, it was possible to calculate the average minutes watched per view, for the lab, interval and exam time periods (
If students were going to ‘give up' on a topic, it is reasonable to assume that they would likely watch the first in a series of videos for a topic and then stop watching the rest of the series; in other words, the proportion of videos watched should decrease over the series.
For treatment 1, we can see that for topic 2 (easy), video 1 was watched for longer than video 2 (63% viewed on average vs. 54%), for both treatments and during the lab and exam periods. However, for all other easy and difficult topics and videos, there is no obvious pattern for either treatment.
Series of Videos in Each Topic |
Treatment 1 |
Treatment 2 |
||||
---|---|---|---|---|---|---|
Lab Period |
Interval Period |
Exam Period |
Lab Period |
Interval Period |
Exam Period |
|
|
|
|
|
|
|
|
Data Filtering |
|
60% |
62% |
58% |
62% |
60% |
Data Sorting |
51% |
53% |
60% |
62% |
54% |
59% |
Data Outlining & Subtotaling Part 1 |
53% |
63% |
68% |
|
63% |
68% |
Data Outlining & Subtotaling Part 2 |
47% |
72% |
|
60% |
50% |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Pivot Tables |
|
66% |
|
|
55% |
|
Conditional Formatting |
54% |
51% |
58% |
62% |
51% |
57% |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
AND/OR |
|
61% |
54% |
64% |
43% |
57% |
IF |
58% |
63% |
57% |
|
44% |
|
Nested Statements Part 1 |
55% |
66% |
|
65% |
48% |
64% |
Nested Statements Part 2 |
54% |
67% |
57% |
59% |
31% |
56% |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Cashflow Part 1 |
56% |
0% |
55% |
|
0% |
52% |
Cashflow Part 2 |
47% |
0% |
52% |
63% |
0% |
49% |
Cashflow-Goalseek |
54% |
0% |
|
62% |
0% |
53% |
Cashflow- Scenarios |
|
0% |
55% |
64% |
0% |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Solver Part 1 |
59% |
0% |
56% |
|
0% |
53% |
Solver Part 2 |
|
0% |
51% |
67% |
0% |
50% |
Solver Part 3 |
58% |
0% |
|
67% |
0% |
|
While we initially assumed that the students would view the videos more in the flipped classroom, when no prior lecture was provided (treatment 2), the opposite is actually the case. Our study found that there were more unique views of videos when a prior lecture was provided (treatment 1), than when no prior lecture was provided (treatment 2). However, when we examined when students used the videos, we found that students used them more as a revision aid prior to the exam, than during the lab period, regardless of whether a lecture was provided (treatment 1) or not (treatment 2). Further studies are required to examine students’ demographic data for the different cohorts (not available to the researchers for this study) and the significance of these observations.
We also examined the data for different video topics, to determine if the difficulty level of the topic affected viewing patterns during the lab and exam periods. We first examined views during the lab period. When students were given a lecture on the topic (Treatment 1), the more difficult topics were viewed more than the easiest topics, suggesting that the lecture was useful as it alerted students to difficult topics. In treatment 2, when no prior lecture was given, while the first (easy) topic was viewed the most during the lab period, there was no other discernible pattern.
During the exam period, students spent the least time watching the difficult videos, for treatment 1. For treatment 2 (no prior lecture), students tended to watch the earlier videos more than the later videos, possibly suggesting fatigue.
When we examined the total number of minutes that were watched for each video topic, we saw that the total number of minutes watched per student was higher in treatment 1. When we analysed the average number of minutes that were watched during the lab and exam periods, for both treatments, we noticed that students spent longer watching each topic during the exam period, than during the lab period (with the exception of the last topic, for treatment 2).
We also observed that, during the lab period for treatment 1 (lectures before labs), the more difficult topics were watched for longer than the easier topics. When compared with the same period for treatment 2, treatment 1 students still watched the difficult videos for longer than the treatment 2 students (possibly because the treatment 2 students did not know they would be difficult).
During the exam period, difficult topics were watched for longer when there was also a lecture (treatment 1), compared to the same period for treatment 2. Students in treatment 2 watched easy videos for longer during the exam period, than during the lab period, possibly because they had not attended a lecture.
When we examined the proportion of minutes watched per time period, we perceived that proportionately, students watched the videos for longer during the exam period than during the lab period, for both treatments.
Finally, when we examined which videos students tended to watch, we noted that for topic 2 (easy), video 1 was watched for longer than video 2, for both treatments and during the lab and exam periods. However, for all other (easy and difficult) topics and videos, there was no obvious pattern for either treatment.
These results have implications for how we might analyse video use moving forward. For example, rather than reviewing the data at the end of a module, it would be worthwhile to monitor video access rates at key points throughout the term, to identify levels of engagement with individual videos and topics. This deeper analysis is also useful to determine the extent of engagement (e.g. length of viewing time) with individual videos, and when students actually view the videos (e.g. during the lab period or during the exam period). It also points to the usefulness of lectures to supplement video resources.
Because YouTube analytics did not enable us to identify individual students using multiple devices to watch the same video, the study instead focused on the number of unique views, which accounted for multiple views from the same PC (cookies were tracked). Unfortunately, it is not currently possible to calculate the number of multiple views from mobile devices (e.g. smartphones and tablet devices), so students viewing a video multiple times from a mobile device could not be accounted for.
As it was not possible to identify individual students, it is possible that some students viewed videos during the lab period and during the exam period, while others did not view any videos. It was not possible to identify the degree to which this occurred.
While it was possible that a video could be accessed and allowed to play to the end, without actually reviewing the content, this was possible in both years. Finally, we were unable to obtain detailed demographic data relating to students in both years; further studies could investigate student demographics (for different cohorts) and the correlation, if any, with video usage patterns.
For future work, we are interested in conducting further analyses to determine:
The number of views at each point in time in the video. For example, are there peak and trough viewing points within each video and can we identify possible reasons for those e.g. are there threshold concepts that may require further exploration in the classroom? This level of analysis would greatly facilitate further refinement of the instructional videos.
Who watches the videos through the end - those who tended to watch the videos during the lab weeks or those who tended to watch the videos immediately prior to the exam?
Of those who tended to watch the videos immediately prior to the exam, are they viewing the videos for the first time or as a revision aid?