Introduction and Need for the Study
Michigan’s K-12 Virtual Learning Effectiveness Report highlighted important differences in online course outcomes for students based on school poverty status, which was measured by the percentage of students who received free or reduced-price lunch (FRL). On average, students in high FRL schools did not do as well as students in lower FRL schools. Further, students from high FRL schools represented a larger share of online course enrollments (Freidhoff, 2023). With prior research suggesting that enrollment patterns and engagement influence student outcomes in online courses (e.g., Kwon, 2019; Ricker et al., 2020, Zweig, Stafford, & Hanita, 2022), Michigan Virtual sought to determine whether there were differences in the ways that students in high FRL schools were interacting with their online courses compared to students in other schools.
This study addressed the following research questions based on data from Michigan Virtual courses in fall 2022: 
- What were the enrollment, access, and submission patterns for students in schools that were categorized as high FRL?
- In what ways were patterns for students in high FRL schools different than for students in schools with other FRL categories?
Methodology
Sample and Data
The sample consisted of 9,382 students in 399 schools across 284 districts who were enrolled in Michigan Virtual’s online courses during the Fall 2022 semester and did not drop the course during the grace period. These 9,382 students represented 12,021 enrollments. Approximately 16% of students enrolled in more than one course. The number of students enrolled in each school ranged from 1 to 348, with 18 schools enrolling more than 100 students each. Michigan Virtual categorized schools into one of four categories based on the percentage of all learners at the school (not just virtual learners) that qualified for free or reduced-price (FRL) meals:
- Low FRL (<=25%)
- Mid-Low FRL (>25% to <=50%)
- Mid-High FRL (>50% to <=75%)
- High FRL (>75%)
Enrollments that did not have FRL data are in a separate category: no demographic data available. In this report, enrollments in the high FRL category were compared to enrollments in all other FRL categories, including those without demographic data. There were 17 schools in the high FRL category enrolling 1 to 338 students in Michigan Virtual (MV) online courses. Of the 9,382 students enrolled in Michigan Virtual courses at the course start date, 742 were in high FRL schools (7.9%). However, students from high FRL schools made up a larger proportion of enrollments, at 9.2% of enrollments (1,108 enrollments; Figure 1).
Figure 1. Distribution of Enrollments based on School Received Free or Reduced Price Lunch Category
Michigan Virtual provided de-identified student data on enrollment and course activity during the first two weeks, as well as course outcomes. Enrollment data included date of enrollment, type of enrollment, and enrollment status at the end of the course. There are three types of enrollments:
- Early start: students have an abbreviated term length, with students enrolling prior to 8/26 and starting as soon as they enroll, with a consistent early end date before the winter holidays.
- Fixed start: students have a consistent term length of 20 weeks, with weekly start dates in August and September, and end dates in January.
- Delayed start: students have an abbreviated term length, with students delayed enrollment after the fixed start dates, and a consistent end date in January regardless of enrollment date.
Course activity included dates of course access and assignment submission, which were used to generate the following binary indicators:
- accessed the course during the first week
- accessed the course during the second week
- submitted an assignment during the first week
- accessed the course before the last date the student could drop the course with a refund (14 days after the start date)
The weeks were counted based on the course start date or the date the student enrolled in the course, whichever was later. For example, if a student enrolled in the course five days after the course start date, the first week for that student would be seven days after the student enrolled, which was 12 days after the course start date. This approach avoided conflating the influence of enrollment timing with course activity.
The main course outcome was final grade, which was equal to points earned divided by points possible. A final grade of at least 60 percent was considered a passing grade, though the final determinations for a grade were made by the student’s school rather than the MV online teacher.
Analytic Approach
The primary analytic approach was a descriptive analysis of the course activity during the first two weeks of the course, calculating frequencies and means by FRL category. In addition, a regression model was used where final grade was the outcome measure, and the independent variables included enrollment timing, course activity, and the school’s FRL category. A hierarchical linear model was most appropriate because students were nested within schools. Unless otherwise noted, the unit of analysis is an enrollment because some students were enrolled in multiple courses.
Results
This section first describes the overall relationships between course outcomes and patterns of enrollment, access, and submission using the data from this study. Then, comparisons are made between enrollments from High FRL schools and enrollments from schools in all other categories.
Overall, 70% of enrollments were on-time, 93% of enrollments accessed their course during the first week, and 76% submitted an assignment during the first week. The average final grade was a 65, with enrollments from high FRL schools having lower average final grades than enrollments from schools in all other categories.
Enrollment, access, and submission were all correlated with final grades. Prior research suggests that students who enroll late are less likely to be successful in online courses (Zweig, Stafford, & Hanita, 2022), and that relationship was replicated with the data from this study (rho=-16). Accessing a course in the first week, number of days until the first access, and submitting an assignment during the first week were also moderately correlated with students’ final grades (rho=-0.31, 0.24 & 0.37 respectively).
Students who accessed their course in the first week had an average final grade of 71 compared to 41 for students who did not access their course that week (Figure 2). Even after excluding students who dropped the course, students who accessed the course in the first week had significantly higher scores than those who did not access the course during the first week. Similarly, students who submitted an assignment in the first seven days after enrolling in the course earned a final grade of 74 compared to 45 for students who did not submit an assignment in the first week.
Figure 2. Average Final Grades for Enrollments that Accessed and Submitted Assignments in the First Week Compared to Enrollments that Did Not
Enrollment
Students in high FRL schools were more likely to register for Late Start courses than students in other school categories (37% compared to less than 1%) and less likely to enroll in traditional Fixed Start courses (61% compared to 89%; Figure 3). There were also differences in the likelihood that students started late in their courses, even after accounting for the different type of enrollments. That is, 48% of enrollments from high FRL schools were late, meaning that students started the course after the course start date regardless of the type of enrollment, compared to 28% of students in all other schools. Students in high FRL schools also enrolled in more courses, 2.5 on average compared to 1.9.
Figure 3. Enrollment Type by Category of School FRL
Course Access
Students in high FRL schools accessed their courses later than students in schools with other FRL categories. A smaller percentage of enrollments from high FRL schools accessed their online courses within seven days of enrolling compared to enrollments from all other schools (68% compared to 95%; Figure 4).
Figure 4. Percent of Enrollments who Accessed the Course in the First Week
Enrollments from high FRL schools accessed their courses on average 11.3 days after starting the course while enrollments in low FRL schools took an average of 2.8 days to access their courses (Figure 5). Students in other FRL categories took a similar amount of time to access their courses – ranging from 2.8 days to 3.3 days for enrollments from schools in the other FRL categories. In only 16% of enrollments from high FRL schools did students access their course the first day they enrolled, and approximately 39% accessed their course 10 days or more after being enrolled. The patterns were reversed for enrollments in the other categories – with 38% accessing their course on the day they enrolled and only 5% accessing it 10 or more days after being enrolled.
Figure 5. Average Number of Days until First Course Access
For students that accessed their courses in the first week, the frequency of accessing the course was similar across FRL categories with an average of 3 times for high FRL schools compared to 3.5 for enrollments from all other school categories.
Assignment Submission
Enrollments from high FRL schools were less likely to submit an assignment in the first week (36% compared to 85%; Figure 6). When examining enrollments from high FRL schools that accessed their courses in the first week, 53% submitted an assignment that first week. The first submission for students in all FRL categories was typically an assignment in the introductory unit of the course. That is, 75% of all enrollments submitted an assignment in the introductory unit as their first submission.
Figure 6. Percent of Enrollments that Submitted an Assignment During the First Week
Students in high FRL schools took longer on average to submit their assignments. For example, students in high FRL schools submitted an assignment on average 15.8 days after they accessed their courses, compared to 4.3 days for enrollments from low FRL schools (Figure 7). The average for all enrollments not from high FRL schools was similar to the low FRL school category, at 4.5.
Figure 7. Average Number of Days Until First Assignment Submission
There was still a gap even when accounting for when the students accessed the course for the first time. On average, students in high FRL schools submitted an assignment 7.4 days after they first accessed the course, whereas students in all other schools took 1.6 days. In many cases, enrollments from high FRL schools submitted their first assignment after the date in which they could withdraw from the course with a full refund (31%). An assignment was never submitted in 19% of enrollments from high FRL schools compared to less than 1% in all other schools.
Combined Influence of Enrollment, Course Access, and Assignment Submission
The relationships between final grades and the variables related to enrollment, access, and submission were consistent with the descriptive findings, even after accounting for the courses in which students were enrolled. Overall, approximately 18% of enrollments from high FRL schools resulted in a passing grade. When the sample was restricted to on-time fixed-start enrollments from high FRL schools where the student accessed and submitted an assignment in the first week, 47% earned a passing grade. The average final grade also doubled, from 24 to 50. The differences for students in other FRL category schools were not as large, with final grades increasing by only 6 percentage points (compared to 29 percentage points for high FRL) when the sample was restricted to on-time fixed-start enrollments where the student accessed and submitted an assignment in the first week.
Implications
Trends Require Further Investigation
This analysis demonstrated a relationship between students’ final grades and students’ enrollment, course access, and assignment submission patterns. One implication is to further examine whether there are differences in structures and supports between high FRL schools and schools in other categories that may be contributing to the differences in enrollment, course access, and assignment submission patterns. It may also be beneficial to review the first assignments in the courses and consider whether any modifications should be made for students to experience early success in an online environment.
Students enroll in online courses for a variety of reasons, and those reasons may also relate to the number of courses a student takes online and the timing of their enrollment. For example, it may not be possible to enroll before the course start date when students enroll in an online course because of unanticipated teacher vacancies in the students’ school or because they have an unexpected conflict in their schedule. Similarly, there are many reasons why a student may not access their course. It could be that a student does not have access to the right technology, is not aware of the need for consistent engagement in online courses, or it is even possible that the student was not aware of the enrollment. A next step from this research is to examine the patterns for each of the 17 high FRL schools and gather information from the school about the enrollment processes and other structures to support student learning.
Access and Submission as Early Indicators
The relationships presented here demonstrate that students’ enrollment timing along with their initial access and submission of assignments might serve as an early signal to online teachers about which students may benefit from intervention. Prior research has used course activity to predict outcomes primarily in MOOCS (e.g., Bañeres et al., 2023; Dalipi, Imran & Kastrati; 2018; Hu, 2022; Mubarak et al., 2019), though some studies have occurred in the secondary school setting (Hung & Rice, 2018; Ricker, Koziarski, & Walters, 2020). The analysis here suggests that even using two or three data points from the first week could create opportunities to intervene with many students whose early engagement is a strong predictor for their outcomes. For example, the teacher could communicate with students or the school administrator to identify the barriers facing students who were not engaging in their online courses. For students in high FRL schools in this sample, 87% of late enrollments who did not access the course in that first week had a final grade below the passing benchmark (60%). If it had been possible to successfully intervene for these students, the outcomes for nearly three-quarters of students from high FRL schools would have likely improved.
Some of these early signals can be implemented in learning management systems to automatically provide information to teachers about students who have not accessed their courses. The analysis in this study suggests that many students did not access or submit an assignment until after the date at which students can drop the course for a full refund. In addition to examining activity within the first week, it might also be helpful to examine activity before the drop date to try to initiate communication with students when there are still opportunities to drop the course without penalty.
Consider Other Factors
The relationships between final grades and the variables related to enrollment, access, and submission, are confirmed through the regression analysis. However, the coefficients on the indicators for a school’s FRL category remain significant even after accounting for the variables of interest and course subject, which suggests that there are other factors beyond those in the model that are influencing the outcomes for students in high FRL schools. Research is needed to understand what other factors and experiences may be contributing to these outcomes, and to the decisions around enrollment, access, and submission.
Conclusion
This study provided descriptive evidence about differences in the ways that students in high FRL schools were interacting with their online courses compared to students in other schools. The results from this analysis suggest that enrollment and activity during the first week are different in high FRL schools, and that further investigation is needed to better understand the context and barriers these students face. The results further suggest that it could be beneficial to communicate with students who do not submit an assignment during the first week to determine how they can be supported to be successful in their online course.
References
Bañeres, D., RodrÃguez-González, M.E., Guerrero-Roldán, AE. et al (2023). An early warning system to identify and intervene online dropout learners. International Journal of Educational Technology in Higher Education 20(3). https://doi.org/10.1186/s41239-022-00371-5
Dalipi, F., Imran, A.S., & Kastrati, Z. (2018). MOOC dropout prediction using machine learning techniques: Review and research challenges. 2018 IEEE Global Engineering Education Conference (EDUCON), 1007-1014.
Freidhoff, J. R. (2023). Michigan’s k-12 virtual learning effectiveness report 2021-22. Michigan Virtual. https://michiganvirtual.org/research/publications/michigans-k-12-virtual-learning-effectiveness-report-2021-22/
Hu, Y.-H. (2022). Using Few-Shot Learning Materials of Multiple SPOCs to Develop Early Warning Systems to Detect Students at Risk. The International Review of Research in Open and Distributed Learning, 23(1), 1–20. https://doi.org/10.19173/irrodl.v22i4.5397
Hung, A & Rice, K. (2018). Combining data and text mining to develop an early warning system using a deep learning approach. Lansing, MI: Michigan Virtual University. Retrieved from https://www.mvlri.org/research/publications/combining-data-and-text-mining-to-develop-an-early-warning-system-using-a-deep-learning-approach/
Kwon, J., Debruler, K., & Kennedy, K. (2019). A snapshot of successful K-12 online learning: focused on the 2015-16 academic year in Michigan. Journal of Online Learning Research, 5(2), 199-225. https://files.eric.ed.gov/fulltext/EJ1229422.pdf
Mubarak, A., Cao, H. & Zhang, W. (2022). Prediction of students’ early dropout based on their interaction logs in online learning environment. Interactive Learning Environments 30(8), 1414-1433, DOI: 10.1080/10494820.2020.1727529
Ricker, G., Koziarski, M., & Walters, A. (2020). Student clickstream data: Does time of day matter? Journal of Online Learning Research 6(2), 155-170. https://www.learn¬techlib.org/p/215688/
Zweig, Stafford, Hanita (2022) Enrollment Timing in Supplementary Online Courses: Do Students Who Enroll On-Time Have Better Course Outcomes? Journal of Online Learning Research 8(2), 163-180
