Out of Order, Out of Reach: Navigating Assignment Sequences for STEM Success

Abstract

Research shows that pacing has an important relationship with online course performance; however, most work has focused on the timing—not the order—of students’ assignment submissions. The current study examined the relationship between the order of students’ assignments and their final course grades in online STEM classes. Using course pacing guides as a benchmark, students’ assignment submissions were categorized as either “in sequence” or “out of sequence.” Then, students were categorized as either moving through their courses “in sequence” or “out of sequence.” Most students were categorized as moving “out of sequence” (~93%) and submitted around 38% of their assignments out of order. As such, going out of sequence was common among students, but done somewhat sparingly within the courses themselves. While this “out of sequence” behavior was common, it was not necessarily advantageous for students’ final grades. On average, students who completely adhered to the pacing guide had final grades 9.5 points higher than students who deviated from the pacing guide at least once. A small but statistically significant negative correlation was observed between the proportion of assignments submitted out of order, the extent to which a student submitted an assignment out of order, and final grades. In other words, as students become increasingly out of order, final grades decrease. Taken together, pacing continues to represent a student behavior that may have important implications for course performance. Instructors and mentors should continue to monitor student pacing, and communication about course progression is encouraged. Future work should focus on examining student submission patterns from multiple perspectives to better understand their relationship to achievement. 

Introduction

Michigan is seeing a rise in student engagement with online learning. The number of K-12 students who took at least one virtual course doubled from 7% in 2017-2018 to 14% in 2021-2022 (Freidhoff, 2019; 2023). As online learning continues to grow in popularity, it is essential to set students up for success as the current virtual pass rate is around 69% (for context, the pass rate for non-virtual coursework was 71%).

Studies have shown that pacing, which refers to how students progress through a course, is crucial for student success (DeBruler, 2021; Michigan Virtual Learning Research Institute, 2019). For example, submitting an assignment within the first week is correlated with students’ final grade, suggesting that this may be an early indicator of students’ engagement with course material (Zweig, 2023). Generally, students who are consistently on-pace throughout the course are more likely to be successful than those who aren’t (DeBruler, 2021). Similarly, students who struggle with pacing (as indicated by cramming assignment submissions at the end of the course) tend to perform more poorly than those who consistently pace out their submissions (DeBruler, 2021; Michigan Virtual Learning Research Institute, 2019).  

Certain Michigan Virtual courses, such as those for core subject areas and electives, do not have assignment deadlines, meaning students may submit any assignment at any time during the enrollment window. Because of this structure, students can progress through assignments in any order they would like, giving students flexibility about when and where learning occurs. To help provide guidance and structure, Michigan Virtual provides pacing guides that show what assignments and activities students should complete in a particular week or sequence. In other words, pacing guides provide clear expectations of students’ course progression and serve as a benchmark for students to evaluate their course progress. While a complete discussion of the ideal blend of structure and flexibility is beyond the scope of this report, providing some guidance around scheduling and routines can help students stay on track (Martin & Whitmer, 2016). 

Although not mandatory, following the pacing guides can help students manage a course’s workload, especially when courses do not have firm deadlines. Research suggests that disengagement from assignments and improper pacing can negatively impact student achievement (DeBruler, 2021; Michigan Virtual Learning Research Institute, 2019; Soffer & Tal, 2019; Wu et al., 2023; Zweig, 2023). While the frequency and consistency of course activity contribute to academic achievement, more research is needed to know how the order or sequence in which students engage with material is associated with course performance. 

This study examined how students’ engagement with course assignments related to their course performance, focusing on understanding how the sequencing of students’ assignment submissions was associated with overall performance (final grade). Identifying practices that promote or limit student progress is important because it could inform policies, instructional design principles, or LMS configurations that may improve outcomes. Understanding how users move through a course can allow for more informed planning and decision-making as well as the development of better student support structures. 

Methods

Data & Sample Overview

Data on graded course item submissions and course performance (final grade) was pulled from the learning management system, BrightSpace, for spring 2023 enrollments (n = 8,810 students). The dataset was filtered to exclusively contain students enrolled in STEM courses, a list of which is available in Appendix A (n = 1,818). Analyses focused on high school-level STEM courses because course content and assignments are scaffolded by design, making them well-suited for investigating the role assignment sequencing plays in course performance. 

Only students who completed their courses were included in the dataset (n = 1,732). Students with >50% of course assignments missing were excluded from the dataset to ensure accuracy given the focus on understanding how assignment submissions were related to final grades and these enrollments were missing most of their assignments (n = 1,481). Students enrolled in multiple courses during the spring 2023 semester (i.e., duplicates) were also removed from the dataset (n = 1,341). 

After all data cleaning, 1,308 students were retained in the dataset. Please note that for this report, “assignments” refers to any graded item within a course. Students in the sample were not first-time online learners; all enrollments had completed at least one course and approximately three courses on average.

Analysis

To gain insights into students’ submission patterns in STEM courses, a benchmarking variable called ‘User Driven’ was created. This numerical value examines the order in which students submit assignments relative to the provided pacing guide. Specifically, this value compares the student’s current assignment submission with the one immediately preceding it. If the value for the current submission is one greater than the previous assignment, it is deemed “in sequence”; otherwise, it is considered “out of sequence.” 

For example, if a student submits assignment 9 immediately following assignment 4, it is considered out of sequence as 9 is not one greater than 4. It’s crucial to highlight that, according to this benchmarking variable, a student’s submission is only deemed “in sequence” concerning the assignment immediately before it. Refer to Table 1 for an illustrative data layout. This method was crafted to evaluate the extent to which a student adheres to the pacing guide and to understand the implications of deviations on their final grade. 

It may be viewed as a relatively “strict” interpretation of pacing as it requires the student to move through the course in sequential order and narrowly defines in/out of sequence based on a singular assignment. However, this conceptualization will serve as a starting place for understanding the impact of assignment sequencing on final grades. Future analyses could look at pacing more holistically by investigating how students’ return to earlier content (after deviating from the pacing guide) influences their final grades.

Based on the order of their assignment submissions, students were assigned to one of two groups: “in-sequence” (if they submitted all assignments in line with the pacing guide) or “out of sequence” (if they submitted at least one assignment out of line with the pacing guide).

Next, a ‘Proportion of Assignments Completed Out of Order’ variable was created. This variable calculates the number of assignments a student completed out of order relative to the total number of assignments they completed in the course. 

Finally, a variable was created to understand the extent to which students deviated from sequentially navigating the course. The ‘Average Magnitude’ variable represents the average difference between consecutive assignment submissions for each student. For example, if a student submitted Assignment 4 and then Assignment 9, the magnitude would be 5.

Appendix B contains a list of all the variables created and referenced above and in subsequent sections of this report and may be helpful for the reader to reference. 

Results

Is going out of sequence a common course behavior?

What percentage of online course enrollments go out of sequence?

Going out of sequence appeared to be the norm among enrollments sampled as approximately 93% indicated going out of sequence at least once. See Table 2 for totals. As such, students are more likely to deviate from the pacing guide than not – at least in STEM courses.

What is the average number of assignments submitted out of sequence?

The number of assignments submitted out of order across the entire sample ranged from zero to sixty, with 17.5 assignments submitted out of order on average (SD = 11.57). Table 3 details the descriptive data. The median value was 16, indicating that half of the students in the data set submitted less than 16 assignments out of order, and half submitted more than 16 assignments out of order. For context, on average, 38.15% of course assignments were submitted out of order (SD = 22.52).

While some students submitted all assignments in their intended order, others submitted almost 94% of course assignments out of sequence. Half of the students submitted less than 38.8% of assignments out of order, while the other half submitted more than 38.8%. Overall, these results suggest that students turn in most of their assignments in the intended order, but there is variation among individual students. 

Investigating students’ assignment completion strategies and how individual differences contribute to course navigation may explain the variation in student submission patterns. When students deviated from the pacing guide and submitted assignments out of sequence, the extent to which they did so was relatively small. The average deviation was 2.31 assignments (SD = 1.06). As such, students were typically about two assignments “off” from the intended order. 

What is the relationship between course progression and students’ overall course performance?

What is the relationship between course progression and final course grade?

Students’ number of current and previous online courses did have a small but statistically significant relationship with final grade, proportion of assignments completed out of order, and magnitude suggesting that experience may factor into student performance but only to a small degree. 

The proportion of assignments completed out of order and magnitude had a statistically significant and negative relationship with final course grade, albeit with a small effect. The correlation coefficient for each variable was approximately -0.2, which was considered small.

This suggests that the proportion of assignments completed out of order and magnitude move in the opposing direction of the final course grade. For example, as a student’s final grade increased, the proportion of assignments completed out of order decreased (and vice versa). Similarly, as the magnitude increases, the final course grade decreases (and vice versa). 

While the current data does not allow for causal inferences, the negative relationship between completing assignments out of sequence and final grade suggests that completing assignments out of sequence may be part of a broader pattern of learner behaviors and characteristics that influence academic performance. For example, characteristics like cramming and poor time management are negatively associated with learning and may influence a student’s assignment submission behaviors (DeBruler, 2021; Hartwig & Malain, 2022; Malekian et al., 2020; Michigan Virtual Learning Research Institute, 2019). 

Are there differences in final course grades between students who go out of sequence and those who do not?

There was a difference of 9.5 points in the final grades of students who went out of sequence and those who stayed in sequence. Students who moved in sequence averaged a final grade of 89.2, outperforming their peers who went out of sequence and averaged a final grade of 79.7. There was slightly more variation in the final grades of students who went out of sequence; however, half of them received a final grade greater than 84.7% while half received a final grade less than that. Table 4 shows the descriptive statistics for final grades among enrollments who went in sequence and those who did not.

To gain a more nuanced understanding of how assignment submission patterns related to students’ final grades, the data was segmented into quartiles based on the proportion of completed assignments submitted out of order and average magnitude (values dividing the data into four equal groups). See Table 5 for quartile breakdowns. 

Students with assignment submissions in the top 25% (4th Quartile) for being out of sequence (meaning this group of enrollments had the highest proportion of “out of order” assignment submissions) consistently had the lowest final grades on average. 

Conversely, students in the bottom 25% (1st Quartile) for being out of sequence (this group had the least proportion of “out of order” assignment submissions) consistently had the highest grades on average. This suggests that final grades drop as students become increasingly out of sequence. 

Similar results were found when examining the magnitude variable. Students in the bottom 25% of magnitude (1st Quartile; i.e., students with the smallest magnitude values) had higher final grades on average than students in the top 25% for the magnitude variable (4th Quartile; i.e., students with the greatest magnitude values).

Discussion

The current study demonstrated the importance of assignment sequencing as it relates to course performance. While submitting assignments out of order was extremely common in STEM courses, it did not necessarily benefit students. Students who stayed in sequence had final grades that were 9.5 points higher on average than students who went out of sequence.   

Correlational analyses showed that the proportion of completed assignments submitted out of order and the magnitude of assignments submitted out of order had a negative relationship with final course grades. Follow-up analyses that looked at the differences in final grades when students were grouped into quartiles based on the proportion of completed assignments submitted out of order and magnitude revealed that grades continually dropped as students submitted more assignments out of order. 

The largest discrepancy in grades was between students in the first (bottom 25%) and second (50%) quartiles of the proportion of completed assignments submitted out of order and magnitude. Students in the first quartile (bottom 25%) for the proportion of completed assignments submitted out of order had an average grade of 86.8. As students moved into the second quartile (50%), their grades dropped by 5.2 points. Because students in the first quartile (bottom 25%) submitted between 0 and 20 assignments out of sequence, this suggests that students may start to exhibit drops in their grades as they surpass that number. Similarly, students in the first quartile (bottom 25%) for magnitude had average final grades of 88.5, which dropped by 7.3 points as they moved into the second quartile (50%). Because students in the first quartile (bottom 25%) had a magnitude ranging from 0 to 0.62, this suggests that even going one assignment out of sequence may be detrimental to students’ grades. 

While causation cannot be inferred based on the current methodology, it may be that submitting assignments out of sequence is part of a broader pattern of student characteristics and/or behaviors that impact students’ academic performance. For instance, self-regulatory skills and metacognitive abilities are associated with online course performance. They may also be related to students’ academic achievement and engagement with certain assignments (Xu et al., 2023; Zion et al., 2015). 

That is to say,  if students are not thinking deeply about their learning progress and making adjustments, they may be more likely to complete assignments out of order and receive lower grades. While research on assignment sequencing with adult learners has demonstrated a different pattern of results (Lim, 2016), the current research suggests that submitting assignments out of sequence may not be helpful for students. Adult learners may have more fully developed self-regulated learning skills, allowing them to more freely direct their learning. Students may still be developing these skills, and thus may more heavily rely on the guidance of instructors to fully conceptualize and draw connections between content. 

While it is unreasonable to expect students to adhere to pacing guides 100% of the time, transparency about course design, the scaffolding of content and material, and the purpose of assignments may help increase adherence. Instructors may also stress to students that following the pacing guide and completing assignments in sequential order may help increase their chances of achieving their desired grade. 

References

DeBruler, K. (2021). Research On K-12 Online Best Practices. Michigan Virtual. https://michiganvirtual.org/blog/research-on-k-12-online-best-practices/

Freidhoff, J. R. (2019). Michigan’s k-12 virtual learning effectiveness report 2017-18. Lansing, MI: Michigan Virtual University. Available from https://mvlri.org/research/publications/michigans-k-12-virtual-learning-effectiveness-report-2017-18/

Freidhoff, J. R. (2023). Michigan’s k-12 virtual learning effectiveness report 2021-22. Michigan Virtual. https://michiganvirtual.org/research/publications/michigans-k-12-virtual-learning-effectiveness-report-2021-22/

Hartwig, M. K., & Malain, E. D. (2022). Do students space their course study? Those who do earn higher grades. Learning and Instruction, 77, 101538. https://doi.org/10.1016/j.learninstruc.2021.101538

Lim, J. (2016). The Relationship between Successful Completion and Sequential Movement in Self-Paced Distance Courses. International Review of Research in Open and Distributed Learning, 17(1), 159–179. https://doi.org/10.19173/irrodl.v17i1.2167

Martin, F., & Whitmer, J. C. (2016). Applying learning analytics to investigate timed release in online learning. Technology, Knowledge and Learning, 21, 59-74. https://doi.org/10.1007/s10758-015-9261-9

Michigan Virtual Learning Research Institute. (2019). Pacing Guide For Success In Online Mathematics Courses. https://michiganvirtual.org/blog/pacing-guide-for-success-in-online-mathematics-courses/

Soffer, T., & Cohen, A. (2019). Students’ engagement characteristics predict success and completion of online courses. Journal of Computer Assisted Learning, 35(3), 378-389. https://doi.org/10.1111/jcal.12340

Wu, D., Li, H., Zhu, S., Yang, H. H., Bai, J., Zhao, J., & Yang, K. (2023). Primary students’ online homework completion and learning achievement. Interactive Learning Environments, 1-15. https://doi.org/10.1080/10494820.2023.2201343

Xu, Z., Zhao, Y., Zhang, B., Liew, J., & Kogut, A. (2023). A meta-analysis of the efficacy of self-regulated learning interventions on academic achievement in online and blended environments in K-12 and higher education. Behaviour & Information Technology, 42(16), 2911-2931. https://doi.org/10.1080/0144929X.2022.2151935

Zion, M., Adler, I., & Mevarech, Z. (2015). The effect of individual and social metacognitive support on students’ metacognitive performances in an online discussion. Journal of Educational Computing Research, 52(1), 50-87. https://doi.org/10.1177/0735633114568855

Zweig. J. (2023). The first week in an online course: Differences across schools. Michigan Virtual. https://michiganvirtual.org/research/publications/first-weeks-in-an-online-course