About this Guide
This guide is intended to share how to review online courses using a model designed by Quality Matters and based upon the National Standards for Quality Online Courses. The online course review process detailed here draws from nationally-recognized best practices, national online learning standards, State of Michigan content standards, and Michigan Virtual’s substantial experience with reviewing online courses and online educational content. The set of national standards for quality have been a benchmark for online learning for more than a decade. The standards were originally created by the Aurora Institute (formerly iNACOL) and were updated in 2019 by Quality Matters (QM) and the Virtual Learning Leadership Alliance (VLLA).
Introduction
Enacted in October 2013, Section 21f of the State School Aid Act established that public school pupils in 6th through 12th grade (with the consent of parent or legal guardian if the student is under 18 unless the pupil is an emancipated minor) may enroll in up to two online courses during an academic term (semester or trimester). Online course offerings may be selected either from the syllabi available within the district catalog of online courses at the pupil’s resident district or from those available in Michigan’s Online Course Catalog (MOCC) maintained by Michigan Virtual®. The catalog houses the online course syllabi that are made available by Michigan school districts, public school academies, intermediate school districts, community colleges, and those provided by Michigan Virtual School® (MVS).
As a condition of offering an online course, either as part of a district catalog or MOCC, a providing district is responsible for producing an online course syllabus. Section 21f(14)(g) requires that an online course syllabus include all of the following components:
- The state academic standards addressed in an online course.
- The online course content outline.
- The online course required assessments.
- The online course prerequisites.
- Expectations for actual instructor contact time with the online learning pupil and other pupil-to-instructor communications.
- Academic support available to the online learning pupil.
- The online course learning outcomes and objectives.
- The name of the institution or organization providing the online content.
- The name of the institution or organization providing the online instructor.
- The course titles assigned by the providing district and the course titles and course codes from the National Center for Education Statistics (NCES) school codes for the exchange of data (SCED).
- The number of eligible nonresident pupils that will be accepted by the district in the online course.
- The results of the online course quality review using the guidelines and model review process published by Michigan Virtual®.
It is important to note several things about Michigan’s Online Course Catalog:
- Michigan Virtual® does not review district online courses.
- The catalog is not where a student enrolls in a course. It is simply a compilation of online courses offered by a variety of course providers, including schools, where Michigan students can compare and choose online courses after comparing course syllabi. It is recommended that the student, parent and school academic advisor work together to select course(s). A course provider includes in MOCC a link to where enrollment information for their courses can be located.
- While the course syllabi must be in the catalog, when a district adds an online course syllabus, the district will select whether the course is publicized in the district catalog only, the state catalog only, or both. Districts may request a unique URL to link to their website and display only their district’s catalog of online courses.
- Users can find detailed information about the functionality of MOCC in the help resources.
Guidelines to Offer Online Courses
For each online course a providing district makes available either through its own district catalog or MOCC, the results from a review must be included in the online course syllabus.
Online Course Quality Standards Second Edition
The National Standards for Quality Online Courses were first published by iNACOL (now the Aurora Institute) in 2007. The second version released in 2011 is the initial version implemented in Michigan’s Online Course Catalog. Drawing from research and online learning experts from across the U.S., the 2011 course standards provide a list of 52 standards divided among five sections. Descriptions of the five sections are:
- Content (13 Standards) – The course provides online learners with multiple ways of engaging with learning experiences that promote their mastery of content and are aligned with state or national content standards.
- Instructional Design (11 Standards) – The course uses learning activities that engage students in active learning; provides students with multiple learning paths to master; the content is based on student needs; and provides ample opportunities for interaction and communication — student to student, student to instructor, and instructor to student.
- Student Assessment (7 Standards) – The course uses multiple strategies and activities to assess student readiness for and progress in course content and provides students with feedback on their progress.
- Technology (11 Standards) – The course takes full advantage of a variety of technology tools, has a user-friendly interface, and meets accessibility standards for interoperability and access for learners with special needs.
- Course Evaluation and Support (10 Standards) – The course is evaluated regularly for effectiveness, using a variety of assessment strategies, and the findings are used as a basis for improvement. The course is kept up-to-date, both in content and in the application of new research on course design and technologies. Online instructors and their students are prepared to teach and learn in an online environment and are provided support during the course.
Instead of the five-point scale (0, 1, 2, 3, 4) iNACOL originally suggested, the review process for the online courses in Michigan uses a three-point scale. Reviewers will be asked to rate a standard as “Not Met,” “Partially Met,” or “Fully Met.” This rating system simplifies categories both for the reviewers and the consumers of the reviews.
Michigan Virtual developed, as a supplemental resource, a three-point rubric for each of the 52 standards based on feedback from school representatives who were looking for more guidance on how to conduct a quality review. The Rubric was developed by modifying a similar resource from the Texas Education Agency’s Texas Virtual School Network (TxVSN), and with the consultation of several helpful resources developed by the California Learning Resource Network (CLRN).
Online Course Quality Standards Third Edition
The third edition of the National Standards for Quality Courses released in 2019 is significantly different when compared to the previous version. Seven standards have replaced the five sections noted above. The seven standards are:
- Standard A: Course Overview and Support
- Standard B: Content
- Standard C: Instructional Design
- Standard D: Learner Assessment
- Standard E: Accessibility and Usability
- Standard F: Technology
- Standard G: Course Evaluation
Detailed descriptions of the changes are available in the Overview of Changes to Third Edition in 2019.
Quality Matters Process
Quality Matters, a non-profit organization and recognized leader in quality assurance for online learning, offers online course review services for K-12 and higher education courses. The QM review process includes a proprietary course review rubric and ensures that review teams, consisting of experienced teachers and instructional designers, have received appropriate training in online course review standards and processes. The results of formal QM reviews can also be in the form of an alternative course review report, and rated along the same scale of “Not Met”, “Partially Met”, and “Fully Met,” and be in compliance with Michigan’s legislative guidelines. It is also important to note that formal QM reviews do not include certain standards that are part of the National Standards for Quality Online Courses. QM has produced a correlation document to show the alignment between editions of the K-12 Secondary Rubric.
Review of Online Course
The purpose of the online course review process detailed below is to share with districts an approach to conducting a review that aligns with best practices.
Review Oversight
Districts should consider appointing a person—a review manager—to lead and oversee the review process. Where possible, including additional individuals to serve on a review oversight committee is preferable. The review manager and/or the review oversight committee is responsible for selecting course reviewers, providing training to the reviewers, and making sure the review process is completed on schedule.
Selecting Reviewers
Course reviews should be completed by at least two reviewers with a more optimal number being between three and five. Teams this size limit the number of reviewers that need to be recruited, help keep training and management efforts down, and yet still provide multiple perspectives on course quality.
To protect the integrity of the review process, reviewers should be selected through a process that minimizes to the greatest extent possible any conflicts of interest that could bias their review. For instance, districts that rely on third-party course providers for their online course(s) should neither rely exclusively on the review materials supplied by the third-party provider nor should they use reviewers who are associated with the third-party provider. Similarly, if the district has created its own online courses, it is best to solicit at least one or more reviewers who are outside the district, perhaps approaching other districts or an intermediate school district. In the case of the Michigan Virtual School®, courses developed by MVS are reviewed by teams that incorporate external reviewers.
It is important when selecting reviewers and assembling review teams that a range of expertise be represented. Subject matter experts who have specialized knowledge both in the substance of the Michigan content standards themselves but also in pedagogical practices that best facilitate their acquisition will be important. It is also important to include reviewers who are knowledgeable about attributes of quality online course design, instructional methodology, and their impacts on student learning.
Training Reviewers
Even though the reviewers selected should have substantial expertise, it is essential that reviewers receive training specific to the review process. Training course reviewers accomplishes the following:
- Familiarizes reviewers with the review guidelines, resources, processes, and timelines.
- Introduces and standardizes vocabulary and language used in the review process to increase shared comprehension.
- Increases inter-rater reliability, meaning that any one of the reviewers on the team is more likely to rate the item with the same score as another reviewer.
Prior to the training session, it will be helpful to provide reviewers with access to the National Standards for Quality Online Courses. The previous iNACOL National Standards for Quality Online Course, Version 2, is also available.
During the training session, it may be helpful to:
- Conduct a brief review of the documents provided to reviewers ahead of time and answer questions reviewers have about the resources.
- Model the process by selecting a sample online course to review. Ask each reviewer to rate the sample course against each of the 45 standards.
- Ask reviewers to discuss any challenges they experienced once they have had time to rate each standard.
- Look specifically at items that everyone scored similarly. Select a few of those items and ask reviewers to share their decision-making process for rating the standards that way. Did reviewers arrive at these similar scores in the same way?
- Look specifically at items that reviewers rated differently. How did reviewers arrive at these scores? Can the group arrive at consensus for the most appropriate rating for that item?
- Provide the reviewers with the number of reviews they will be expected to complete, instructions for how they will submit the results of their reviews, and the deadline for when reviews should be completed.
- Discuss any questions or concerns reviewers have about the review process.
It is estimated that it will take between 2-3 hours to review the sample course using the standards. Districts may want to schedule a single training session where the reviews and the subsequent discussion are conducted the same day. Another option is to have a brief meeting to answer questions and review the standards, then provide a few days for reviewers to conduct their reviews and bring them back together to discuss the rating experiences.
Conducting a Course Review
Important steps to complete when conducting the actual course review include the following:
- Provide reviewers with the link to and access information for the course(s) they are assigned to review.
- Make sure reviewers understand the process for submitting their standards ratings for a course to the review manager. For instance, Michigan Virtual is a QM member and uses their review process.
- Agree upon a set amount of time for reviewers to complete their independent reviews. Plan for 10-12 hours per course review and consider other factors such as review team members’ schedules and the time of year.
- Assemble the reviewers once the independent reviews are submitted. A single set of ratings agreed upon by all reviewers is needed to complete the review. Work to resolve any discrepancies in ratings for all standards.
- Once the reviewers agree on the final ratings, the ratings become finalized and represent the results that are to be shared in the online course syllabus.
Pilot Courses
A school district, ISD, or Michigan Virtual may wish to offer a newly developed virtual course that, due to the course development timeline, will not be able to have a course review conducted on it prior to the start of a semester. In such a case, the district, ISD, or Michigan Virtual is permitted to create an offering for the course in MOCC and must indicate the pending status of the review by doing the following:
- In the Course Description field, include: “This course is being offered in a pilot phase and has not been through a full review against the online course standards. Any course that has not yet been reviewed is marked as ‘Pilot: Review Pending’ to alert schools, parents, and students to this fact. Because of this, a school may choose to, but is not required to, deny a pupil enrollment request in this course out of concern over rigor or quality.”
- Under the course review section, select the rating of “Pilot: Review Pending” for any standard that has not yet been reviewed and indicate in the comment for that standard that the course is in a pilot phase and that the review is pending.
- In the overall notes section of a course review, select the date of review as the expected date of completed review and include: “This course is being offered in a pilot phase and has not been through a full review against the online course standards. Any standard that has not yet been reviewed is marked as ‘Not Met’ to alert schools, parents, and students to this fact. Because of this, a school may choose to, but is not required to, deny a pupil enrollment request in this course out of concern over rigor or quality.”
Districts, ISDs, and Michigan Virtual may only offer a pilot course without a completed course review for one academic term (semester, trimester, etc.). Subsequent offerings of the course must include the results of the course review.
Documenting the Review Process
It is suggested that review teams maintain accurate records of each review conducted, including supporting documentation used during the review. This documentation could include original rubrics used by each team member, including their own individual ratings for each standard and corresponding comments or suggestions to be considered by the review team as a whole. Additional supporting documentation may include screenshots of the actual course interface with comments or suggestions that relate to specific standards being reviewed. Maintaining this documentation ensures that accurate ratings for each standard for each course can be verified in MOCC.
Including Review Results in MOCC
Michigan Virtual currently maintains MOCC. In order for districts to offer online courses, they request a user account and are provided a unique district-specific code. After using that code to login to MOCC, a district can add multiple users who have the ability to add, edit, and delete the district’s online course syllabi. Below is a snapshot of how a review result appears in the course syllabus in MOCC.
As part of the syllabus listing process, districts use an interface that allows them to select a standard’s rating from a drop-down menu. There is also the ability to download a preformatted Microsoft Excel spreadsheet, add the ratings and comments for each standard, and upload the results to attach them to the syllabus. Users can find more information about the functionality of MOCC by reviewing the help resources posted on the Michigan Virtual website.
Review Cycle
Maintaining online course quality does not end with the initial online course review. Any model must incorporate a cycle of review to assure the online courses remain in alignment with Michigan’s content standards and legislative requirements for MOCC as either or both are updated.
Conclusion
QM’s process for review outlined in this guide is one example of a review process that is based on years of research and continuous improvement. There are other models that exist and are omitted here only for the sake of brevity. What is important is that an online course review process is chosen with an understanding of the requirements of Michigan’s Online Course Catalog.
Research and Resources for Online Learning Programs
Michigan’s K-12 Virtual Learning Effectiveness Report, 2020-21, produced annually, reflects continued growth in K-12 online enrollments in Michigan. The report provides school districts with the opportunity to benchmark their own virtual learning programs against their peers in the state. This opportunity should be an important step in a program’s continuous quality improvement activities. The report is organized into several sections. The first section looks at schools as the unit of analysis. The next section focuses on the virtual courses taken. The third section focuses on students. The fourth section captures performance on statewide assessments. There is also a brief section containing maps of virtual use. Each section is meant to capture the essential findings without being overly data intensive; however, data tables have been included in the appendices to provide those interested with more in-depth information.
For additional information and insights for developing and supporting your online and blended learning program, please visit the following web pages on the Michigan Virtual website:
- Michigan’s Online Course Catalog contains syllabi information (such as state academic standards, prerequisites, instructor contact time expectations, available academic support, and outcomes and objectives) as well as enrollment and course dates for online courses made available by Michigan school districts and Michigan Virtual.
- The Digital Backpack blog that shares findings and expertise related to K-12 online and blended learning from both a state and national perspective.
- Michigan schools are obligated to address the learning needs of students of all abilities so everyone has equitable access to education. When students have the tools to learn according to their abilities, everyone wins. By learning more about accommodations, accessibility, and inclusive pedagogy, educators can apply best practices in meeting the needs of all students in their classrooms.
- Research Publications that provide a foundation to examine, engage, and explore educational practices in the industry.
- Research Clearinghouse contains references to important research and publications in the field of K-12 online and blended learning.
- Michigan’s Online Learning Law page is dedicated to information on Michigan’s Section 21f legislation. It includes resources and samples developed by and for schools.
- A family of Guides to Online Learning details the world of online learning from the perspective of the people integral to creating a positive learning experience. Each guide outlines key definitions, research and resources, and practical strategies that paint a picture of what kind of preparations and support systems are necessary to ensure students succeed in their online courses.
- A page dedicated to Mentors, developed in partnership with school leaders and mentors, links educators to a professional learning community where they can ask questions, problem solve, and share ideas and resources with other mentors around the state including sample forms.
- The set of national standards for quality online programs, teaching, and courses have been a benchmark for online learning for more than a decade. All three sets of standards were updated and published in 2019 by Quality Matters and the Virtual Learning Leadership Alliance.
- The International Society for Technology in Education (ISTE) published Standards to provide a framework for innovation in education and help educators and education leaders worldwide prepare learners to thrive in work and life.
- Tools and samples developed to facilitate enrollment decisions:
