Reconceptualizing the Peer Review Process for Online Courses 

Over the past two decades, academic deans, directors, and faculty have struggled to develop an effective peer review process for online courses. Most institutional review processes begin and end with the Quality Matters (QM) standard, the most widely accepted quality standard in the field of online education. But while QM provides an industry standard for evaluating online course design using peer review, it does not address other aspects of online courses, such as course delivery and how an instructor engages with students.

The comprehensive review process used in the Washburn University School of Nursing (SON) focuses on three domains: design, delivery, and content. The evaluation is intended to be formative rather than summative. Faculty obtain feedback related to each of the domain areas and use the feedback to improve their courses. Deans use an overall administrative process to coordinate the various reviews. The online peer review process is depicted in Figure 1.

The design domain is based on Quality Matters standards and interaction design principles. The course design peer review consists of an instructional design consultation and a formal QM review.

• Instructional design consultation
  In the new model, the faculty member works with the school’s instructional designer to ensure that the course meets QM standards and that from an interaction design perspective, the course is engaging, consistent, and easy to use and clearly communicates content and expectations to students.
• QM review
  A formal external or internal QM review with three reviewers is completed. Each of the reviewers has received training to become a peer reviewer from Quality Matters. The final report from the QM review includes specific recommendations for each of the standards included in the QM rubric. The QM rubric is used to evaluate the course in eight areas:
  • Course overview and introduction
  • Learning objectives
  • Assessment and measurement
  • Instructional materials
  • Course activities and learner interaction
  • Course technology
  • Learner support
  • Accessibility and usability

The delivery domain focuses exclusively on how well instructors engage with students in the online environment. Each course delivery peer review is completed by a colleague who is given complete access to all areas of a course. The team that developed the model gleaned elements for this domain from a variety of sources. They compiled a list of desirable behaviors in the following areas:

• Social presence and availability
  Social presence and availability is the degree to which students perceive that an instructor is accessible in an online course. It is the difference between an online course being perceived as a correspondence course verses a course resembling a traditional class experience. A strong online presence allows instructors to not only discuss class content and assignments but also to share their expertise, experience, and interests to build a community of learners. Communicating early and frequently with students can improve student retention in the course. Instructors can achieve a strong online presence by posting a picture of and introducing themselves, sending regular announcements or emails, providing introductions to the course and to each unit or module, being active in discussions, scheduling virtual class sessions, and providing feedback on assignments.
• Instructor feedback
  Providing personalized, constructive feedback is critical to the success of an online course. In a face-to-face course, students receive verbal and nonverbal feedback from their instructor in the classroom. In an online course, feedback needs to be more explicit. One criterion in this area is providing personalized feedback to each student. It is often tempting to copy and paste one response to many students, but personalizing the feedback is more meaningful. Constructive feedback guides students on how to improve their performance.
• Coaching
  Students in an online class have a variety of experience levels with the format. Some will have taken several online courses, while for others the course marks their first online experience. In a face-to-face course, students are either present or absent. Student participation in an online course is often more subtle. The instructor needs to monitor student participation to identify students who are struggling and should send personal emails to reach out to students who are missing assignments.
• Privacy
  As with all classes, student privacy must be protected in online courses. For example, instructors need to be aware the replies to discussions posts can be seen by all students.
• Class management
  Class management includes instructor communication regarding student feedback, deadline reminders, and expectations. Timely student feedback is important in general and crucial in an accelerated course. Rather than relying on lectures, many online courses incorporate a variety of activities to engage students. Without timely feedback, students won’t know how to adapt and improve on future assignments. The instructor can use email and announcements to remind the class of upcoming deadlines. Because students access the course at different times during the week, it is essential that the instructor post all materials related to a unit or module when they first make the module available to students. The instructor should also post expectations for email communication, feedback, discussion postings, submitted assignments, and grades to ensure that students understand them.
• Virtual class sessions
  Incorporating virtual class sessions is a great way for instructors to engage with students. They not only increase the instructor’s social presence but also provide students with an alternative means of communicating with the instructor and each other. That said, they add another layer of complexity and a new set of best practices to follow, such as publishing dates well in advance, providing expectations for interaction, recording live sessions for students who were unable to attend, and posting the recordings.

Content

• Collaborative faculty review brings together faculty to examine a course as a whole and within the context of how content from the course builds on previous courses or lays a foundation for future courses. Faculty meet to conduct a review of each course and to develop recommendations for improvement. They evaluate the course to determine whether it is as follows:
  • Complete
    Is the course content complete? Within the context of the course within the program, are the key topics covered? Does the course build on previous courses, and how does it prepare students for upcoming courses?
  • Accurate
    Is the course content free of errors and supported by strong empirical evidence?
  • Current
    Does the course content reflect the latest evidence and/or advances in the field?
  • Relevant
    Does the course content relate to the program’s expected outcomes and to the roles that students are being prepared to assume.
  • Grounded in real-world context
    Is the course content supported by examples that provide a real-world context for understanding course topics?
  • Appropriately rigorous
    Does the course challenge students at the appropriate level of rigor (e.g., for a graduate or undergraduate course)?
• External expertise review recognizes that community partners and stakeholders can offer constructive insight during the evaluation process. The goal is to include community partners as part of the evaluation team and to involve stakeholders who practice in the role for which students are preparing or are knowledgeable about a specific topic. Community partners and stakeholders include educators from sister institutions, peers from community and professional organizations, and advanced practice nurses. External stakeholders are compensated for reviewing a section in a course matched to their area of expertise. Similarly, the content expert evaluates a selected aspect of the course to determine whether it meets the above evaluation criteria.

The administrative process supports the following:

• Budgeting for online peer reviews
  A budget must be maintained for online peer reviews. There are costs for submitting a course for formal QM review and for contracting with an external expert. Currently, these costs for the SON are $950 per course.
• Identification of faculty, courses, and schedules for reviews
  The dean identifies which faculty and courses will be reviewed each academic year. Tenure-track faculty who teach in online programs are given priority. Each summer, the dean informs those faculty who have been selected for review when specific courses will be reviewed that upcoming academic year.
• Procuring external reviewers
  Engaging external reviewers requires managing a formal contract process.
• Faculty implementation of changes
  Documentation of the online peer review is included in a faculty member’s employment file.

Impact occurs when faculty incorporate feedback generated by the review process into the courses they teach. Faculty reflect on the recommendations they receive and consider ways to improve the design, delivery, and content of their courses. Faculty use the feedback to identify specific goals, action items, or plans to improve the course. The administrative team may assign a coach or mentor. After a period of one year, the faculty member provides a summary of changes they made as a result of the peer review process to the SON’s instructional designer, associate dean, and dean.

Incorporating an online peer review process is a valuable strategy for ensuring an online program’s continued success. Reviewing an online course and instructor in the different domains of design, delivery, and content from the perspective of a diverse group of people can be insightful and transformative. The thoughtful and detailed feedback gleaned from external reviewers has been exceptional and has given SON faculty a deeper, more nuanced view of how to improve their courses, while the adoption of regular collaborative faculty reviews has expanded the focus of curriculum reviews from individual courses to include mapping and threading key themes throughout the program. Most importantly, it has allowed faculty who teach exclusively online to collaborate with and learning from their peers.

Linda Merillat, PhD, is a lecturer and instructional designer, Bobbe Mansfield, DNP, is an associate dean, Marian Jamison, PhD, is a professor, Monica Scheibmeir, PhD, is a retired dean, and Jeanne Catanzaro, EdD, is an assistant professor and director of the RN-to-BSN program at Washburn University.