Blended Teaching Readiness: Phase 2 – Instrument Development

Executive Summary

Blended teaching involves combining online and in-person instruction.  There has been significant growth in the demand for blended learning in schools which has increased the need for teachers who know how to teach in a blended learning environment.

In Phase 1 of the study, a blended teaching readiness instrument containing four dimensions was tested and statistically validated (see Graham, Borup, Pulham, & Larsen, 2017).  We determined that the instrument was missing an important dimension related to management of the space and routines within the blended environment.  As a result, in Phase 2, the subject of this report, we added a fifth dimension of Management to the instrument for a total of 65 items.  We report here on the completed instrument and the work to have experts in the field, administrators, and teachers review the new items prior to testing it statistically with over 250 teachers.  The domains include the following five top-level constructs, each containing 2-4 sub-constructs:

1. Foundations
   1. Technological Literacy
   2. Digital Citizenship
   3. Dispositions
2. Planning
   1. Planning Blended Activities
   2. Planning Blended Assessments
3. Instructional Methods & Strategies
   1. Personalized Instruction
   2. Facilitating Student-Student Interaction
   3. Facilitating Student-Instructor Interaction
   4. Facilitating Student-Content Interaction
4. Evaluation & Assessment
   1. Implementing Blended Assessments
   2. Evaluating and Reflecting
5. Management
   1. Managing Blended Environments
   2. Managing Blended Routines

Additionally, we report on the early development of a shorter instrument focused specifically on teacher dispositions and the following five key pedagogical competencies:

1. Dispositions – beliefs and attitudes related to blended teaching and learning.
2. Online Integration – ability to effectively combine online instruction with in-person instruction.
3. Real-time Data Practices – ability to use digital tools to monitor student activity and performance in order to guide student growth.
4. Personalizing Instruction – ability to implement a learning environment that allows for student customization of goals, pace, and/or learning path.
5. Online Interaction – ability to facilitate online interactions with and between students.

While there is overlap between the two instruments, the shorter instrument organizes the blended teaching constructs in a way that we believe will be simpler for administrators and teachers to build into focused professional development.  We created the readiness instruments and a professional development guidebook on blended teaching competencies for open and free access by teachers, districts, and university educator preparation programs (See K-12 Blended Teaching: A Practical Guide for Teachers ).

Introduction and Background

As the demand for quality, flexible K-12 educational options increases, there is a growing need for teachers who are prepared to teach effectively in blended learning classrooms that combine elements of in-person and online instruction (Pulham & Graham, 2018; Pulham, Graham, & Short, 2018).  At the same time, the demands on teacher time and the needs for professional development are as high as they have ever been (Pulham & Graham, 2018; Pulham, Graham, & Short, 2018).  The purpose of both Phase 1 and Phase 2 of this project has been to help facilitate and evaluate this process by developing a blended teaching readiness instrument that can,

• help educators know what specific skills are necessary for success in a blended environment,
• allow educators to evaluate their readiness for teaching in a blended environment,
• provide information that would allow for a personalized approach to teacher professional development for acquiring blended teaching skills, and
• measure teachers’ change in readiness following targeted professional development.

This research and development has been conducted over a two-year period.  The 2017 MVLRI Phase 1 Report contains an overview and study details from Phase 1 of the project (Graham, Borup, Pulham, & Larsen, 2017).  Additionally, since that time, we have published two papers that do a systematic synthesis of the existing literature related to K-12 blended teaching competencies.

• Comparing K-12 online and blended teaching competencies: A literature review (Pulham & Graham, 2018)
• Generic vs. Modality-Specific Competencies for K-12 Online and Blended Teaching (Pulham, Graham, & Short, 2018)

The review of existing literature provided a foundation for the development of the blended teaching competency structures that are described in this report.  Readers are invited to review the resources linked above for a comprehensive overview of the project.

During Phase 2 of the project, we did the following:

• Added a Management construct to the Blended Teaching Readiness instrument with sub-constructs of (a) managing the blended learning environment and (b) managing blended learning routines for a total of 65 items.
• Solicited feedback on the new items from experts, district administrators, and teachers.
• Administered the survey to 258 current teachers and performed statistical tests to evaluate the fit of the competency model used.
• Developed a shorter instrument with 47 items focusing on four core pedagogical competencies needed for blended teaching: (1) online integration, (2) real-time data practices, (3) personalization, and (4) online interaction.
• Solicited feedback from experts and teachers on the items.

Instrument Development

Blended Teaching Competency Models

The blended teaching competency model that was initially developed during Phase 1 of the research had two levels as shown in Figure 1.  This model is what we have begun to refer to as the process model because its top-level structure is process-oriented (e.g., planning, methods, assessment/evaluation, management).  At the second-level, you begin to see constructs specific to blended teaching.  The process model was quite comprehensive, but also made it hard to emphasize some important pedagogical concepts to blended teaching, like real-time data practices or personalization, that had elements distributed across multiple constructs.  Additionally, the complexity inherent in a model with five top-level constructs, 13 sub-constructs, and 65 items, introduced measurement challenges as well as challenges to designing professional development aligned with each of the constructs.

In Phase 2 of the research, we continued to update and do the validation research for the process model, and we also conceptualized and began the development work for a pedagogical model that would focus on a small number of core blended teaching pedagogical skills.  While the pedagogical model would be less comprehensive, it would also be more focused and therefore more useful in a professional development context with limited time and resources.  Figure 2 shows the core constructs of the pedagogical model.

• Dispositions – beliefs and attitudes related to blended teaching and learning.
• Online Integration – ability to effectively combine online instruction with in-person instruction.
• Real-time Data Practices – ability to use digital tools to monitor student activity and performance in order to guide student growth.
• Personalizing Instruction – ability to implement a learning environment that allows for student customization of goals, pace, and/or learning path.
• Online Interaction – ability to facilitate online interactions with and between students.

Instrument Development Procedures

The steps below were used in the development of both process and pedagogical instruments.  The steps were also used for the blended teaching process model in Phase 1 and then with the Management items again in Phase 2.  During Phase 1 we determined that the instrument was missing an important aspect of implementation which deals with classroom management of the learning environment as well as routines that are often disrupted when moving to a blended approach.  Because of timing issues with our district partners, we were not able to delay administration of the Phase 1 instrument for the amount of time that was required for proper vetting of new items.  Therefore, the Management section was developed using the same process outlined below added to the instrument during Phase 2 of the research. For development of the pedagogical model we did not get feedback from district administrators (step 5) due to logistical and time constraints.

1. Reviewed existing frameworks – This consisted of the extensive analysis outlined previously along with familiarizing ourselves with the existing standards and frameworks related to blended learning and reviewing the competency items (Pulham & Graham, 2018; Pulham, Graham, & Short, 2018).
2. Developed a competency structure – In this step, we sought a balance between comprehensiveness and parsimony of the model (Whetten, 1989). Ultimately the process model leaned towards comprehensiveness while the pedagogical model leaned towards parsimony.
3. Developed items for constructs – In this step, we developed items for each of the constructs in the two models. As would be expected, there is some overlap between items in the two models.  We ended up with five items for each of the 13 constructs in the process model (65 total) and 8-11 items for each of the five constructs in the pedagogical model (47 items with the expectation of pruning the items even more after statistical tests were run).
4. Solicited external expert feedback – In this step we sought feedback from four to six expert reviewers who are top scholars and leaders in the area of K-12 online and blended learning. We made adjustments to items based on their feedback.
5. Solicited district administrator feedback – In this step, we received several rounds of feedback in both Phases 1 and 2 from a large school district’s blended and personalized learning instructional service team. We made adjustments to items based on their feedback.
6. Solicited practicing teacher feedback – Finally, we held interviews with a handful of current teachers at elementary, middle, and high school levels who followed a think aloud protocol to react to each item in the instrument in order to identify confusing language or jargon. Several minor changes were made to make items easier to understand.

The result of this process can be found in the latest versions of the two blended teaching readiness instruments found in Appendix A.  In Phase 2, the process instrument went through another round of statistical analysis with the new Management constructs and items that will be described below.  While the shorter pedagogical model instrument has been developed, it has yet to be administered to teachers.  As a result, the following sections will focus on the methods and findings regarding the process model.

Methods

In Phase 2 of this research, the Management construct and two sub-constructs were added to the process model that was successfully tested for fit in Phase 1 (See Figure 3). This section describes the data collection and analysis procedures that were conducted to determine if the management item works with the model.

The participants for Phase 2 of the process model developed in this research were 258 in-service teachers in a district in the eastern United States.  We had broad representation across teaching levels and subject areas.  The majority of participants had little to no blended teaching experience.  Table 1 summarizes the demographic profile of the Phase 1 and Phase 2 samples.

Data Collection

Data Analysis

We ran a confirmatory factor analysis (CFA) on theoretical structure as represented in Figure 3.  The models were tested in Mplus using established cutoffs for four different fit statistics (RMSEA < .08, CFI > .9, TLI > .9, SRMR < .08; Wang & Wang, 2012).  Each fit statistic makes slightly different assumptions about the structural models; achieving the cutoff levels for more fit statistics indicates a stronger model.  In the case that the model did not fit one or more of the indices, we planned to test each of the top-level factors individually for structural fit using the same CFA process.

Results

Process Model Fit

The CFA for the full theoretical process model (Figure 3) met two of the four fit statistics thresholds set forth in the literature (see Table 2).  These results were not as good as the Phase 1 model which met all four fit statistics thresholds.  However, they are acceptable considering the additional complexity of the model and large number of parameters being estimated in the model.  Factor loadings and communalities were examined (see Table A1 in Appendix A), and all factor loadings were statistically significant at the 0.01 level.  Additionally, only three of 65 items had communalities that were below 0.5, and those three items were in the previously validated portion of the Phase 1 instrument and are theoretically important so we chose to keep them in the model. (Worthington & Whittaker, 2006).

Each superfactor CFA (e.g., foundations, planning, methods, assessment/evaluation, management) was run separately and the fit statistics calculated (Table 3) to isolate any local misfit in each superfactor.  The fit statistics for all the superfactors were within acceptable ranges (with exceptions of RMSEA for the management and assessment factors).  This result gives further evidence that the lack of fit for the theoretical model has to do with model complexity rather than any model misspecification. (Wang & Wang, 2012). Thus, we determined the model to be defensible.  This also lends weight to our decision to begin development of a model that is less complex with fewer constructs and no second-order factors.

Discussion & Conclusion

This research will complete the validation work for a process model of the blended teaching readiness instrument.  We present data based on the first 258 respondents, which is a small N for the large number of items and constructs included in the model.  Nevertheless, we determined that the instrument was suitable for K-12 inservice teachers.  We plan to add an addendum to this report when additional data is received and expect that the additional data will add to the confidence we have in the validity of the instrument.  Validation work involved input from experts, district administrators, and practicing teachers, as well as statistical tests to determine the fit of the theoretical model.  Phase 2 of the research added 10 items related to managing a blended environment and blended routines.  These were deemed important additions recommended in Phase 1.  However, we were a little disappointed that while deemed adequate, the statistical fit of the new model was not as strong as the Phase 1 model.  There are many potential reasons for this fact.  The most likely reason is that the model was quite complex with five high-level constructs and 13 sub-constructs, and, at the time of this report, we only had 258 respondents.  A model of this complexity is quite challenging to test and typically requires a larger number of participants than we had access to.

In our initial work we opted for creating a model that tried to be fairly comprehensive from planning to evaluation.  However, we also recognized that the length, structure, and complexity of the process model instrument could limit its utility in some contexts.  We felt confident that we could capture the most essential skills in a simpler pedagogical model that focused on a smaller and highly relevant set of competencies.  We have presented these in this report as the pedagogical model.  We have used experts and teachers to help us develop items for a simplified blended teaching readiness instrument.  Future research will take the pedagogical model instrument through the statistical tests to further establish its validity for use with inservice as well as preservice teachers.

An important goal of this project has been to create tools and resources for measuring K-12 blended teaching readiness that are openly and freely available to individuals, school, districts, and universities who are trying to improve the blended teaching skills of K-12 teachers.  We feel that we have accomplished this important goal and we continue to do scholarly work that will advance this foundational research.

Below are the current resources that are available to you for measuring and developing K-12 blended teaching competencies.  Some possibilities for use of these resources include:

• A school district could administer the blended teaching readiness survey to teachers across the district to determine schools that are most prepared to begin a blended learning initiative or to identify teachers in each school who can be recruited to be prepared as blended teaching coaches for their schools.
• Teachers in a school take the blended teaching readiness survey as a pre-assessment, and then professional development activities using the guidebook are focused on greatest areas of need. Teachers could also be surveyed following the professional development to measure growth.
• Individual teachers wanting to learn more about blended teaching and upgrade their skills in this area could use the blended teaching readiness survey as a self-assessment and then use the guidebook as a source of classroom examples and implementation ideas in each blended teaching competency area.

References

Graham, C. R., Borup, J., Pulham, E., & Larsen, R. (2017). K-12 blended teaching readiness: Phase 1 – instrument development. Lansing, MI. Retrieved from https://michiganvirtual.org/wp-content/uploads/2017/11/k12-blended-teaching-readiness-phase-1-instrument-development.pdf

Pulham, E. B., & Graham, C. R. (2018). Comparing K-12 online and blended teaching competencies: A literature review. Distance Education, 39(3), 411–432. http://doi.org/10.1080/01587919.2018.1476840

Pulham, E., Graham, C. R., & Short, C. R. (2018). Generic vs. Modality-Specific Competencies for K-12 Online and Blended Teaching. Journal of Online Learning Research, 4(1), 33–52. Retrieved from https://www.learntechlib.org/j/JOLR/v/4/n/1/

Wang, J., & Wang, X. (2012). Structural equation modeling: Applications using Mplus. West Sussex, UK: John Wiley & Sons.

Whetten, D. A. (1989). What constitutes a theoretical contribution? The Academy of Management Review, 14(4), 490–495. Retrieved from https://www.jstor.org/stable/258554

Worthington, R. L., & Whittaker, T.A. (2006). Scale development research: A content analysis and recommendations for best practices. The Counseling Psychologist, 34(6), 806-838.

Appendix – Constructs and Items for Instrument