Learning Sciences International Blog Updates

Keep up with news about Dr. Marzano’s Teacher and Leadership Evaluation Models -- articles, state launches, and more.

Use the archive menu on the right to view previous releases.

Nov. 10, 2011

Building Inter-rater Reliability and Accuracy into a System of Teacher Observation

By: Stacy Holland - PDS Product Development Manager

The 2002 No Child Left Behind Act and, more recently, President Obama’s Race to the Top have focused teacher observation on the idea of growing teacher effectiveness.  Many recent studies have supported this focus, connecting high quality, effective teaching to increases in student achievement (Goldhaber, 2002; Hanushek, Kain, O’Brien, & Rivkin, 2005).

Certain concerns arise when considering the link between observation and growing teacher effectiveness.  How do we know which teaching strategies are tied most closely to student achievement?  How can we be sure that the observers are able to identify a strategy in use?  How can we be sure that the observers are able to rate the use of identified strategies?  How can we be sure that all observers within a building or district are seeing things the same way?

In his book The Art and Science of Teaching (Marzano, 2007), Marzano lays out the research supporting 41 categories of instructional strategies and shows why these strategies are considered to be “high probability.” He identifies which teaching strategies are tied most closely to student achievement, noting the studies he examined and showing the context of each.  He makes the connection between growing teacher practice and the student achievement impact that is highly probable when teachers use specific strategies in the correct lesson segment, with a specific group of students.

For an observer to consistently identify use of specific strategies as well “rate” the use of those strategies and provide accurate feedback to teachers, the observer must participate in a system that promotes growth of practice. Systems that incorporate aspects of the Deming Cycle (Plan, Do, Check, Act) while making considerations for adult learning behaviors help foster professional growth and continuous improvement.  Domain I training provided by Learning Sciences International takes participants through a series of activities and training to help teachers self-reflect about their personal use of the 41 categories of instructional strategies and helps supervisors identify these strategies in use.
Within the Art and Science of Teaching framework and through the use of the Marzano Resource Library in iObservation, observers are encouraged to both grow their ability to identify when strategies are being used as well as to grow their ability to provide teachers with accurate feedback on their use of the strategies.  Observers can view videos with commentary by Dr. Marzano.  In the introduction to each video Dr. Marzano explains what the observer will see.  The clip then shows the classroom teacher using the strategy explained followed by additional commentary by Dr. Marzano.  These clips help observers walk through the process of what to look for, as well as to determine the level of use of each strategy.  Observers are encouraged to use the Resource Library to grow their personal practice.

To ensure that all observers are on the same page in their ability to identify and rate observations with consistency, Learning Sciences suggests that within any system of observation observers take the time to view teacher practice together and then discuss what each person observed. Two of the techniques we use in our training to improve inter-rater reliability is the use of video observation and performing instructional rounds.

During our facilitated training, groups of observers use the videos to view and rank teachers on chosen elements and then participate in discussions about the evidence they collected as well as why teachers were rated at a certain level on the scale.  As a practice for continued improvement, teams of observers use instructional rounds to grow their inter-rater reliability.  To do this, teams of observers go into a teacher’s classroom with the intent to look for evidences of a specific element.  The observers mark the evidences and then leave the room to discuss what they saw together,.  Again, they talk about the evidences they collect and the rationale they use to assign the score.  The intent of the instructional round in this case is for inter-rater reliability only and is not intended to give feedback to the teacher.

The important thing to understand is that there are many parts of a process that must work together in order to help teachers through their continuum of growth connected to student achievement. While observation systems themselves are part of a plan for systematic teacher improvement, there are interconnecting processes that should also be improving systematically.  The more observers work through these processes, the more accurate the feedback they can provide to teachers.  And if all observers at a site are able to talk through the processes together, the better their inter-rater reliability will be.  Teachers need observers to have high levels of both to get the most out of their systems of improvement.

Goldhammer, R. (1969). Clinical supervision: Special methods for the supervision of teachers. New York, NY: Holt, Rinehart and Winston.

Goldhaber, D. (2002). The mystery of good teaching: Surveying the evidence on student achievement and teachers’ characteristics. Education Next, 2(1), 50-55.

Hanushek, E. A., Kain, J. F., O’Brien, D. M., & Rivkin, S. G. (2005). The market for teacher quality (NBER Working Paper [1]54). Cambridge, MA: National Bureau of Economic Research.

Marzano, R. (2007).  The art and science of teaching: a comprehensive framework for effective instruction. Alexandria, VA: ASCD.

Visit MarzanoEvaluation.com and learn more about our Teacher Evaluation Implementation and Redevelopment Services.

Page 1 of 1 pages

News Updates Home

Subscribe to the Learning Sciences International RSS Feed


Use the Archive menu to select and view previous releases.

    -Nov. 2, 2012: Teacher Evaluation: What’s Fair? What’s Effective?
    -Jul. 10, 2012: Oklahoma State Board of Education Approves Marzano School Leadership Evaluation Model
    -May. 8, 2012: Education Reform & Student Achievement: Learning Sciences Marzano Center Announces National Marzano
    -Apr. 12, 2012: Learning Sciences Marzano Center Launches as School Leadership and Teacher Evaluation Resource for N
    -Jan. 31, 2012: Marzano Causal Teacher Evaluation Model Recommended by Oklahoma Department of Education
    • Marzano Causal Teacher Evaluation Model Recommended by Oklahoma Department of Education

    -Jan. 11, 2012: Dr. Marzano Discusses Research Base and Insights from New Studies
    • New Live Webinar with Dr. Marzano 1/18 at 3pm EDT

    -Dec. 8, 2011: Informal Observations – It’s not a “Gotcha” Tactic
    -Nov. 10, 2011: Building Inter-rater Reliability and Accuracy into a System of Teacher Observation
    -Nov. 2, 2011: Teacher Evaluation with Dr. Robert Marzano and Boone County
    -Oct. 31, 2011: Dr. Marzano Webinar: Measuring and Improving Teacher Effectiveness Through Teacher Observation
    • Dr. Marzano joins us as a co-presenter to provide an overview of the Marzano Causal Teacher Evaluation Model

    -Oct. 21, 2011: Celebrating Teaching through a Comprehensive Evaluation Model
    -Oct. 10, 2011: Dr. Marzano’s Evaluation Model alignment to Arizona’s Framework for Measuring Educator Effectiveness
    -Aug. 17, 2011: LSI Welcomes Washington Districts Seeking to Pilot the Marzano Causal Teacher Evaluation Model
    -Aug. 15, 2011: Technology Platform Supports Two NYSED-Approved Rubrics
    -Jul. 21, 2011: NYSED Approves Dr. Robert Marzano’s Teacher Evaluation Model
    -Jun. 20, 2011: New Jersey Department of Education Approves Dr. Robert Marzano
    -Feb. 16, 2011: LSI Selected by the Florida DOE to Implement a Statewide Teacher Evaluation System