Inter-Rater Reliability

A description of a strong teacher evaluation system is never complete without discussion on inter-rater reliability. Inter-rater reliability remains essential to the employee evaluation process to eliminate biases and sustain transparency, consistency, and impartiality (Tillema, as cited in Soslau & Lewis, 2014, p. 21). In addition, a data-driven system of evaluation creating a feedback-rich culture is considered best practice. Examination by school leadership of quantitative trend data and comparison of evaluators is essential to professional growth.  (Graham, Milanowski, & Miller, as cited in Soslau & Lewis, 2014, p. 39).

But what is inter-relater reliability, and how can we ensure it is executed with fidelity in our schools? In its simplest terms, inter-rater reliability requires multiple observations by various administrative staff members who have been trained in the evaluation process. The next level of inter-relater reliability happens when school leadership teams analyze and track analytical teacher evaluation data and ensure that observations are being completed with fidelity.

For a teacher evaluation program to exist without bias and be focused on professional growth, the following elements of inter-rater reliability must be nurtured continually:

1.) Training — Training for evaluators is not only essential in the beginning stages of establishing an employee evaluation system, but must be ongoing. Assurance of inter-rater reliability decreases biases and increases ethical practice in the evaluation process. Assigning a numerical value to teacher effectiveness based upon a predetermined rubric helps to decrease bias in the process.

2.) Multiple evaluators — While perhaps one of the most important elements of effective employee evaluation, using multiple evaluators has often been the most overlooked for practical reasons. This is challenging for small schools or companies with one administrator bearing the sole responsibility for employee evaluations. Leadership may need to consider using external evaluators to ensure inter-rater reliability and safeguard against bias in the evaluation process.

3.) System of documentation — Crucial to the process of teacher evaluation is documentation.  Regardless of the school district’s individual steps that have been established for employee evaluation, all correspondence and steps in the process require thorough documentation. Once this data is stored in an evaluation system or employee file, it needs to be available for review by all parties.

4.) Identify trends — Once trends among various evaluators have been identified, leadership teams should distinguish areas of strengths and weaknesses among evaluators and define characteristics present in specific instructional domains among teachers.  For example, a large number of ineffective ratings in a specific indicator in a building signals a need for school-wide professional development in this area.  

5.) Communication — What effective instruction looks like remains a matter of local control, and frequent conversations on this topic are helpful to the development of inter-rater reliability.  Norming activities that allow evaluators to co-evaluate either teaching videos or actual classroom visits and then compare results help evaluators become more consistent in their practices.

It is within all of us to want to do well.  While this statement is subjective, we must remember that the desire to improve, especially among those in-service professions such as teaching, is fundamental.

As educational leaders, we must ensure that evaluation practices are fair and consistent in order to establish trust among those being evaluated. Are we analyzing evaluator data to ensure that evaluators are consistently providing appropriate feedback? Are we establishing district-wide norms for what effective instruction looks like among our evaluators? When these criteria are met, we move beyond compliance to providing fair, consistent evaluation and feedback for teachers and ultimately improving instruction for our students.
Author

Further Reading

1. Statistics How To – Inter-rater Reliability IRR: Definition, Calculation
2. EdWeek –Gates Ends Investment in Teacher Evaluation: What That Means for the Field
3. UDK – Center for Teaching Excellence awarded grant to help improve faculty evaluations