Intervening to align grader standards

Overview

I would suggest managing your interventions based on the audience you are addressing. The two strategies outlined below combined to work well for me, but your grader population may respond differently than ours.

In it's simplest form, my intervention was to start conversations with my graders, not to present an evaluation. Wherever possible I presented data and the team or individuals worked to interpret it. We worked through much of the information you found on the How to analyze grade data page. Our graders expressed that once they had a better understanding of assessment data they were able to assess more consistently and accurately. I would suggest that the implementation of this program helped to stake this burgeoning interest in assessment.

Notes before we start

in line with the caveats listed at the end of How to analyze grade data, keep in mind that inconsistent distributions between graders does not explicitly mean that these graders are applying different standards. Assessment standards are one of a range of issues that will effect the scores earned by students.

-----------------------------------

Intervention with your entire teaching team

Our course is based around active learning and inquiry so we employ these methods wherever possible with the training of our staff to model effective teaching techniques. I've found that our graduate TAs love data and are interested in thinking about data. This portion of our intervention played to this strength by facilitating a discussion where graders will analyze the work of their peers and self.

Tone

When you are editorializing and analyzing grade distributions you can take care to do so in a sensitive matter. In addition to calibrating the assessment standards of your graders you should also work to keep this conversation civil and educational. Your graders will work as a team; regardless of whether a TA is justified in pointing out the errors of a colleague, your team will function poorly if this TA creates discord with your group.

Privacy

One way to help implement a welcoming tone is to keep the identity of your graders private. If you have multiple graders you can assign each a number known only to them and use this as an alias. It should also be pointed out to graders that as they do not know the identity of the owner of an alias all of their critiques should be addressed sensitively.

Practice

I've found that it is effective to structure this as an opportunity to practice developing a tool. I return to the quote below at least a few times in a semester:

We are asking you to do something new, something that takes practice to perfect, and this job is important!

I structure the beginning and end of these exercises so that graders reflect on the fact that; I do not expect them to be experts at assessment, only that they strive to become better at assessment throughout a semester.

This meeting will help shape standards in this course

Your graders are not the only ones responsible for inconsistent standards. Oftentimes our standards are written in a way that we comprehend but that our graders do not. Emphasize that through this activity you wish to receive feedback from your graders to help make your rubrics and effective tool for their use. This will put nerves at ease, but the process of reviewing these categories will also prompt graders to better learn the rubrics that they're using.

Just operate with mutual respect and your graders will respond positively

-----------------------------------

Individual interventions with graders

We see many new graders initially exaggerate student performance for one of two reasons. First, some graders recognize that their students are not meeting the course's standards but give their students high marks because perceived as difficult graders, or as more difficult than their peers. Second, some graders perceive student performance in a course to be higher than occurs in actuality. How we deal with each of these issues will be discussed below.

Tone

Identifying "potentially" divergent grading standards does not mean that a grader has erred, it just suggest the possibility. These meetings should be a discussion about how to align a graders evaluations with the standards set in a class. I try to emphasize two way communication about the importance and utility of grading fairly.

Discuss how your grader perceived their class' progress; is there actually a problem?

Before this meeting you should take a look at individual feedback and evaluations by a grader and identify portions where you perceive students have been under or over evaluated.

Begin by querying how your grader perceived student success in their classroom. Ask what skills students ahve mastered, and some skills that they could improve on. Most graders that over-grade perceive that students are not meeting course goals in class but their assignment grades do not fit this reality.

Emphasize the benefits of evaluating based on standards, not normative grading

Many of the graders we meet with recognize that student performance that student performance does not meet our course standards. They tend to over-grade because the perceive that most students should receive "good" marks.

I explain to these graders that this practice to trying to find the median quality assignment and assign it a B or a C and extrapolating standards from there is called normative grading. We then discus that we assess our students based upon their ability to achieve the standards we've set for each assignment, not how they perform relative to their peers. This error typically occurs early in a semester when fewer of our students are meeting our standards. We emphasize to our TAs that an honest assessment of their students informs their students of what progress must be made in order to meet the class goals. Further we typically see our students excel towards the end of a semester; we evaluate more heavily here to assess what students are leaving our course with and our student grades tend to be acceptably aligned with what is expected of an introductory course.

Create a plan to align the graders evaluations with course standards

Before your grader leaves decide whether you want to set up practice grading assignments or reoccurring meetings to evaluate their assessments. Follow through on these plans and we see greater alignment of standards between sections.