When are groups of graders wise?

Abstract: Juries, committees and expert panels get together to appraise all manner of things. Often they do so by awarding grades: qualitative evaluative predicates such as Excellent, Average, and Poor.

Certain features of grades tend to enhance a “crowd wisdom” effect, in which collective judgements improve on individual ones. Grades are coarse grained, enabling timely and accurate individual inputs. Also, they are absolute judgements of merit, not comparative ones. This means we can aggregate the several grades assigned to any one thing by taking the median; and that, it turns out, is enough to dodge certain problems about aggregating ordinal information, familiar from social choice theory. Another prominent feature tends to pull in the other direction, though. Grades are contextual in the sense that thresholds for assigning them vary from person to person and, for any given person, they vary from case to case. This may be expected to create a sort of equivocation among members of a group and to degrade crowd wisdom.

We’ll see that diversity among thresholds can indeed cause juries, committees and expert panels to make mistakes. Furthermore, uncertainty about the extent of this diversity can deprive collective judgements of all content. This claim can be made precise and rigorous as a corollary of Arrow’s “impossibility” theorem. Even so, as we’ll also see, collective grading can be a good way to appraise things if complete reliability is not needed. A computer simulation shows strong crowd effects even when thresholds vary at random. It seems that variety in standards can even improve the judgements of the group, if for every tough grader there’s another who is correspondingly easy.