Unfairness in Testing

We frequently hear that testing is unfair. In multiple-choice examinations, the complaints are numerous, but there are other, less obvious sources of unfairness that may widen the gap between high-performing and struggling students. Indeed, some degree of unfairness may be inherent in almost every form of testing. This problem extends well beyond K–20 education. It applies to driver's examinations, medical board examinations, and competitive examinations for government positions, including G-12 positions.

The fundamental question is whether tests measure what they are intended to measure, or whether they also measure other abilities that give some examinees an unintended advantage. Another key point may be that testing can magnify differences in ability rather than merely measure them.

1. The Strategic Advantage of Good Students

Good students are often better at distinguishing difficult questions from easy ones. This ability allows them to allocate their time more effectively. They may answer the easier questions first, postpone difficult ones, and return to them if time permits. A struggling student may be less able to recognize which questions deserve immediate attention and which should be deferred.

Consequently, two students with similar knowledge of a subject might earn different scores because one is better at managing the examination itself. The test therefore measures not only subject knowledge but also the ability to develop an effective testing strategy.

Good students may also be better at guessing. Their guesses are often not entirely random; they may recognize patterns, eliminate implausible answers, recall related information, or make more informed inferences. When they answer incorrectly, their errors may reveal a more sophisticated misunderstanding than the errors of struggling students.

This raises an important question about distractors, the incorrect options included in multiple-choice questions. A well-designed distractor may expose a misconception, but it may also favor students who are better at recognizing and eliminating misleading alternatives. In this sense, questions designed to identify misconceptions may inadvertently disadvantage struggling students.

The problem is not necessarily that such questions are poorly constructed. It is that the same question may measure different combinations of knowledge, reasoning, and test-taking skill in different students.

2. Should Scoring Account for These Differences?

If good students are better at eliminating incorrect distractors, should their correct answers receive greater credit? One might argue that a correct answer reached through sophisticated reasoning demonstrates more knowledge than an answer obtained through a lucky guess.

However, awarding different amounts of credit based on presumed ability would be difficult to justify in practice. Examiners generally cannot determine with confidence how a particular answer was obtained. Moreover, if the purpose of a test is to assess subject knowledge, the fairest approach may be to improve the questions rather than adjust scores according to assumptions about the examinee.

A more promising alternative is to distinguish between knowing the material and knowing how to take a test. If both abilities are educational objectives, they should be measured explicitly rather than allowing one to influence the measurement of the other unintentionally.

3. Is a Fair Test Possible?

The fairest testing format I have encountered is the oral examination, particularly when conducted by two or three examiners who make a deliberate effort to balance the pressure of the examination with a sympathetic determination to discover what the student actually knows. Clear criteria for passing are essential.

The examination begins with a relatively simple question that most adequately prepared students should be able to answer. The examiner then progressively increases the difficulty until the student gives an incorrect answer or can no longer respond. The examiner moves to another topic and repeats the process. In this way, the examination explores the limits of the student's knowledge across several areas rather than relying on a fixed collection of questions.

Such examinations can provide a nuanced assessment of understanding, especially when examiners use consistent criteria and allow students to explain their reasoning. They are not, however, free from bias.

Oral examinations may favor students with rapid recall, verbal fluency, confidence, and an ability to infer what an examiner is seeking. Shy or easily intimidated students may perform below their actual level of knowledge. The phenomenon associated with Clever Hans, the horse that appeared to perform calculations but was responding to subtle cues from observers, offers a reminder that assessments can be influenced by signals that neither the examiner nor the examinee fully recognizes.

Oral examinations also require considerable time, trained examiners, and consistent procedures. Developing the necessary expertise may take years, and students accustomed to conventional examinations may struggle with the format. What appears fair in principle may therefore be difficult to implement consistently on a large scale.

Setting aside oral exams, I have seen reasonably fair tests made. At my college, we teach multiple calculus sections. Fairness in testing across sections was almost impossible until all the instructors worked together to build a common test. We submitted our questions, and the group considered them to produce an exam. This improved fairness and also fixed vagueness and grammar problems.

4. Improving Multiple-Choice Mathematics Tests

Several measures might reduce unintended advantages in multiple-choice mathematics examinations.

First, distractors should be carefully constructed. No incorrect option should be so obviously implausible that it can be eliminated without understanding the mathematics. Distractors should represent credible errors or alternative approaches, but they should not depend on obscure tricks or superficial clues. This may substantially increase the time required to construct a good test, particularly when questions have no simple, objective answer. Nevertheless, the additional effort may produce a more informative assessment.

There is an important qualification: plausible distractors can help identify misconceptions, but they should not be designed merely to trick students. A test should distinguish genuine mathematical understanding from misunderstanding, not reward the ability to detect the examiner's traps.

Second, consider indicating the difficulty of each question. Labels such as easy, medium, and hard might help struggling students allocate their time and avoid spending too long on a question beyond their present abilities. Yet such labels could also help stronger students optimize their testing strategy. The net effect on the performance gap is uncertain and would need to be investigated empirically.

The central objective should be to determine whether these practices improve the validity and fairness of scores, not to assume that any particular adjustment will benefit one group without affecting another.

5. The Psychology of Struggling Students

Testing practices cannot be fully understood without considering how students interpret their own performance.

Some struggling students do not carefully review their examinations after receiving their grades. They may place the test in a folder and never examine it again. As a result, they lose an important opportunity to discover why they made mistakes, correct misconceptions, and develop better strategies for future examinations.

Another possibility is that some students attribute the success of stronger students primarily to luck rather than ability or preparation. The metaphor of a blind person being unable to imagine what a sighted person sees captures the difficulty of recognizing abilities that one does not personally possess.

A student who does not understand how successful problem-solving works may interpret a correct answer as a fortunate guess rather than evidence of growing competence. Conversely, an incorrect answer may be attributed to bad luck rather than an identifiable error in reasoning.

This outlook can influence study habits. If students fail to distinguish between knowledge, strategy, and chance, they may see little value in reviewing their mistakes. When a test contains deceptive distractors, it may reinforce their belief that success depends on luck rather than on skills they can develop.

This is not a universal description of struggling students. Their difficulties may also reflect inadequate instruction, limited preparation time, anxiety, gaps in prior knowledge, or other circumstances. Nevertheless, the failure to learn from mistakes is an important problem that educators should address.

Conclusion

Perfectly fair testing may be unattainable, but more defensible testing is possible. The first step is to recognize that examination scores can reflect several abilities simultaneously: subject knowledge, reasoning, memory, verbal fluency, strategic judgment, and familiarity with testing conventions.

The challenge is to determine which abilities a particular examination is intended to measure and to minimize the influence of those that are irrelevant to its purpose. This requires better question design, clearer scoring criteria, opportunities for students to learn from their errors, and research into the effects of alternative testing practices.

Ultimately, the goal should not be to eliminate every difference between students. It should be to ensure that the differences reflected in their scores are as closely related as possible to the knowledge and abilities the examination is intended to assess.

 

©2026 G Donald Allen