Standard setting for written examinations
Standard setting refers to the process of demarcating the minimal level of knowledge required e.g. identifying the level that is “just good enough”. Standard setting is performed to decide the cut off score for passing or failing an examination.
As there is no clearly defined level of medical “competence”, standards are based on an expression of judgement by a panel of experts, i.e. faculty members who have experience in the area being studied and are familiar with what is taught to candidates at that level.
Standard setting can be absolute or relative.
Relative (norm) standard setting is based on a group of standard setters deciding what proportion of candidates should pass and averaging this to give an overall proportion, thus the passmark would be determined relative to the performance of the overall group, e.g. the lowest 10% of students in the cohort could be classed as failing by this method.
Absolute (criterion) standard setting refers to relating performance to a set requirement, and this is the method utilised for examinations at the University of Aberdeen. Absolute methods can be either examinee-centred or test-centred.
Examinee-centred methods are utilised in determining the passmark for OSCE examinations (see standard setting for OSCE examinations) whereas test-centred methods are utilised for written examinations.
The exact method utilised at the University of Aberdeen for standard setting in written examinations is called the Angoff method. This involves a group of clinicians or relevant persons meeting, where each individually specifies what they would expect of a borderline candidate for each question. The mean mark is then calculated for each question and summed to give the overall passmark for the examination.
For Single Best Answer questions, each standard setter estimates the percentage likelihood of a borderline candidate responding correctly. The scores for each question are summed and averaged. This is performed for every question in the paper and the mean scores are summed to give the overall passmark.