Skip to main content
placeholder image

Cut-scores revisited: feasibility of a new method for group standard setting

Journal Article


Download full-text (Open Access)

Abstract


  • Background: Standard setting is one of the most contentious topics in educational measurement. Commonly-used

    methods all have well reported limitations. To date, there is not conclusive evidence suggesting which standard

    setting method yields the highest validity.

    Methods: The method described and piloted in this study asked expert judges to estimate the scores on a real

    MCQ examination that they consider indicated a clear pass, clear fail, and pass mark for the examination as a whole.

    The mean and SD of the judges responses to these estimates, Z scores and confidence intervals were used to

    derive the cut-score and the confidence in it.

    Results: In this example the new method’s cut-score was higher than the judges’ estimate. The method also yielded

    estimates of statistical error which determine the range of the acceptable cut-score and the estimated level of confidence

    one may have in the accuracy of that cut-score.

    Conclusions: This new standard-setting method offers some advances, and possibly advantages, in that the decisions

    being asked of judges are based on firmer constructs, and it takes into account variation among judges.

UOW Authors


  •   Shulruf, Boaz (external author)
  •   Coombes, Lee (external author)
  •   Damodaran, Arvin (external author)
  •   Freeman, Adrian (external author)
  •   Jones, Philip (external author)
  •   Lieberman, Steve (external author)
  •   Poole, Phillippa (external author)
  •   Rhee, Joel
  •   Wilkinson, Tim (external author)
  •   Harris, Peter (external author)

Publication Date


  • 2018

Citation


  • Shulruf, B., Coombes, L., Damodaran, A., Freeman, A., Jones, P., Lieberman, S., Poole, P., Rhee, J., Wilkinson, T. & Harris, P. (2018). Cut-scores revisited: feasibility of a new method for group standard setting. BMC Medical Education, 18 (1), 126-1-126-8.

Scopus Eid


  • 2-s2.0-85048192807

Ro Full-text Url


  • https://ro.uow.edu.au/cgi/viewcontent.cgi?article=1254&context=smhpapers1

Ro Metadata Url


  • http://ro.uow.edu.au/smhpapers1/250

Start Page


  • 126-1

End Page


  • 126-8

Volume


  • 18

Issue


  • 1

Place Of Publication


  • United Kingdom

Abstract


  • Background: Standard setting is one of the most contentious topics in educational measurement. Commonly-used

    methods all have well reported limitations. To date, there is not conclusive evidence suggesting which standard

    setting method yields the highest validity.

    Methods: The method described and piloted in this study asked expert judges to estimate the scores on a real

    MCQ examination that they consider indicated a clear pass, clear fail, and pass mark for the examination as a whole.

    The mean and SD of the judges responses to these estimates, Z scores and confidence intervals were used to

    derive the cut-score and the confidence in it.

    Results: In this example the new method’s cut-score was higher than the judges’ estimate. The method also yielded

    estimates of statistical error which determine the range of the acceptable cut-score and the estimated level of confidence

    one may have in the accuracy of that cut-score.

    Conclusions: This new standard-setting method offers some advances, and possibly advantages, in that the decisions

    being asked of judges are based on firmer constructs, and it takes into account variation among judges.

UOW Authors


  •   Shulruf, Boaz (external author)
  •   Coombes, Lee (external author)
  •   Damodaran, Arvin (external author)
  •   Freeman, Adrian (external author)
  •   Jones, Philip (external author)
  •   Lieberman, Steve (external author)
  •   Poole, Phillippa (external author)
  •   Rhee, Joel
  •   Wilkinson, Tim (external author)
  •   Harris, Peter (external author)

Publication Date


  • 2018

Citation


  • Shulruf, B., Coombes, L., Damodaran, A., Freeman, A., Jones, P., Lieberman, S., Poole, P., Rhee, J., Wilkinson, T. & Harris, P. (2018). Cut-scores revisited: feasibility of a new method for group standard setting. BMC Medical Education, 18 (1), 126-1-126-8.

Scopus Eid


  • 2-s2.0-85048192807

Ro Full-text Url


  • https://ro.uow.edu.au/cgi/viewcontent.cgi?article=1254&context=smhpapers1

Ro Metadata Url


  • http://ro.uow.edu.au/smhpapers1/250

Start Page


  • 126-1

End Page


  • 126-8

Volume


  • 18

Issue


  • 1

Place Of Publication


  • United Kingdom