Skip to main content
placeholder image

Assessment of Psychology Competencies in Field Placements: Standardized Vignettes Reduce Rater Bias

Journal Article


Download full-text (Open Access)

Abstract


  • Supervisors' ratings of psychology trainees' competence in field settings are a critical component of training assessment. There has been little systematic research regarding the validity of these assessments, but the available evidence suggests we have a problem! Supervisors' judgments may be affected by systemic biases that pose a serious threat to assessment credibility. The current study is part of a research collaboration among six universities that endeavors to develop and evaluate a new method the use of vignettes against outcomes derived from a conventional rating scale. Individual vignettes were designed and subjected to a rigorous process of peer-review and revisions, before final vignettes were assigned calibration scores by a group of experts. A catalogue of vignettes (n = 41) that represent various domains of competence across several developmental stages was compiled. University and field supervisors used the conventional rating scale and the vignette-matching procedure (VMP) to evaluate competencies at end-placement. Data from a pilot (n = 20) and a follow-up study (n = 57) suggest that compared with a conventional rating scale, the VMP reduced leniency and halo biases. The VMP has the potential to improve outcomes of competency assessments in field placements and merits further research and development.

UOW Authors


  •   Gonsalvez, Craig J. (external author)
  •   Bushnell, John (external author)
  •   Blackman, Russell J. (external author)
  •   Deane, Frank
  •   Bliokas, Vida
  •   Nicholson-Perry, Kathryn (external author)
  •   Shires, Alice (external author)
  •   Nasstasia, Yasmina (external author)
  •   Allan, Christopher D. (external author)
  •   Knight, Roslyn (external author)

Publication Date


  • 2013

Citation


  • Gonsalvez, C. J., Bushnell, J., Blackman, R., Deane, F. P., Bliokas, V., Nicholson-Perry, K., Shires, A., Nasstasia, Y., Allan, C. & Knight, R. (2013). Assessment of Psychology Competencies in Field Placements: Standardized Vignettes Reduce Rater Bias. Training And Education In Professional Psychology, 7 (2), 99-111.

Scopus Eid


  • 2-s2.0-84881264196

Ro Full-text Url


  • http://ro.uow.edu.au/cgi/viewcontent.cgi?article=1267&context=sspapers

Ro Metadata Url


  • http://ro.uow.edu.au/sspapers/268

Has Global Citation Frequency


Number Of Pages


  • 12

Start Page


  • 99

End Page


  • 111

Volume


  • 7

Issue


  • 2

Place Of Publication


  • United States

Abstract


  • Supervisors' ratings of psychology trainees' competence in field settings are a critical component of training assessment. There has been little systematic research regarding the validity of these assessments, but the available evidence suggests we have a problem! Supervisors' judgments may be affected by systemic biases that pose a serious threat to assessment credibility. The current study is part of a research collaboration among six universities that endeavors to develop and evaluate a new method the use of vignettes against outcomes derived from a conventional rating scale. Individual vignettes were designed and subjected to a rigorous process of peer-review and revisions, before final vignettes were assigned calibration scores by a group of experts. A catalogue of vignettes (n = 41) that represent various domains of competence across several developmental stages was compiled. University and field supervisors used the conventional rating scale and the vignette-matching procedure (VMP) to evaluate competencies at end-placement. Data from a pilot (n = 20) and a follow-up study (n = 57) suggest that compared with a conventional rating scale, the VMP reduced leniency and halo biases. The VMP has the potential to improve outcomes of competency assessments in field placements and merits further research and development.

UOW Authors


  •   Gonsalvez, Craig J. (external author)
  •   Bushnell, John (external author)
  •   Blackman, Russell J. (external author)
  •   Deane, Frank
  •   Bliokas, Vida
  •   Nicholson-Perry, Kathryn (external author)
  •   Shires, Alice (external author)
  •   Nasstasia, Yasmina (external author)
  •   Allan, Christopher D. (external author)
  •   Knight, Roslyn (external author)

Publication Date


  • 2013

Citation


  • Gonsalvez, C. J., Bushnell, J., Blackman, R., Deane, F. P., Bliokas, V., Nicholson-Perry, K., Shires, A., Nasstasia, Y., Allan, C. & Knight, R. (2013). Assessment of Psychology Competencies in Field Placements: Standardized Vignettes Reduce Rater Bias. Training And Education In Professional Psychology, 7 (2), 99-111.

Scopus Eid


  • 2-s2.0-84881264196

Ro Full-text Url


  • http://ro.uow.edu.au/cgi/viewcontent.cgi?article=1267&context=sspapers

Ro Metadata Url


  • http://ro.uow.edu.au/sspapers/268

Has Global Citation Frequency


Number Of Pages


  • 12

Start Page


  • 99

End Page


  • 111

Volume


  • 7

Issue


  • 2

Place Of Publication


  • United States