Development of a Self-Evaluation Instrument with Programmatic Assessment Components for Undergraduate Medical Students
Dina Qurratu Ainin
,
Yoyo Suhoyo
,
Artha Budi Susila Duarsa
,
Mora Claramita
This study aimed to develop and test a student self-assessment instrument based on the programmatic assessment (PA) components. We applied a series of.
- Pub. date: April 15, 2023
- Pages: 649-662
- 414 Downloads
- 890 Views
- 2 Citations
This study aimed to develop and test a student self-assessment instrument based on the programmatic assessment (PA) components. We applied a series of psychometric research methods by (a) conducting a literature study to find PA constructs, (b) developing the students' self-questionnaires, (c) ensuring content validity, (d) testing face validity, and (e) conducting reliability tests that involve medical students, medical teachers, medical educationalist, and an international PA expert. Face validity (readability test) was conducted with 30 medical students from an Indonesian university who were in their last year of pre-clinical education and had average scores above or equal to their classmates. The confirmatory factor analysis (CFA) was used to report the instruments’ validity and reliability. The final instrument was tested on 121 medical students with excellent GPAs from another medical school with a middle-level accreditation. The PA consists of five components: ‘learning activities’, ‘assessment activities’, 'supporting activities’, 'intermediate evaluations’, and ‘final evaluations'. These components are conveyed through 41 relevant statements with a four-point Likert scale and three yes/no statements. According to the respondents, there was a lack of 'supporting activities' and 'intermediate evaluation' components in the PA in their universities. This study has developed and tested a five-component evaluation instrument based on medical students' perceptions regarding PA implementation.
Keywords: Instrument development, medical education, programmatic assessment.
References
Ainin, D. Q. (2018). Penyusunan instrumen evaluasi penerapan programmatic assessment di institusi pendidikan kesehatan [Development of evaluation instruments for implementation of programmatic assessment in health professions education institutions] (Publication No. 164087) [Master's thesis, Universitas Gadjah Mada]. UGM Campus Repository. http://bit.ly/3RuRTe0
Ajzen, I. (1991). The theory of planned behavior. Organizational Behavior and Human Decision Processes, 50(2), 179–211. https://doi.org/10.1016/0749-5978(91)90020-T
Amin, Z., Seng, C. Y., & Eng K. H., (2006). Practical guide to medical student assessment. World Scientific. https://doi.org/10.1142/9789812773586
Attard, A., di Iorio, E., Geven, K., & Santa, R. (2010). Student-centred learning - Toolkit for students, staff and higher education institutions in learning. The European Students’ Union Student-Centre.
Bok, H. G. J., de Jong, L. H., O’Neill, T., Maxey, C., & Hecker, K. G. (2018). Validity evidence for programmatic assessment in competency-based education. Perspectives on Medical Education, 7(6), 362–372. https://doi.org/10.1007/s40037-018-0481-2
Cook, D. A., & Beckman, T. J. (2006). Current concepts in validity and reliability for psychometric instruments: Theory and application. American Journal of Medicine, 119(2), 166.e7-166.e16. https://doi.org/10.1016/j.amjmed.2005.10.036
Davis, A. E. (1996). Instrument development: Getting started. Journal of the American Association of Neuroscience Nurses, 28(3), 204–207. https://doi.org/10.1097/01376517-199606000-00009
Driessen, E. W., van Tartwijk, J., Govaerts, M., Teunissen, P., & van der Vleuten, C. P. M. (2012). The use of programmatic assessment in the clinical workplace: A Maastricht case report. Medical Teacher, 34(3), 226–231. https://doi.org/10.3109/0142159X.2012.652242
Heeneman, S., de Jong, L. H., Dawson, L. J., Wilkinson, T. J., Ryan, A., Tait, G. R., Rice, N., Torre, D., Freeman, A., & van der Vleuten, C. P. M. (2021). Ottawa 2020 consensus statement for programmatic assessment – 1. Agreement on the principles. Medical Teacher, 43(10), 1139–1148. https://doi.org/10.1080/0142159X.2021.1957088
Heeneman, S., Pool, A. O. , Schuwirth, L. W. T., van der Vleuten, C. P. M., & Driessen, E. W. (2015). The impact of programmatic assessment on student learning: Theory versus practice. Medical Education, 49(5), 487–498. https://doi.org/10.1111/medu.12645
Hofstede, G. (2011). Dimensionalizing cultures: The hofstede model in context. Online Readings in Psychology and Culture, 2(1), 1–26. https://doi.org/10.9707/2307-0919.1014
Kerlinger, F. N. (1986). Foundations of behavioral research (3rd ed.). Holt, Rinehart and Winston.
Lacasse, M., Renaud, J. S., Côté, L., Lafleur, A., Codsi, M. P., Dove, M., Pélissier-Simard, L., Pitre, L., & Rheault, C. (2022). Développement et mise à l’essai du guide de rétroaction francophone pour l’observation directe des résidents en médecine familiale au Canada [Feedback guide for direct observation of family medicine residents in Canada: A francophone tool]. Canadian Medical Education Journal, 13(1), 29–54. https://doi.org/10.36834/cmej.72587
Martinez, M. E. , & Lipson, J. I. (1989). Assessment for learning. Educational Leadership, 46(7), 73–75.
McComas, W. F. (Ed.). (2014). Summative assessments bt - the language of science education: An expanded glossary of key terms and concepts in science teaching and learning. SensePublishers. https://doi.org/10.1007/978-94-6209-497-0_93
Netemeyer, R. G., Bearden, W., & Sharma, S. (2003). Scaling procedures: Issues and applications. SAGE Publications. https://doi.org/10.4135/9781412985772
Perry, M., Linn, A., Munzer, B. W., Hopson, L., Amlong, A., Cole, M., & Santen, S. A. (2018). Programmatic assessment in emergency medicine: Implementation of best practices. Journal of Graduate Medical Education, 10(1), 84–90. https://doi.org/10.4300/JGME-D-17-00094.1
Said, H., Badru, B. B., & Shahid, M. (2011). Confirmatory factor analysis (CFA) for testing validity and reliability instrument in the study of education. Australian Journal of Basic and Applied Sciences, 5(12), 1098-10103. https://bit.ly/3EE2VIO
Schut, S., Driessen, E., van Tartwijk, J., van der Vleuten, C., & Heeneman, S. (2018). Stakes in the eye of the beholder: An international study of learners’ perceptions within programmatic assessment. Medical Education, 52(6), 654–663. https://doi.org/10.1111/medu.13532
Shumway, J. M., & Harden, R. M. (2003). AMEE Guide No. 25: The assessment of learning outcomes for the competent and reflective physician. Medical Teacher, 25(6), 569–584. https://doi.org/10.1080/0142159032000151907
Torre, D. M., Schuwirth, L. W. T., & van der Vleuten, C. P. M. (2020). Theoretical considerations on programmatic assessment. Medical Teacher, 42(2), 213–220. https://doi.org/10.1080/0142159X.2019.1672863
van der Vleuten, C., Lindemann, I., & Schmidt, L. (2018). Programmatic assessment: The process, rationale and evidence for modern evaluation approaches in medical education. The Medical Journal of Australia, 209(9), 386–388. https://doi.org/10.5694/mja17.00926
van der Vleuten, C. P. M., Schuwirth, L. W. T., Driessen, E. W., Dijkstra, J., Tigelaar, D., Baartman, L. K. J., & van Tartwijk, J. (2012). A model for programmatic assessment fit for purpose. Medical Teacher, 34(3), 205–214. https://doi.org/10.3109/0142159X.2012.652239
van der Vleuten, C. P. M., Schuwirth, L. W. T., Scheele, F., Driessen, E. W., & Hodges, B. (2010). The assessment of professional competence: Building blocks for theory development. Best Practice and Research: Clinical Obstetrics and Gynaecology, 24(6), 703–719. https://doi.org/10.1016/j.bpobgyn.2010.04.001
Wass, V., van der Vleuten, C., Shatzer, J., & Jones, R. (2001). Assessment of clinical competence. Lancet, 357(9260), 945–949. https://doi.org/10.1016/S0140-6736(00)04221-5
Wilkinson, T. J., & Tweed, M. J. (2018). Deconstructing programmatic assessment. Advances in Medical Education and Practice, 9, 191–197. https://doi.org/10.2147/AMEP.S144449