Study Item Parameters of Classical and Modern Theory of Differential Aptitude Test: Is it Comparable?
Farida Agus Setiawati , Rizki Nor Amelia , Bambang Sumintono , Edi Purwanta
This study aimed to find the Classical Test Theory (CTT) and Modern Test Theory (MTT) item parameters of the Differential Aptitude Test (DAT) and exam.
- Pub. date: April 15, 2023
- Pages: 1097-1107
- 456 Downloads
- 1227 Views
- 2 Citations
This study aimed to find the Classical Test Theory (CTT) and Modern Test Theory (MTT) item parameters of the Differential Aptitude Test (DAT) and examined their comparability of them. The item parameters being studied are difficulty level and discrimination index. 5.024 data of the result sub-test DAT were documented by the Department of Psychology and Guidance and Counselling bureau. The parameter of classical and modern test items was estimated and correlated by examining the comparability between parameters. The results show that there is a significant correlation between item parameter estimates. The Rasch and IRT 1-PL models have the highest correlation toward CTT regarding the item difficulty level. In contrast, model 2-PL has the highest correlation toward CTT in the item discrimination index. Overall, the study concluded that CTT and MTT were comparable in estimating item parameters of DAT and thus could be used independently or complementary in developing DAT.
Keywords: Classical test theory, differential aptitude test, item parameter, modern test theory.
References
Abed, E. R., Al-Absi, M. M., & Abu shindi, Y. A. (2016). Developing a numerical ability test for students of education in Jordan: An application of Item Response Theory. International Education Studies, 9(1), 161. https://doi.org/10.5539/ies.v9n1p161
Adedoyin, O. O., Nenty, H. J., & Chilisa, B. (2020). Investigating the invariance of item difficulty parameter estimates based on CTT and IRT. International Journal of Educational Research and Reviews, 7(11). https://bit.ly/3JFG5T3
Ahmadi, A., & Thompson, N. A. (2012). Issues affecting Item Response Theory fit in language assessment: A study of differential item functioning in the Iranian National University entrance exam. Journal of Language Teaching and Research, 3(3), 401–412. https://doi.org/10.4304/jltr.3.3.401-412
AL-khadher, M. M. A., & Albursan, I. S. (2017). Accuracy of measurement in the classical and the modern test theory: An empirical study on a children intelligence test. International Journal of Psychological Studies, 9(1), 71–80. https://doi.org/10.5539/ijps.v9n1p71
Andrich, D. (2011). Rating scales and Rasch measurement. Expert Review of Pharmacoeconomics & Outcomes Research, 11(5), 571–585. https://doi.org/10.1586/erp.11.59
Avvannavar, S. M., Ambrose, B., & Chandavarkar, M. (2013). Determination of effectiveness of aptitude test to improve sincerity in the recruitment process. International Journal of Management, 4(4), 75–81.
Awopeju, O. A., & Afolabi, E. R. I. (2016). Comparative analysis of classical test theory and item response theory based item parameter estimates of senior school certificate mathematics examination. European Scientific Journal, 12(28), 263. https://doi.org/10.19044/esj.2016.v12n28p263
Baker, F. B. (2001). The basics of item response theory (2nd ed.). ERIC Clearinghouse on Assessment and Evaluation. https://eric.ed.gov/?id=ED458219
Bennet, G. K., Seashore, H. G., & Wesman, A. G. (1956). The Differential Aptitude Test: An overview. The Personnel and Guidance Journal, 35(2), 81–91. https://doi.org/10.1002/j.2164-4918.1956.tb01710.x
Bichi, A. A., Embong, R., Talib, R., Salleh, S., & Bin Ibrahim, A. (2019). Comparative analysis of classical test theory and item response theory using chemistry test data. International Journal of Engineering and Advanced Technology, 8(5C), 1260–1266. https://bit.ly/40Ifc8k
Bond, T. G., & Fox, C. M. (2015). Applying the Rasch model: Fundamental mesurement in the human sciences. In Applying the Rasch Model (Third ed.). Routledge. https://doi.org/10.4324/9781315814698
Boone, W. J., & Scantlebury, K. (2006). The role of Rasch analysis when conducting science education research utilizing multiple-choice tests. Science Education, 90(2), 253–269. https://doi.org/10.1002/sce.20106
Boopathiraj, C., & Chellamani, K. (2013). Analysis of test items on difficulty level and discrimination index in the test for research in education. International Journal of Social Science & Interdisciplinary Research, 2(2), 189–193. https://bit.ly/3lzCVYQ
Brennan, R. L. (2010). Generalizability theory and classical test theory. Applied Measurement in Education, 24(1), 1–21. https://doi.org/10.1080/08957347.2011.532417
Cohen, R. J., & Swerdlik, M. E. (2018). Psychological testing and assessment. In G. K. Zammit & J. W. Hull (Eds.), Guidebook for clinical psychology interns (pp. 121-134). Springer. https://doi.org/10.1007/978-1-4899-0222-1_8
Courville, T. G. (2004). An empirical comparison of Item Response Theory and Classical Test Theory item/person statistics. Texas A&M University. https://core.ac.uk/download/pdf/147123147.pdf
Çıkrıkçı, N. (2002). A study of Raven Standard Progressive Matrices Test’s item measure under classic and item response models: An empirical comparison. Journal of Faculty of Education Sciences, 35(1), 71-80. https://doi.org/10.1501/Egifak_0000000055
D’Oliveira, T. C. (2004). Dynamic spatial ability: An Exploratory analysis and a confirmatory study. The International Journal of Aviation Psychology, 14(1), 19–38. https://doi.org/10.1207/s15327108ijap1401_2
Dardick, W. R., & Mislevy, R. J. (2016). Reweighting data in the spirit of Tukey: Using Bayesian Posterior Probabilities as Rasch Residuals for studying misfit. Educational and Psychological Measurement, 76(1), 88–113. https://doi.org/10.1177/0013164415583351
de Ayala, R. J. (2009). The theory and practice of item response theory. The Guildford Press.
DeMars, C. (2010). Item Response Theory: Understanding statistics measurement. Oxford University Press. https://doi.org/j3rq
Dewberry, C. (2011). Aptitude testing and the legal profession. In Aptitude tests currently used in the professional services sector (Issue June). https://bit.ly/3JAUaAT
du Toit, M. (Ed.). (2003). IRT from SSI. Scientific Software International, Inc.
Ekpo, E. O., Egbonyi, I. C., & Bassey, S. W. (2016). Transformation from classical test theory (CTT) to item response theory (IRT) in research instrument validation. Journal of Educational Research, 1(4), 104–117. https://bit.ly/40wyFZ3
Eleje, L. I., Onah, F. E., & Abanobi, C. C. (2018). Comparative study of Classical Test Theory and Item Response Theory using diagnostic quantitative economics skill test item analysis results. European Journal of Educational & Social Sciences, 3(1), 57–75. https://bit.ly/3TQjKa5
Embretson, S. E., & Reise, S. P. (2000). Item response theory for psychologists. Lawrence Erlbaum Associates, Inc.
Fan, X. (1998). Item response theory and classical test theory: An empirical comparison of their item/person statistics. Educational and Psychological Measurement, 58(3), 357–381. https://doi.org/10.1177/0013164498058003001
Hakstian, A. R., & Bennet, R. W. (1978). Validity studies using the comprehensive ability battery (CAB): II. Relationship with the DAT and GATB. Educational and Psychological Measurement, 38(7), 1003–1015. https://doi.org/10.1177/001316447803800419
Hambleton, R. K., & Swaminathan, H. (1985). Item response theory: Principles and appliction (Vol. 1). Springer.
Hashmi, M., Zeeshan, A., Saleem, M., & Akbar, R. (2012). Development and validation of an aptitude test for secondary school mathematics students. Bulletin of Education and Research, 34(1), 65-76. https://bit.ly/3ZGmmZc
Hernandez, R. (2009). Comparison of the item discrimination and item difficulty of the Quick-Mental Aptitude Test using CTT and IRT methods. The International Journal of Educational and Psychological Assessment, 1(1), 12–18. https://bit.ly/3JCh9Mb
Hu, Z., Lin, L., Wang, Y., & Li, J. (2021). The integration of Classical Testing Theory and Item Response Theory. Psychology, 12(9), 1397–1409. https://doi.org/10.4236/psych.2021.129088
Kohli, N., Koran, J., & Henn, L. (2015). Relationships among Classical Test Theory and Item Response Theory frameworks via Factor Analytic Models. Educational and Psychological Measurement, 75(3), 389–405. https://doi.org/10.1177/0013164414559071
Košir, K., & Pečjak, S. (2007). Personality, motivational factors and difficulties in career decision-making in secondary school students. Psychological Topics, 16(1), 141–158. https://hrcak.srce.hr/file/32320
Linacre, J. M. (2012). Winsteps help for Rasch analysis. Winsteps. https://www.winsteps.com/winman/copyright.htm
MacDonald, P., & Paunonen, S. V. (2002). A Monte Carlo comparison of item and person statistics based on Item Response Theory versus Classical Test Theory. Educational and Psychological Measurement, 62(6), 921–943. https://doi.org/10.1177/0013164402238082
Magno, C. (2009). Demonstrating the difference between Classical Test Theory and Item Response Theory using derived test data. In The International Journal of Educational and Psychological Assessment, 1(1), 1-11. https://bit.ly/3Z9TSXE
Mahakud, G. C. (2013). Is it essential to measure intelligence along with aptitude test for career guidance. Research World-Journal of Arts, Science & Commerce, 4(1), 92–102. https://bit.ly/3Z9TiZY
Mankar, J., & Chavan, D. (2013). Differential aptitude testing of youth. International Journal of Scientific and Research Publications, 3(7), 1–6. https://bit.ly/3K0BImK
Marais, A. C. (2007). Using the differential aptitude test to estimate intelligence and scholastic achievement at grade nine level (Issue June) [University South Africa]. https://bit.ly/3lBZ5d1
Masriah, Z., Nursalim, M. M., & Fitriani, A. (2018). Persepsi mahasiswa terhadap jurusan perguruan tinggi dan konsep diri dengan kesesuaian minat memilih [Effect of student perceptions of college majors and self-concept on the suitability of interest in choosing]. Anfusina: Journal of Psychology, 1(1), 61–76. https://doi.org/10.24042/ajp.v1i1.3639
Mead, A. D., & Meade, A. W. (2010). CTT and IRT 1Test Construction using CTT and IRT with unrepresentative samples. Paper Presented at the Annual Meeting of the Society for Industrial and Organizational Psychology in Atlanta GA, 56. https://bit.ly/3naYBeg
Mehta, G., & Mokhasi, V. (2014). Item analysis of multiple choice questions-An assessment of the assessment tool. International Journal of Health Sciences and Research, 4(7), 197–202. https://bit.ly/42KzWxF
Mitra, N. K., Nagaraja, H. S., Ponnudurai, G., & Judson, J. P. (2009). The levels of difficulty and discrimination indices in type a multiple choice questions of pre-clinical semester 1 multidisciplinary summative tests. International EJournal of Science Medicine Education, 3(1), 2–7. https://doi.org/10.56026/imu.3.1.2
Muhid, A., Yusuf, A., Kusaeri, Novitasari, D. C. R., Asyhar, A. H., & Ridho, A. (2020). Determining scholastic aptitude test as predictors of academic achievement on students of Islamic School in Indonesia. New Educational Review, 61, 211–221. https://bit.ly/3ZJOafu
Petrillo, J., Cano, S. J., McLeod, L. D., & Coon, C. D. (2015). Using Classical Test Theory, Item Response Theory, and Rasch Measurement Theory to evaluate patient-reported outcome measures: A comparison of worked examples. Value in Health, 18(1), 25–34. https://doi.org/10.1016/j.jval.2014.10.005
Philip, A., & Odunayo, O. B. (2017). Application of Item Characteristic Curve (ICC) in the selection of test items. British Journal of Education, 5(2), 21–41. https://bit.ly/409wvif
Pollard, B., Dixon, D., Dieppe, P., & Johnston, M. (2009). Measuring the ICF components of impairment, activity limitation and participation restriction: An item analysis using Classical Test Theory and Item Response Theory. Health and Quality of Life Outcomes, 7, Article 41. https://doi.org/10.1186/1477-7525-7-41
Progar, S., Socan, G., & Slovenija, M. P. (2008). An empirical comparison of Item Response Theory and Classical Test Theory. Horizons of Psychology, 17(3), 5–24. https://bit.ly/3JGdr3Y
Pyari, P., Mishra, K., & Dua, B. (2016). A study of impact of aptitude in mathematics as stream selection at higher secondary level. Issues and Ideas in Education, 4(2), 141–149. https://doi.org/10.15415/iie.2016.42011
Qasem, M. A. N. (2013). A comparative study of Classical Theory (CT) and Item Response Theory (IRT) in relation to various approaches of evaluating the validity and reliability of research tools. IOSR Journal of Research & Method in Education, 3(5), 77–81. https://doi.org/10.9790/7388-0357781
Saputro, M. (2017). Analisis faktor-faktor yang mempengaruhi keputusan mahasiswa dalam memilih program studi [Analysis of the factors that influence student decisions in choosing a study program]. Jurnal Pendidikan Informatika Dan Sains, 6(1), 83–94. http://bit.ly/3MAr8or
Sayyah, M., Vakili, Z., Alavi, N. M., Bigdeli, M., Soleymani, A., Assarian, M., & Azarbad, Z. (2012). An item analysis of written multiple-choice questions: Kashan University of Medical Sciences. Nursing and Midwifery Studies, 1(2), 83–87. https://core.ac.uk/download/pdf/143838611.pdf
Setiawati, F. A., Izzaty, R. E., & Hidayat, V. (2018a). Evaluasi karakteristik psikometrik tes bakat [Evaluation of psychometric property of the aptitude test]. Humanitas, 15(1), 46–61. https://bit.ly/3ZC7laV
Setiawati, F. A., Izzaty, R. E., & Hidayat, V. (2018b). Analisis respon butir pada tes bakat skolastik [The item response theory analysis of scholastic test]. Jurnal Psikologi/Psychology Journal, 17(1), 1–17. https://doi.org/10.14710/jp.17.1.1-17
Setiawati, F. A., Izzaty, R. E., & Hidayat, V. (2018c). Items parameters of the space-relations subtest using Item Response Theory. Data in Brief, 19, 1785–1793. https://doi.org/10.1016/j.dib.2018.06.061
Shah, A. F., & Raza, M. A. (2009). The impact of parents ’ education towards the science aptitude of the students at elementary level in Southern Punjab. Pakistan Journal of Social Sciences, 29(1), 117–125. https://bit.ly/3n9qGme
Stemler, S. E., & Naples, A. (2021). Rasch measurement v. Item Response Theory: Knowing when to cross the line. Practical Assessment, Research and Evaluation, 26, Article 11. https://doi.org/10.7275/v2gd-4441
Sumintono, B., & Widhiarso, W. (2015). Aplikasi permodelan Rasch pada assessment pendidikan [Rasch modeling application in educational assessment]. Trim Komunikata.
Thomas, M. L. (2011). The value of Item Response Theory in clinical assessment: A review. Assessment, 18(3), 291–307. https://doi.org/10.1177/1073191110374797
Thorpe, G. L., & Favia, A. (2012). Data analysis using item response theory methodology: An introduction to selected programs and applications. Psychology, 20, 1–34.
Tractenberg, R. E. (2010). Classical and modern measurement theories, patient reports, and clinical outcomes. Contemporary Clinical Trials, 31(1), 1–3. https://doi.org/10.1016/S1551-7144(09)00212-2
Yang, F. M., & Kao, S. T. (2014). Item response theory for measurement validity. Shanghai Archives of Psychiatry, 26(3), 171–177.