A Proposal for Holistic Assessment of Computational Thinking for Undergraduate: Content Validity
Studies have acknowledged computational thinking (CT) as an efficient approach for problem-solving particularly required in digital workplaces. This r.
Studies have acknowledged computational thinking (CT) as an efficient approach for problem-solving particularly required in digital workplaces. This research aims to identify indicators for a holistic CT assessment instrument for undergraduate students. A three-round fuzzy Delphi study has been conducted to gain comprehensive opinions and consensus from undergraduate lecturers of computer science disciplines and experts from the information technology industry. In round 1, the experts judged a set of predefined indicators describing CT skills and attitudes identified from the literature, while rounds 2 and 3 focused on variables selection. The consensus was achieved on holistic CT, and the indicators are teamwork, communication, spiritual intelligence, generalization, problem-solving, algorithmic thinking, evaluation, abstraction, decomposition, and debugging. Results demonstrate the importance of attitudes in the process of solving a problem and suggest higher education institutions to consider holistic CT in preparing qualified future graduates. Many CT studies focused only on the skills of CT. This study outlines the assessment indicators that consider both CT skills and attitudes, particularly at the undergraduate level.
Keywords: Computational thinking, assessment, fuzzy Delphi method, undergraduate education.
References
ACM and IEEE Computer Society. (2013). Computer science curricula 2013. New York, NY: ACM-IEEE. https://doi.org/10.1145/2534860
Aho, A. V. (2012). Computation and computational thinking. Computer Journal, 55(7), 833–835. https://doi.org/10.1093/comjnl/bxs074
Amram, Y., & Dryer, D. C. (2008, August). The integrated spiritual intelligence scale (ISIS): Development and preliminary validation. Paper presented at the 116th Annual Conference of the American Psychological Association. University of Padova, Boston, MA.
Angeli, C., Voogt, J., Fluck, A., Webb, M., Cox, M., Malyn-Smith, J., … Angeli Charoula; Voogt, J. F. A. (2016). A K-6 Computational thinking curriculum framework: Implications for teacher knowledge. Educational Technology & Society, 19(3), 47–57. https://doi.org/10.2307/jeductechsoci.19.3.47
Atmatzidou, S., & Demetriadis, S. (2014). How to Support students’ computational thinking skills in educational robotics activities. In Proceedings of 4th International Workshop Teaching Robotics, Teaching with Robotics & 5th International Conference Robotics in Education (pp. 43–50). Padova, Italy. :University of Padova
Barr, D., Harrison, J., & Conery, L. (2011). Computational thinking: A digital age skill for everyone. Learning and Leading with Technology, 38(6), 20–23.
Bell, T., & Roberts, J. (2016). Computational Thinking is More About Human than Computers. SET, 1(3), 3–7. https://doi.org/10.1145/1595496.1562941
Boateng, G. O., Neilands, T. B., Frongillo, E. A., Melgar-Quinonez, H. R., & Young, S. L. (2018). Best practices for developing and validating scales for health , social , and behavioral research : A primer. Frontiers in Public Health, 6(June), 1–18. https://doi.org/10.3389/fpubh.2018.00149
Bocconi, S., Chioccariello, A., Dettori, G., Ferrari, A., & Engelhardt, K. (2016). JRC science for policy report. developing computational thinking in compulsory education-implications for policy and practice. Retrieved from https://doi.org/10.2791/792158
Brennan, K., & Resnick, M. (2012, April). New frameworks for studying and assessing the development of computational thinking. Paper presented at Annual American Educational Research Association Meeting, Vancouver, BC, Canada.
Britton, E., Simper, N., Leger, A., & Stephenson, J. (2015). Assessing teamwork in undergraduate education: a measurement tool to evaluate individual teamwork skills. Assessment and Evaluation in Higher Education, 42(3), 378–397. https://doi.org/10.1080/02602938.2015.1116497
Bubno, K., & Takacs, V. L. (2017). The mathability of word problems as initial computer programming exercises. In Proceeding of The 8th IEEE International Conference on Cognitive Infocommunications 2017 (pp. 39–44). Debrecen, Hungary: IEEE. https://doi.org/10.1109/CogInfoCom.2017.8268213
Chang, C. K., Tsai, Y. T., & Chin, Y. L. (2017). A visualization tool to support analyzing and evaluating scratch projects. In T. Matsuo, N. Fukuta, M. Mori, K. Hashimoto, & S. Hirokawa (Eds.), Proceedings - 2017 6th IIAI International Congress on Advanced Applied Informatics, IIAI-AAI 2017, (pp. 498–502). Hamamatsu, Japan: CPS https://doi.org/10.1109/IIAI-AAI.2017.83
Chang, P. L., Hsu, C. W., & Chang, P. C. (2011). Fuzzy Delphi method for evaluating hydrogen production technologies. International Journal of Hydrogen Energy, 36(21), 14172–14179. https://doi.org/10.1016/j.ijhydene.2011.05.045
Cheng, C., & Lin, Y. (2002). Evaluating the best main battle tank using fuzzy decision theory with linguistic criteria evaluation. European Journal of Operational Research, 142(1), 174–186.
Curzon, P., Dorling, M., Selby, C., & Woollard, J. (2014). Developing computational thinking in the classroom: A framework. Retrieved from http://eprints.soton.ac.uk/369594/10/DevelopingComputationalThinkingInTheClassroomaFramework.pdf
Damigos, D., & Anyfantis, F. (2011). The value of view through the eyes of real estate experts: A Fuzzy Delphi Approach. Landscape and Urban Planning, 101(2), 171–178. https://doi.org/10.1016/j.landurbplan.2011.02.009
de Lima, J. P. C., Carlos, L. M., Schardosim Simfo, J. P., Pereira, J., Mafra, P. M., & da Silva, J. B. (2016). Design and implementation of a remote lab for teaching programming and robotics. IFAC-PapersOnLine, 49(30), 86–91. https://doi.org/10.1016/j.ifacol.2016.11.133
Fronza, I., El Ioini, N., & Corral, L. (2016). Teaching software design engineering across the K-12 curriculum. In Proceedings of the 17th Annual Conference on Information Technology Education - SIGITE ’16 (pp. 97–101). New York, NY: ACM. https://doi.org/10.1145/2978192.2978220
Gouws, L., Bradshaw, K., & Wentworth, P. P. (2013). Computational thinking in educational activities an evaluation of the educational game Light-Bot. Paper presented at the 18th Annual Conference on Innovation and Technology in Computer Science Education. University of Kent, Canterbury, United Kingdom.
Gribble, C. (2014). Exploring “employability” in different cultural contexts. Deakin University Australia. Retrieved from https://www.srhe.ac.uk/downloads/gribble-cate.pdf
Grover, S., & Pea, R. (2013). Computational thinking in K-12: A review of the state of the field. Educational Researcher, 42(1), 38–43. https://doi.org/10.3102/0013189X12463051
Gunbatar, M. S., & Karalar, H. (2018). Gender differences in middle school students’ attitudes and self-efficacy perseptions towards mblock programming. European Journal of Educational Research, 7(4), 925–933. https://doi.org/10.12973/eu-jer.7.4.923
Hsu, Y., Lee, C., & Kreng, V. B. (2010). The application of Fuzzy Delphi Method and Fuzzy AHP in lubricant regenerative technology selection. Expert Systems With Applications, 37(1), 419–425. https://doi.org/10.1016/j.eswa.2009.05.068
Jonassen, D. (2000). Toward a design theory of problem solving. Educational Technology Research and Development, 48(4), 63–85. https://doi.org/10.1007/bf02300500
Kadkhoda, M., & Jahanic, H. (2012). Problem-solving capacities of spiritual intelligence for artificial intelligence. Procedia - Social and Behavioral Sciences, 32, 170–175. https://doi.org/10.1016/j.sbspro.2012.01.027
Kalelioğlu, F., Gülbahar, Y., & Kukul, V. (2016). A framework for computational thinking based on a systematic research review. Baltic J . Modern Computing, 4(3), 583–596.
Kazimoglu, C., Kiernan, M., Bacon, L., & MacKinnon, L. (2012). Learning programming at the computational thinking level via digital game-play. Procedia Computer Science, 9, 522–531. https://doi.org/10.1016/j.procs.2012.04.056
Kert, S. B. (2019). A proposal of in-service teacher training approach for computer science teachers. European Journal of Educational Research, 8(2), 477–489. https://doi.org/10.12973/eu-jer.8.2.477
Korkmaz, O., Cakir, R., & Ozden, M. Y. (2017). A Validity and Reliability Study of the Computational Thinking Scales (CTS). Computers in Human Behavior, 72, 558–569. https://doi.org/10.1016/j.chb.2017.01.005
L’Heureux, J., Boisvert, D., Cohen, R., & Sanghera, K. (2012). IT problem solving: an implementation of computational thinking in information technology. In R. Connolly (Ed.), Proceedings of the 13th annual conference on Information technology education - SIGITE ’12 (pp. 183–188). New York, NY: ACM. https://doi.org/10.1145/2380552.2380606
Lee, I., Martin, F., Denner, J., Coulter, B., Allan, W., Erickson, J., … Werner, L. (2011). Computational thinking for youth in practice. ACM Inroads, 2(1), 32–37.
Leopold, T. A., Ratcheva, V., & Zahidi, S. (2016). The Future of jobs employment, skills and workforce strategy for the fourth industrial revolution. World Economic Forum. https://doi.org/10.1177/1946756712473437
Li, T., & Wang, T. (2012). A unified approach to teach computational thinking for first year non–cs majors in an introductory course. IERI Procedia, 2, 498–503. https://doi.org/10.1016/j.ieri.2012.06.123
Microsoft. (n.d.). Education competencies: Dealing with ambiguity - Microsoft Education. Retrieved July 23, 2017, from https://www.microsoft.com/en-us/education/training-and-events/education-competencies/dealing_with_ambiguity.aspx
Ministry of Research Technology and Higher Education. (2015). Peraturan menteri riset, teknologi, dan pendidikan Tinggi Republik Indonesia No. 44 Tahun 2015 [Regulation of Ministry of Research, Technology and Higher Education No. 44 of 2015 of the Republic of Indonesia]. Retrieved from https://img.akademik.ugm.ac.id/unduh/2015/PERMENRISTEKDIKTI_Nomor_44_Tahun_2015_SNPT.pdf
Missiroli, M., Russo, D., & Ciancarini, P. (2017). Cooperative Thinking, or: Computational Thinking Meets Agile. In 30th IEEE Conference on Software Engineering Education and Training (pp. 187–191). Savannah, GA: IEEE. https://doi.org/10.1109/CSEET.2017.37
Mueller, J., Beckett, D., Hennessey, E., & Shodiev, H. (2017). Assessing computational thinking across the curriculum. In P. J. Rich & C. B. Hodges (Eds.), Emerging Research, Practice, and Policy on Computational Thinking (pp. 251–267). Cham, Switzerland: Springer International Publishing. https://doi.org/10.1007/978-3-319-52691-1
Nasel, D. D. (2004). Spiritual orientation in relation to spiritual intelligence: A consideration of traditional Christianity and New Age/individualistic spirituality. University of South Australia.
Nikaido, B., & Ventura, J. (2016). Code puzzles - Robot Chronicle. In Proceeding IEEE Southeastcon 2016 (pp. 1-4). Norfolk, VA: IEEE. https://doi.org/10.1109/SECON.2016.7506772
Orchard, C. A., King, G. A., Khalili, H., & Bezzina, M. B. (2012). Assessment of Interprofessional Team Collaboration Scale (AITCS): Development and Testing of the Instrument. Journal of Continuing Education in the Health Professions, 32(1), 58–67. https://doi.org/10.1002/chp.21123.
Parmar, P. I. (2014). Construction and standardization of a spiritual intelligence scale for the pupils of higher secondary schools of Gujarat State. Gujarat University. Retrieved from https://shodhganga.inflibnet.ac.in/handle/10603/110890
Perkovic, L., Settle, A., Hwang, S., & Jones, J. (2010). A framework for computational thinking across the curriculum. In A. Clear & L. R. Dag (Eds.), Proceedings of the 15th Annual Conference on Innovation and Technology in Computer Science Education (pp. 123–127). New York, NY: ACM. https://doi.org/10.1145/1822090.1822126
Pulimood, S. M., Pearson, K., & Bates, D. C. (2016). A Study on the Impact of Multidisciplinary Collaboration on Computational Thinking. In C. Alphonche (Ed.), Proceedings of the 47th ACM Technical Symposium on Computing Science Education - SIGCSE ’16 (pp. 30–35). New York, NY: ACM. https://doi.org/10.1145/2839509.2844636
Qin, H. (2009). Teaching computational thinking through bioinformatics to biology students. ACM SIGCSE Bulletin, 41(1), 188–191. https://doi.org/10.1145/1539024.1508932
Rahayu, T., & Osman, K. (2018). Early study : Self-confidence on the computational thinking skills among science teacher candidates. Paper presented at the 5th International Conference on Islam and Higher Education, Universitas Negeri Padang, Indonesia.
Rahayu, T., & Osman, K. (2019). Knowledge level and self-confidence on the computational thinking skills among science teacher candidates. Al-Biruni Journal of Physics Education/Jurnal Ilmiah Pendidikan Fisika Al-Biruni, 8(April), 117–126. https://doi.org/10.24042/jipfalbiruni.v8i1.4450
Roman-Gonzalez, M., Perez-Gonzelez, J. C., & Jimenez-Fernandez, C. (2016). Which cognitive abilities underlie computational thinking? Criterion validity of the Computational Thinking Test. Computers in Human Behavior, 72, (pp. 1–14). https://doi.org/10.1016/j.chb.2016.08.047
Rowe, G., & Wright, G. (2001). Expert opinions in forecasting: The role of the Delphi technique. In J. S. Armstrong (Ed.), Principles of Forcasting (pp. 125–144). Boston, MA: Springer. https://doi.org/10.1016/j.accreview.2004.12.084
Ruthmann, A., Heines, J. M., Greher, G. R., Laidler, P., & Ii, C. S. (2010). Teaching computational thinking through musical live coding in Scratch. In G. Lewandowksi, S. Wolfman, T. J. Cortina, E. L. Walker, & D. R. Musicant (Eds.) SIGCSE’10 (pp. 3–7). New York, NY: ACM.
Saido, G. A. M., Siraj, S., DeWitt, D., & Al-Amedy, O. S. (2018). Development of an instructional model for higher order thinking in science among secondary school students: a fuzzy Delphi approach. International Journal of Science Education, 40(8), 847–466. https://doi.org/10.1080/09500693.2018.1452307
Saputra, W. S. (2014). Employers’ needs for employability skills of engineering graduates in Indonesia. In A. G. Abdullah, T. Aryanti, D. Kurnia, & S. Elvyanti (Eds.), The 3rd UPI International Conference on Technical and Vocational Education and Training (pp. 223–227). Bandung, Indonesia: Atlantis Press.
Schwab, K. (2016). The Fourth Industrial Revolution. Geneva, Switzerland: World Economic Forum.
Selby, C., & Woollard, J. (2013). Computational thinking : The developing definition. University of Southampton. Retrieved from https://eprints.soton.ac.uk/id/eprint/356481.
Senin, S., & Nasri, N. M. (2019). Teachers’ concern towards applying computational thinking skills in teaching and learning teachers ’ concern towards applying computational thinking skills in teaching and learning. International Journal of Academic Research in Business and Social Sciences, 9(1), 296–310. https://doi.org/10.6007/IJARBSS/v9-i1/5398
Shanmugam, L., Yassin, S. F., & Khalid, F. (2019). Enhancing students ’ motivation to learn computational thinking through mobile application development module ( M-CT ). International Journal of Engineering and Advanced Technology, 8(5), 1293–1303.
Shute, V. J., Sun, C., & Clarke, J. A. (2017). Demystifying computational thinking. Educational Research Review, 22, 142–158. https://doi.org/10.1016/j.edurev.2017.09.003
Sisk, D. A. (2002). Spiritual Intelligence: The tenth intelligence that integrates all other intelligences. Gifted Education International, 16(3), 208–213. https://doi.org/10.1177/026142940201600304
Sisk, D. A. (2016). Spiritual intelligence : Developing higher consciousness revisited. Gifted Education International, 32(3) , 1–15. https://doi.org/10.1177/0261429415602567
Snow, E., Rutstein, D., Bienkowski, M., & Xu, Y. (2017). Principled assessment of student learning in high school computer science. In J. Tenenberg (Ed.), Proceedings of the 2017 ACM Conference on International Computing Education Research - ICER ’17 (pp. 209–216). New York, NY: ACM. https://doi.org/10.1145/3105726.3106186
Strom, P. S., & Strom, R. D. (2011). Teamwork skills assessment for cooperative learning. Educational Research and Evaluation, 17(4), 233–251. https://doi.org/10.1080/13803611.2011.620345
Talib, N., Yasin, S. F. M., & Mohd, K. M. (2017). Teaching and learning computer programming using gamification and observation through action research. International Journal of Academic Research in Progressive Education and Development, 6(3), 1–11. https://doi.org/10.6007/IJARPED/v6-i3/3045
Walden, J., Doyle, M., Garns, R., & Hart, Z. (2013). An informatics perspective on computational thinking. In M. Goldwebber (Ed.), Proceedings of the 18th ACM Conference on Innovation and Technology in Computer Science Education. (pp. 4–9). New York, NY: ACM. https://doi.org/10.1145/2462476.2483797
Wing, J. M. (2006). Computational thinking. Communication of the ACM, 49(3), 33–35.
Wing, J. M. (2008). Computational thinking and thinking about computing. In 2008 IEEE International Symposium on Parallel and Distributed Processing (pp. 1-1.) Miami, FL: IEEE. https://doi.org/10.1109/IPDPS.2008.4536091
Wing, J. M. (2011). Research notebook: Computational thinking—What and why? The Link Magazine. Retrieved from http://www.cs.cmu.edu/link/research-notebook-computational-thinking-what-and-why
World Economic Forum (2018). The future of jobs report 2018. Retrieved from https://www.weforum.org/reports/the-future-of-jobs-report-2018
Yevseyeva, K., & Towhidnejad, M. (2012). Work in progress: Teaching computational thinking in middle and high school. In 2012 Frontiers in Education Conference Proceedings (pp. 1–2). Seattle, WA: IEEE. https://doi.org/10.1109/FIE.2012.6462487
Zakaria, E., Haron, Z., & Daud, M. (2004). The Reliability and construct validity of scores on the attitudes toward problem solving scale. Journal of Science and Mathematics Education in South East Asia, 27(2), 81–91.