Writing PISA-Like Mathematics Items: The Case of Tertiary Mathematics Instructors from a State University in the Philippines
Mark Lester B. Garcia , Derren N. Gaylo , Catherine P. Vistro-Yu
Mathematics test items in International Large-Scale Assessments (ILSAs) such as the Programme of International Student Assessment (PISA) and the Trend.
- Pub. date: October 15, 2024
- Online Pub. date: May 25, 2024
- Pages: 1441-1453
- 554 Downloads
- 1701 Views
- 0 Citations
Mathematics test items in International Large-Scale Assessments (ILSAs) such as the Programme of International Student Assessment (PISA) and the Trends in International Mathematics and Science Study (TIMSS) are nested in contexts defined in their assessment framework (e.g., the Personal, Occupational, Societal, and Scientific contexts in PISA). This study followed the item-writing activities of four tertiary mathematics instructors in the Philippines as they constructed context-based mathematics items. They were tasked to write four items each, following a set of specifications for PISA content and context categories. The data consisted of transcripts from the focus-group discussion which was conducted days after the task. The transcripts were then analyzed using thematic analysis. The results of this study showed that the phenomenon of item-writing in the context of writing PISA-like mathematics items had two themes: the phases of item-writing and the dimensions of item-writing. Findings showed that the respondents struggled to find realistic contexts and that they engaged in a problem-solving task likened to solving a puzzle as they attempted to satisfy the content, context, and process categories in the table of specifications (TOS). This study contributes to filling in the research gap on item-writing activities, particularly those of mathematics teachers in the Philippines- a country whose recent mathematical performance in the PISA 2018, TIMSS 2019, and PISA 2022 was nothing short of dismal.
context based math items item writing mathematical literacy pisa mathematics assessment framework pisa like mathematics items
Keywords: Context-based math items, item-writing, mathematical literacy, PISA mathematics assessment framework, PISA-like mathematics items.
References
Al-Bahlani, S. M. (2019). Assessment literacy: A study of EFL teachers’ assessment knowledge, perspectives, and classroom behaviors [Doctoral dissertation, The University of Arizona]. University of Arizona Repository. http://hdl.handle.net/10150/633240
Baranowski, R. A. (2006). Item editing and editorial review. In T. M. Haladyna & S. M. Downing (Eds.), Handbook of test development (pp. 349-357). Routledge. https://doi.org/10.4324/9780203874776
Bernardo, A. B. I. (2021). Socioeconomic status moderates the relationship between growth mindset and learning in math and science: Evidence from PISA 2018 Philippine data. International Journal of School and Educational Psychology, 9(2), 208-222. https://doi.org/10.1080/21683603.2020.1832635
Bicar, V., & Gaylo, D. (2022). Cluster characterization of countries’ performance in mathematics olympiad: Input to mathematics education. Science International (Lahore), 34(5), 503-506. https://doi.org/10.5281/zenodo.7240929
Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77-101. https://doi.org/10.1191/1478088706qp063oa
Braun, V., & Clarke, V. (2012). Thematic analysis. In H. Cooper, P. M. Camic, D. L. Long, A. T. Panter, D. Rindskopf, & K. J. Sher (Eds.), APA handbook of research methods in psychology, Vol. 2. Research designs: Quantitative, qualitative, neuropsychological, and biological (pp. 57-71). American Psychological Association. https://doi.org/10.1037/13620-004
Cagasan, L., Luo, R., Robertson, P., & Care, E. (2016). Formative assessment project: Phase 2 research report. Assessment, Curriculum, and Technology Research Centre (ACTRC). Melbourne and Manila. http://bit.ly/3up0Ej3
Caine, V., Clandinin, D. J., & Lessard, S. (2022). Narrative inquiry: Philosophical roots. Bloomsbury Publishing. https://doi.org/10.5040/9781350142084
Chi, C. (2023, December 6). Philippines still lags behind world in math, reading and science — PISA 2022. Philstar Global Corp. https://bit.ly/3whiHIx
Clores, M. A., & Reganit, A. A. R. (2020). Investigating the assessment literacy of teachers in private junior high schools in the Philippines. Humanities, Arts and Social Sciences Studies, 20(2), 461-476. https://doi.org/10.14456/hasss.2020.17
Close, S., & Shiel, G. (2014, June 24-25). A Comparison of TIMSS 2011 and PISA 2012 math frameworks and performance for Ireland and selected countries [Paper Presentation]. Science and Mathematics Education Conference, Dublin City University.
Department of Education. (2019). PISA 2018: National report of the Philippines. https://bit.ly/3SYYsbq
Downing, S. M. (2006). Twelve steps for effective test development. In S. M. Downing & T. M. Haladyna (Eds.), Handbook of test development (pp. 3-25). Lawrence Erlbaum.
Duff, P. A. (2012). How to carry out case study research. In A. Mackey & S. M. Gass (Eds.), Research methods in second language acquisition: A practical guide (pp. 95-116). Blackwell Publishing.
Elshawa, N., Heng, C. S., Abdullah, A. N., & Rashid, S. (2016). Teachers' assessment literacy and washback effect of assessment. International Journal of Applied Linguistics and English Literature, 5(4), 135-141. https://bit.ly/3UlhCri
Espinosa, A. A., Gomez, M. A. C., Reyes, A. S., Macahilig, H. B., Cortez, L. A. S., & David, A. P. (2023). International large-scale assessment (ILSA): Implications for pre-service teacher education in the Philippines. Issues in Educational Research, 33(2), 553-569. https://www.iier.org.au/iier33/espinosa.pdf
Frey, B. B., Petersen, S., Edwards, L. M., Pedrotti, J. T., & Peyton, V. (2005). Item-writing rules: Collective wisdom. Teaching and Teacher Education, 21(4), 357-364. https://doi.org/10.1016/j.tate.2005.01.008
Fulkerson, D., Nichols, P., & Mittelholtz, D. (2010, May 3). What item writers think when writing items: Towards a theory of item-writing expertise [Paper Presentation]. Annual Meeting of the American Educational Research Association, Denver, CO.
Fulkerson, D., Nichols, P., & Snow, E. (2011, April 8-12). Expanding the model of item-writing expertise: Cognitive processes and requisite knowledge structures [Paper Presentation]. Annual Meeting of the American Educational Research Association, New Orleans, LA.
Gaylo, D. N., Caingcoy, M. E., & Mugot, D. C. (2020). Utilization of scholarly journal articles in the teaching and learning of teacher education courses. Balkan and Near Eastern Journal of Social Sciences, 6(3), 59-66. https://doi.org/10.47696/adved.202038
Geisinger, K. F., & Usher-Tate, B. J. (2016). A brief history of educational testing and psychometrics. In C. S. Wells & M. Faulkner-Bond (Eds.), Educational measurement from foundations to future (pp. 3-20). The Guilford Press.
Griffin, P., Cagasan, L., Care, E., Vista, A., & Nava, F. (2016). Formative assessment policy and its enactment in the Philippines. In D. Laveault & L. Allal (Eds.), Assessment for learning: Meeting the challenge of implementation (pp. 75-92). Springer. https://doi.org/10.1007/978-3-319-39211-0_5
Hailaya, W. M. (2014). Teacher assessment literacy and student outcomes in the Province of Tawi-Tawi, Philippines [Doctoral thesis, The University of Adelaide]. https://bit.ly/3I0BeLD
Hanafi, N. B. M., Ali, N. B. M., Zamani, S. B., Yamin, N. A. B., & Ismail, N. N. B. (2020). Examining assessment literacy: A study of technical teacher. European Journal of Molecular and Clinical Medicine, 7(8), 705-717. https://bit.ly/3I1OT4Y
Haw, J. Y., King, R. B., & Trinidad, J. E. R. (2021). Need supportive teaching is associated with greater reading achievement: What the Philippines can learn from PISA 2018. International Journal of Educational Research, 110, Article 101864. https://doi.org/10.1016/j.ijer.2021.101864
International Association for the Evaluation of Educational Achievement. (2020). TIMSS 2019 trends in international math and science study Philippines country report. https://bit.ly/4bG4DbM
Johnson, M., Constantinou, F., & Crisp, V. (2017). How do question writers compose external examination questions? Question writing as a socio‐cognitive process. British Educational Research Journal, 43(4), 700-719. https://doi.org/10.1002/berj.3281
Khalid, N. H. M., Latif, A. A., & Yusof, I. J. (2021). Assessment literacy: A systematic literature review and research agenda. Annals of the Romanian Society for Cell Biology, 25(3), 4668-4696. https://bit.ly/3uG31hd
King, N. (2004). Using templates in the thematic analysis of text. In C. Cassell & G. Symon (Eds.), Essential guide to qualitative methods in organizational research (pp. 256-270). Sage. https://doi.org/10.4135/9781446280119.n21
Kohar, A. W., Zulkardi, Z., & Darmawijoyo, D. (2014). Developing PISA-like math tasks to promote students’ mathematical literacy. In R. Ilma (Ed.), Proceeding in the Second South East Asia Design - Development Research (SEA-DR) Conference (pp. 14-26). Universitas Sriwijaya.
Kuger, S., & Klieme, E. (2016). Dimensions of context assessment. In S. Kuger, E. Klieme, N. Jude, & D. Kaplan (Eds), Assessing contexts of learning (pp. 3-37). Methodology of Educational Measurement and Assessment. Springer. https://doi.org/10.1007/978-3-319-45357-6_1
Kyriakides, L., Christoforou, C., & Charalambous, C. Y. (2013). What matters for student learning outcomes: A meta-analysis of studies exploring factors of effective teaching. Teaching and Teacher Education, 36, 143-152. https://doi.org/10.1016/j.tate.2013.07.010
Lapinid, M. R. C., Cordell, M. O., II, Teves, J. M., Yap, S. A., Chua, U., & Bernardo, A. B. I. (2022). Addressing the poor mathematics performance of Filipino learners: Beyond curricular and instructional interventions. DLSU-AKI Policy Brief, 14(2), 1-4. https://bit.ly/3WmQ9Ip
Lasaten, R. C. S. (2016). Assessment methods, problems and training needs of public high school teachers in English. International Journal of Languages, Literature and Linguistics, 2(2), 55-60. https://doi.org/10.18178/ijlll.2016.2.2.67
Lian, L. H., Yew, W. T., & Meng, C. C. (2014). Enhancing Malaysian teachers’ assessment literacy. International Education Studies, 7(10), 74-81. https://doi.org/10.5539/ies.v7n10p74
Magno, C. (2003). The profile of teacher-made test construction of the professors of University of Perpetual Help Laguna. UPHL Institutional Journal, 1(1), 48-55. https://ssrn.com/abstract=1429347
Mellati, M., & Khademi, M. (2018). Exploring teachers' assessment literacy: Impact on learners' writing achievements and implications for teacher development. Australian Journal of Teacher Education, 43(6), Article 1. https://doi.org/10.14221/ajte.2018v43n6.1
Memisevic, H., & Biscevic, I. (2022). Mathematics, gender and the meaning in life: The results of PISA testing in Bosnia and Herzegovina. European Journal of Mathematics and Science Education, 3(2), 171-179. https://doi.org/10.12973/ejmse.3.2.171
Montemayor, M. T. (2023, December 6). CHED to address PH students’ low int’l assessment ranking. Philippine News Agency. https://www.pna.gov.ph/articles/1215002
Mullis, I. V. S., Martin, M. O., & Loveless, T. (2016). 20 years of TIMSS: International trends in math and science achievement, curriculum, and instruction. International Association for the Evaluation of Educational Achievement (IEA). https://bit.ly/3UESGN8
Namoco, S., & Zaharudin, R. (2021). Pedagogical beliefs and learning assessment in Science: Teacher’s experiences anchored on theory of reasoned action. Journal of Turkish Science Education, 18(2), 304-319. https://doi.org/10.36681/tused.2021.67
Napanoy, J. B., & Peckley, M. K. (2020). Assessment literacy of public elementary school teachers in the indigenous communities in Northern Philippines. Universal Journal of Educational Research, 8(11B), 5693-5703. https://doi.org/10.13189/ujer.2020.082203
Orbeta, A. C., Melad, K. A. M., & Potestad, M. (2020). Correlates of test performance of 15-year-old students in the Philippines: Evidence from PISA (No. 2020-57). PIDS Discussion Paper Series. https://bit.ly/3T03sw8
Organisation for Economic Cooperation and Development. (2019). PISA 2018 assessment and analytical framework. OECD Publishing. https://bit.ly/44lKfcC
Piosang, T. L. (2017). A cross-sectional analysis of classroom assessment literacy of English teachers in secondary and tertiary levels. Educational Measurement and Evaluation Review, 8(1), 30-48. https://bit.ly/3y0d2qX
Popham, W. J. (2011). Assessment literacy overlooked: A teacher educator's confession. The Teacher Educator, 46(4), 265-273. https://doi.org/10.1080/08878730.2011.605048
Radišić, J., & Baucal, A. (2018). Teachers’ reflection on PISA items and why they are so hard for students in Serbia. European Journal of Psychology of Education, 33, 445-466. https://doi.org/10.1007/s10212-018-0366-0
Rodriguez, M. C. (1997, March 24-28). The art & science of item-writing: A meta-analysis of multiple-choice item format effects [Paper Presentation]. Annual Meeting of the American Educational Research Association, Chicago.
Rodriguez, M. C., & Haladyna, T. M. (2013). Writing selected-response items for classroom assessment. In J. McMillan (Ed.), SAGE handbook of research on classroom assessment (pp. 293-312). SAGE Publications, Inc. https://doi.org/10.4135/9781452218649.n17
Schleicher, A. (2019). PISA 2018: Insights and interpretations. OECD Publishing. https://bit.ly/4bG36lK
Sonday, A., Ramugondo, E., & Kathard, H. (2020). Case study and narrative inquiry as merged methodologies: A critical narrative perspective. International Journal of Qualitative Methods, 19, 1-5. https://doi.org/10.1177/1609406920937880
Tatto, M. T., Rodriguez, M. C., Reckase, M. D., Smith, W. M., Bankov, K., & Pippin, J. (2020). The FIRSTMATH study: Concepts, methods, and strategies for comparative international research in mathematics education. In M. T. Tatto, M. C. Rodriguez, M. D. Reckase, W. M. Smith, K. Bankov, & J. Pippin, The first five years of teaching mathematics (FIRSTMATH): Concepts, methods, and strategies for comparative international research in math education (pp. 1-20). Springer International Publishing. https://doi.org/10.1007/978-3-030-44047-3_1
United Nations Children's Fund. (2021). SEA-PLM 2019 Southeast Asia primary learning metrics Philippines country report. https://bit.ly/49cBlQ5
United Nations Children’s Fund & Southeast Asian Ministers of Education Organization. (2019). SEA-PLM 2019 Assessment Framework (1st ed.). United Nations Children’s Fund (UNICEF) & Southeast Asian Ministers of Education Organization (SEAMEO) – SEA-PLM Secretariat. https://bit.ly/3SWlSOA
Wu, M. L. (2009, June 2). A critical comparison of the contents of PISA and TIMSS mathematics assessments [Paper Presentation]. NCES “What we can learn from PISA” research conference, Washington, DC.
Xu, Y., & Brown, G. T. L. (2016). Teacher assessment literacy in practice: A reconceptualization. Teaching and Teacher Education, 58, 149-162. https://doi.org/10.1016/j.tate.2016.05.010
Zulkardi, Z., & Kohar, A. W. (2018). Designing PISA-like math tasks in Indonesia: Experiences and challenges. Journal of Physics: Conference Series, 947, Article 012015. https://doi.org/10.1088/1742-6596/947/1/012015