logo logo European Journal of Educational Research

EU-JER is is a, peer reviewed, online academic research journal.

Subscribe to

Receive Email Alerts

for special events, calls for papers, and professional development opportunities.

Subscribe

Publisher (HQ)

Eurasian Society of Educational Research
Eurasian Society of Educational Research
Christiaan Huygensstraat 44, Zipcode:7533XB, Enschede, THE NETHERLANDS
Eurasian Society of Educational Research
Headquarters
Christiaan Huygensstraat 44, Zipcode:7533XB, Enschede, THE NETHERLANDS
context based math items item writing mathematical literacy pisa mathematics assessment framework pisa like mathematics items

Writing PISA-Like Mathematics Items: The Case of Tertiary Mathematics Instructors from a State University in the Philippines

Mark Lester B. Garcia , Derren N. Gaylo , Catherine P. Vistro-Yu

Mathematics test items in International Large-Scale Assessments (ILSAs) such as the Programme of International Student Assessment (PISA) and the Trend.

M

Mathematics test items in International Large-Scale Assessments (ILSAs) such as the Programme of International Student Assessment (PISA) and the Trends in International Mathematics and Science Study (TIMSS) are nested in contexts defined in their assessment framework (e.g., the Personal, Occupational, Societal, and Scientific contexts in PISA). This study followed the item-writing activities of four tertiary mathematics instructors in the Philippines as they constructed context-based mathematics items. They were tasked to write four items each, following a set of specifications for PISA content and context categories. The data consisted of transcripts from the focus-group discussion which was conducted days after the task. The transcripts were then analyzed using thematic analysis. The results of this study showed that the phenomenon of item-writing in the context of writing PISA-like mathematics items had two themes: the phases of item-writing and the dimensions of item-writing. Findings showed that the respondents struggled to find realistic contexts and that they engaged in a problem-solving task likened to solving a puzzle as they attempted to satisfy the content, context, and process categories in the table of specifications (TOS). This study contributes to filling in the research gap on item-writing activities, particularly those of mathematics teachers in the Philippines- a country whose recent mathematical performance in the PISA 2018, TIMSS 2019, and PISA 2022 was nothing short of dismal.

Keywords: Context-based math items, item-writing, mathematical literacy, PISA mathematics assessment framework, PISA-like mathematics items.

cloud_download PDF
Cite
Article Metrics
Views
126
Download
383
Citations
Crossref
0

Scopus
0

References

Al-Bahlani, S. M. (2019). Assessment literacy: A study of EFL teachers’ assessment knowledge, perspectives, and classroom behaviors [Doctoral dissertation, The University of Arizona]. University of Arizona Repository.  http://hdl.handle.net/10150/633240  

Baranowski, R. A. (2006). Item editing and editorial review. In T. M. Haladyna & S. M. Downing (Eds.), Handbook of test development (pp. 349-357). Routledge. https://doi.org/10.4324/9780203874776

Bernardo, A. B. I. (2021). Socioeconomic status moderates the relationship between growth mindset and learning in math and science: Evidence from PISA 2018 Philippine data. International Journal of School and Educational Psychology9(2), 208-222. https://doi.org/10.1080/21683603.2020.1832635

Bicar, V., & Gaylo, D. (2022). Cluster characterization of countries’ performance in mathematics olympiad: Input to mathematics education. Science International (Lahore), 34(5), 503-506. https://doi.org/10.5281/zenodo.7240929

Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77-101. https://doi.org/10.1191/1478088706qp063oa

Braun, V., & Clarke, V. (2012). Thematic analysis. In H. Cooper, P. M. Camic, D. L. Long, A. T. Panter, D. Rindskopf, & K. J. Sher (Eds.), APA handbook of research methods in psychology, Vol. 2. Research designs: Quantitative, qualitative, neuropsychological, and biological (pp. 57-71). American Psychological Association. https://doi.org/10.1037/13620-004

Cagasan, L., Luo, R., Robertson, P., & Care, E. (2016). Formative assessment project: Phase 2 research report. Assessment, Curriculum, and Technology Research Centre (ACTRC). Melbourne and Manila. http://bit.ly/3up0Ej3

Caine, V., Clandinin, D. J., & Lessard, S. (2022). Narrative inquiry: Philosophical roots. Bloomsbury Publishing. https://doi.org/10.5040/9781350142084

Chi, C. (2023, December 6). Philippines still lags behind world in math, reading and science — PISA 2022. Philstar Global Corp. https://bit.ly/3whiHIx

Clores, M. A., & Reganit, A. A. R. (2020). Investigating the assessment literacy of teachers in private junior high schools in the Philippines. Humanities, Arts and Social Sciences Studies, 20(2), 461-476. https://doi.org/10.14456/hasss.2020.17

Close, S., & Shiel, G. (2014, June 24-25). A Comparison of TIMSS 2011 and PISA 2012 math frameworks and performance for Ireland and selected countries [Paper Presentation]. Science and Mathematics Education Conference, Dublin City University.

Department of Education. (2019). PISA 2018: National report of the Philippines. https://bit.ly/3SYYsbq

Downing, S. M. (2006). Twelve steps for effective test development. In S. M. Downing & T. M. Haladyna (Eds.), Handbook of test development (pp. 3-25). Lawrence Erlbaum.

Duff, P. A. (2012). How to carry out case study research. In A. Mackey & S. M. Gass (Eds.), Research methods in second language acquisition: A practical guide (pp. 95-116). Blackwell Publishing.

Elshawa, N., Heng, C. S., Abdullah, A. N., & Rashid, S. (2016). Teachers' assessment literacy and washback effect of assessment. International Journal of Applied Linguistics and English Literature, 5(4), 135-141. https://bit.ly/3UlhCri

Espinosa, A. A., Gomez, M. A. C., Reyes, A. S., Macahilig, H. B., Cortez, L. A. S., & David, A. P. (2023). International large-scale assessment (ILSA): Implications for pre-service teacher education in the Philippines. Issues in Educational Research, 33(2), 553-569.  https://www.iier.org.au/iier33/espinosa.pdf

Frey, B. B., Petersen, S., Edwards, L. M., Pedrotti, J. T., & Peyton, V. (2005). Item-writing rules: Collective wisdom. Teaching and Teacher Education21(4), 357-364. https://doi.org/10.1016/j.tate.2005.01.008

Fulkerson, D., Nichols, P., & Mittelholtz, D. (2010, May 3). What item writers think when writing items: Towards a theory of item-writing expertise [Paper Presentation]. Annual Meeting of the American Educational Research Association, Denver, CO.

Fulkerson, D., Nichols, P., & Snow, E. (2011, April 8-12). Expanding the model of item-writing expertise: Cognitive processes and requisite knowledge structures [Paper Presentation].  Annual Meeting of the American Educational Research Association, New Orleans, LA.

Gaylo, D. N., Caingcoy, M. E., & Mugot, D. C. (2020). Utilization of scholarly journal articles in the teaching and learning of teacher education courses. Balkan and Near Eastern Journal of Social Sciences, 6(3), 59-66. https://doi.org/10.47696/adved.202038

Geisinger, K. F., & Usher-Tate, B. J. (2016). A brief history of educational testing and psychometrics. In C. S. Wells & M. Faulkner-Bond (Eds.), Educational measurement from foundations to future (pp. 3-20). The Guilford Press.

Griffin, P., Cagasan, L., Care, E., Vista, A., & Nava, F. (2016). Formative assessment policy and its enactment in the Philippines. In D. Laveault & L. Allal (Eds.), Assessment for learning: Meeting the challenge of implementation (pp. 75-92). Springer.  https://doi.org/10.1007/978-3-319-39211-0_5

Hailaya, W. M. (2014). Teacher assessment literacy and student outcomes in the Province of Tawi-Tawi, Philippines [Doctoral thesis, The University of Adelaide]. https://bit.ly/3I0BeLD

Hanafi, N. B. M., Ali, N. B. M., Zamani, S. B., Yamin, N. A. B., & Ismail, N. N. B. (2020). Examining assessment literacy: A study of technical teacher. European Journal of Molecular and Clinical Medicine7(8), 705-717. https://bit.ly/3I1OT4Y

Haw, J. Y., King, R. B., & Trinidad, J. E. R. (2021). Need supportive teaching is associated with greater reading achievement: What the Philippines can learn from PISA 2018. International Journal of Educational Research, 110, Article 101864. https://doi.org/10.1016/j.ijer.2021.101864

International Association for the Evaluation of Educational Achievement. (2020). TIMSS 2019 trends in international math and science study Philippines country report. https://bit.ly/4bG4DbM

Johnson, M., Constantinou, F., & Crisp, V. (2017). How do question writers compose external examination questions? Question writing as a socio‐cognitive process. British Educational Research Journal43(4), 700-719. https://doi.org/10.1002/berj.3281

Khalid, N. H. M., Latif, A. A., & Yusof, I. J. (2021). Assessment literacy: A systematic literature review and research agenda. Annals of the Romanian Society for Cell Biology, 25(3), 4668-4696.  https://bit.ly/3uG31hd

King, N. (2004). Using templates in the thematic analysis of text. In C. Cassell & G. Symon (Eds.), Essential guide to qualitative methods in organizational research (pp. 256-270). Sage. https://doi.org/10.4135/9781446280119.n21

Kohar, A. W., Zulkardi, Z., & Darmawijoyo, D. (2014). Developing PISA-like math tasks to promote students’ mathematical literacy. In R. Ilma (Ed.), Proceeding in the Second South East Asia Design - Development Research (SEA-DR) Conference (pp. 14-26). Universitas Sriwijaya.

Kuger, S., & Klieme, E. (2016). Dimensions of context assessment. In S. Kuger, E. Klieme, N. Jude, & D. Kaplan (Eds), Assessing contexts of learning (pp. 3-37). Methodology of Educational Measurement and Assessment. Springer. https://doi.org/10.1007/978-3-319-45357-6_1

Kyriakides, L., Christoforou, C., & Charalambous, C. Y. (2013). What matters for student learning outcomes: A meta-analysis of studies exploring factors of effective teaching. Teaching and Teacher Education36, 143-152. https://doi.org/10.1016/j.tate.2013.07.010

Lapinid, M. R. C., Cordell, M. O., II, Teves, J. M., Yap, S. A., Chua, U., & Bernardo, A. B. I. (2022). Addressing the poor mathematics performance of Filipino learners: Beyond curricular and instructional interventions. DLSU-AKI Policy Brief, 14(2), 1-4. https://bit.ly/3WmQ9Ip

Lasaten, R. C. S. (2016). Assessment methods, problems and training needs of public high school teachers in English. International Journal of Languages, Literature and Linguistics, 2(2), 55-60. https://doi.org/10.18178/ijlll.2016.2.2.67

Lian, L. H., Yew, W. T., & Meng, C. C. (2014). Enhancing Malaysian teachers’ assessment literacy. International Education Studies, 7(10), 74-81. https://doi.org/10.5539/ies.v7n10p74

Magno, C. (2003). The profile of teacher-made test construction of the professors of University of Perpetual Help Laguna. UPHL Institutional Journal, 1(1), 48-55. https://ssrn.com/abstract=1429347

Mellati, M., & Khademi, M. (2018). Exploring teachers' assessment literacy: Impact on learners' writing achievements and implications for teacher development. Australian Journal of Teacher Education, 43(6), Article 1. https://doi.org/10.14221/ajte.2018v43n6.1

Memisevic, H., & Biscevic, I. (2022). Mathematics, gender and the meaning in life: The results of PISA testing in Bosnia and Herzegovina. European Journal of Mathematics and Science Education, 3(2), 171-179. https://doi.org/10.12973/ejmse.3.2.171

Montemayor, M. T. (2023, December 6). CHED to address PH students’ low int’l assessment ranking. Philippine News Agency. https://www.pna.gov.ph/articles/1215002

Mullis, I. V. S., Martin, M. O., & Loveless, T. (2016). 20 years of TIMSS: International trends in math and science achievement, curriculum, and instruction. International Association for the Evaluation of Educational Achievement (IEA).  https://bit.ly/3UESGN8

Namoco, S., & Zaharudin, R. (2021). Pedagogical beliefs and learning assessment in Science: Teacher’s experiences anchored on theory of reasoned action. Journal of Turkish Science Education, 18(2), 304-319. https://doi.org/10.36681/tused.2021.67  

Napanoy, J. B., & Peckley, M. K. (2020). Assessment literacy of public elementary school teachers in the indigenous communities in Northern Philippines. Universal Journal of Educational Research, 8(11B), 5693-5703. https://doi.org/10.13189/ujer.2020.082203

Orbeta, A. C., Melad, K. A. M., & Potestad, M. (2020). Correlates of test performance of 15-year-old students in the Philippines: Evidence from PISA (No. 2020-57). PIDS Discussion Paper Series. https://bit.ly/3T03sw8

Organisation for Economic Cooperation and Development. (2019). PISA 2018 assessment and analytical framework. OECD Publishing. https://bit.ly/44lKfcC

Piosang, T. L. (2017). A cross-sectional analysis of classroom assessment literacy of English teachers in secondary and tertiary levels. Educational Measurement and Evaluation Review, 8(1), 30-48. https://bit.ly/3y0d2qX

Popham, W. J. (2011). Assessment literacy overlooked: A teacher educator's confession. The Teacher Educator, 46(4), 265-273. https://doi.org/10.1080/08878730.2011.605048

Radišić, J., & Baucal, A. (2018). Teachers’ reflection on PISA items and why they are so hard for students in Serbia. European Journal of Psychology of Education, 33, 445-466. https://doi.org/10.1007/s10212-018-0366-0

Rodriguez, M. C. (1997, March 24-28). The art & science of item-writing: A meta-analysis of multiple-choice item format effects [Paper Presentation]. Annual Meeting of the American Educational Research Association, Chicago.

Rodriguez, M. C., & Haladyna, T. M. (2013). Writing selected-response items for classroom assessment. In J. McMillan (Ed.), SAGE handbook of research on classroom assessment (pp. 293-312). SAGE Publications, Inc. https://doi.org/10.4135/9781452218649.n17

Schleicher, A. (2019). PISA 2018: Insights and interpretations. OECD Publishing. https://bit.ly/4bG36lK

Sonday, A., Ramugondo, E., & Kathard, H. (2020). Case study and narrative inquiry as merged methodologies: A critical narrative perspective. International Journal of Qualitative Methods, 19, 1-5. https://doi.org/10.1177/1609406920937880

Tatto, M. T., Rodriguez, M. C., Reckase, M. D., Smith, W. M., Bankov, K., & Pippin, J. (2020). The FIRSTMATH study: Concepts, methods, and strategies for comparative international research in mathematics education. In M. T. Tatto, M. C. Rodriguez, M. D. Reckase, W. M. Smith, K. Bankov, & J. Pippin, The first five years of teaching mathematics (FIRSTMATH): Concepts, methods, and strategies for comparative international research in math education (pp. 1-20). Springer International Publishing. https://doi.org/10.1007/978-3-030-44047-3_1

United Nations Children's Fund. (2021). SEA-PLM 2019 Southeast Asia primary learning metrics Philippines country report. https://bit.ly/49cBlQ5

United Nations Children’s Fund & Southeast Asian Ministers of Education Organization. (2019). SEA-PLM 2019 Assessment Framework (1st ed.). United Nations Children’s Fund (UNICEF) & Southeast Asian Ministers of Education Organization (SEAMEO) – SEA-PLM Secretariat. https://bit.ly/3SWlSOA

Wu, M. L. (2009, June 2). A critical comparison of the contents of PISA and TIMSS mathematics assessments [Paper Presentation]. NCES “What we can learn from PISA” research conference, Washington, DC.

Xu, Y., & Brown, G. T. L. (2016). Teacher assessment literacy in practice: A reconceptualization. Teaching and Teacher Education, 58, 149-162. https://doi.org/10.1016/j.tate.2016.05.010

Zulkardi, Z., & Kohar, A. W. (2018). Designing PISA-like math tasks in Indonesia: Experiences and challenges. Journal of Physics: Conference Series, 947, Article 012015. https://doi.org/10.1088/1742-6596/947/1/012015

...