Quantifying Influence: Propensity Score Matching Unravels the True Effect Sizes of Learning Management Models on Students’ Analytical Thinking
Analytical thinking is crucial for developing problem-solving, decision-making, and higher-order thinking skills. Many researchers have consistently d.
- Pub. date: October 15, 2024
- Online Pub. date: May 26, 2024
- Pages: 1535-1553
- 259 Downloads
- 925 Views
- 0 Citations
Analytical thinking is crucial for developing problem-solving, decision-making, and higher-order thinking skills. Many researchers have consistently developed learning management models to enhance students' analytical thinking, resulting in extensive knowledge but lacking clear systematic summaries. This study aims to: (a) explore the effect sizes and research characteristics influencing students' analytical thinking, and (b) compare the effect sizes of learning management models after adjusting for propensity score matching. In exploring 131 graduate research papers published between 2002 and 2021, the research utilized forms for recording research characteristics and questionnaires for assessing research quality for data collection. Effect sizes were calculated using Glass's method, while data analysis employed random effects, fixed effects, and regression meta-analysis methods. The findings indicate that (a) research on learning management models significantly impacts students' analytical thinking at a high level (d̅ = 1.428). Seven research characteristics, including year of publication, field of research, level, duration per plan, learning management process, measurement and evaluation, and research quality, statistically influence students' analytical thinking, and (b) after propensity score matching, learning through techniques such as KWL, KWL-plus, Six Thinking Hats, 4MAT, and Mind Mapping had the highest influence on students' analytical thinking. Recommendations for developing students' analytical thinking involve creating a learning management process that fosters understanding, systematic practical training, expanding thinking through collaborative exchanges, and assessments using learning materials and tests to stimulate increased analytical thinking.
analytical thinking learning management models meta analysis propensity score matching research synthesis
Keywords: Analytical thinking, learning management models, meta-analysis, propensity score matching, research synthesis.
References
Adesope, O. O., Trevisan, D. A., & Sundararajan, N. (2017). Rethinking the use of tests: A meta-analysis of practice testing. Review of Educational Research, 87(3), 659-701. https://doi.org/10.3102/0034654316689306
Ahmed, I., Sutton, A. J., & Riley, R. D. (2012). Assessment of publication bias, selection bias, and unavailable data in meta-analyses using individual participant data: a database survey. BMJ, 344, Article d7762. https://doi.org/10.1136/bmj.d7762
Akinbadewa, B. O., & Sofowora, O. A. (2020). The effectiveness of multimedia instructional learning packages in enhancing secondary school students’ attitudes toward Biology. International Journal on Studies in Education, 2(2), 119-133. https://doi.org/10.46328/ijonse.19
Akkerman, S., Admiraal, W., Brekelmans, M., & Oost, H. (2008). Auditing quality of research in social sciences. Quality and Quantity, 42, 257-274. https://doi.org/10.1007/s11135-006-9044-4
Amer, A. (2005). Analytical thinking. Center for Advancement of Postgraduate Studies and Research in Engineering Sciences, Faculty of Engineering- Cairo University. https://bit.ly/3TuvErD
Ariyasinsomboon, W. (2001). การสังเคราะห์งานวิจัยในสาขาจิตวิทยาการศึกษา : การวิเคราะห์อภิมาน [Synthesis of research in the field of educational psychology: a meta-analysis] [Doctoral dissertation, Chulalongkorn University]. Chulalongkorn University Intellectual Repository. https://bit.ly/3W2zB8v
Austin, P. C. (2009). The relative ability of different propensity score methods to balance measured covariates between treated and untreated subjects in observational studies. Medical Decision Making, 29(6), 661-677. https://doi.org/10.1177/0272989X09341755
Badhiwala, J. H., Karmur, B. S., & Wilson, J. R. (2021). Propensity score matching: A powerful tool for analyzing observational nonrandomized data. Clinical Spine Surgery, 34(1), 22-24. https://doi.org/10.1097/BSD.0000000000001055
Bai, H. (2011). A comparison of propensity score matching methods for reducing selection bias. International Journal of Research and Method in Education, 34(1), 81-107. https://doi.org/10.1080/1743727X.2011.552338
Balta, N., & Sarac, H. (2016). The effect of 7e learning cycle on learning in science teaching: a meta-analysis study. European Journal of Educational Research, 5(2), 61-72. https://doi.org/10.12973/eu-jer.5.2.61
Benedetto, U., Head, S. J., Angelini, G. D., & Blackstone, E. H. (2018). Statistical primer: propensity score matching and its alternatives. European Journal of Cardio-Thoracic Surgery, 53(6), 1112-1117. https://doi.org/10.1093/ejcts/ezy167
Berkhout, S. W., Haaf, J. M., Gronau, Q. F., Heck, D. W., & Wagenmakers, E.-J. (2024). A tutorial on Bayesian model-averaged meta-analysis in JASP. Behavior Research Methods, 56, 1260-1282. https://doi.org/10.3758/s13428-023-02093-6
Binkley, M., Erstad, O., Herman, J., Raizen, S., Ripley, M., Miller-Ricci, M., & Rumble, M. (2012). Defining twenty-first century skills. In P. Griffin, B. McGaw, & E. Care (Eds.), Assessment and teaching of 21st century skills (pp. 17-66). Springer. https://doi.org/10.1007/978-94-007-2324-5_2
Bloom, B. S. (Ed.). (1956). Taxonomy of educational objectives: The Classification of Educational Goals. Longmans.
Borenstein, M., Hedges, L. V., Higgins, J. P., & Rothstein, H. R. (Eds.). (2021). Introduction to meta-analysis (2nd ed.). John Wiley & Sons Ltd.
Burch, G. F., Giambatista, R., Batchelor, J. H., Burch, J. J., Hoover, J. D., & Heller, N. A. (2019). A meta‐analysis of the relationship between experiential learning and learning outcomes. Decision Sciences Journal of Innovative Education, 17(3), 239-273. https://doi.org/10.1111/dsji.12188
Card, N. A. (2012). Applied meta-analysis for social science research. The Guilford Press. https://bit.ly/3HWcMuP
Clark, K. R. (2018). Learning theories: constructivism. Radiologic Technology, 90(2), 180-182. https://bit.ly/3SafIsq
Cooper, H., Hedges, L. V., & Valentine, J. C. (Eds.). (2019). The handbook of research synthesis and meta-analysis (3rd ed.). Russell Sage Foundation. https://doi.org/10.7758/9781610448864
Cornell, D. (2024, January 3). 17 Analytical thinking examples. Helpful Professor. https://bit.ly/3S3lImV
Czodrowski, P. (2014). Count on kappa. Journal of Computer-Aided Molecular Design, 28, 1049-1055. https://doi.org/10.1007/s10822-014-9759-6
Donoghue, G. M., & Hattie, J. A. C. (2021). A meta-analysis of ten learning techniques. Frontiers in Education, 6, Article 581216. https://doi.org/10.3389/feduc.2021.581216
Dowdy, A., Tincani, M., & Schneider, W. J. (2020). Evaluation of publication bias in response interruption and redirection: A meta‐analysis. Journal of Applied Behavior Analysis, 53(4), 2151-2171. https://doi.org/10.1002/jaba.724
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. BMJ, 315, 629-634. https://doi.org/10.1136/bmj.315.7109.629
Elder, L., & Paul, R. (2007). The thinker’s guide to analytic thinking. The Foundation for Critical Thinking. https://bit.ly/4bOFcE1
Esterhuizen, T. M., & Thabane, L. (2016). Con: Meta-analysis: some key limitations and potential solutions. Nephrology Dialysis Transplantation, 31(6), 882-885. https://doi.org/10.1093/ndt/gfw092
Ferguson, C. J., & Brannick, M. T. (2012). Publication bias in psychological science: Prevalence, methods for identifying and controlling, and implications for the use of meta-analyses. Psychological Methods, 17(1), 120-128. https://doi.org/10.1037/a0024445
Few, S. (2015, November). A course of study in analytical thinking. Perceptual Edge. https://bit.ly/3yz8SGT
Gillies, R. M. (2014). Cooperative learning: Developments in research. International Journal of Educational Psychology, 3(2), 125-140. http://dx.doi.org/10.4471/ijep.2014.08
Glass, G. V. (1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5(10), 3-8. https://doi.org/10.3102/0013189X005010003
González-Pérez, L. I., & Ramírez-Montoya, M. S. (2022). Components of education 4.0 in 21st century skills frameworks: systematic review. Sustainability, 14(3), Article 1493. https://doi.org/10.3390/su14031493
Harbord, R. M., Harris, R. J., & Sterne, J. A. (2009). Updated tests for small-study effects in meta-analyses. The Stata Journal, 9(2), 197-210. https://doi.org/10.1177/1536867X0900900202
Harris, H., & Horst, S. J. (2016). A brief guide to decisions at each step of the propensity score matching process. Practical Assessment, Research, and Evaluation, 21(4), 1-11. https://doi.org/10.7275/yq7r-4820
Harrison, J. S., Banks, G. C., Pollack, J. M., O’Boyle, E. H., & Short, J. (2017). Publication bias in strategic management research. Journal of Management, 43(2), 400-425. https://doi.org/10.1177/0149206314535438
Haukoos, J. S., & Lewis, R. J. (2015). The propensity score. JAMA, 314(15), 1637-1638. https://doi.org/10.1001/jama.2015.13480
Itsarangkul Na Ayutthaya, T., & Damrongpanit, S. (2022a). A meta-analysis of instructional management models affecting creative thinking development. European Journal of Educational Research, 11(4), 2069-2085. https://doi.org/10.12973/eu-jer.11.4.2069
Itsarangkul Na Ayutthaya, T., & Damrongpanit, S. (2022b). A meta-analysis of instructional management model for students' creative thinking development: An application of propensity score matching. European Journal of Educational Research, 11(4), 2429-2444. https://doi.org/10.12973/eu-jer.11.4.2429
Jacobs, G. M., & Renandya, W. A. (2019). Student centered cooperative learning: Linking concepts in education to promote student learning. Springer. https://doi.org/10.1007/978-981-13-7213-1
Kane, L. T., Fang, T., Galetta, M. S., Goyal, D. K. C., Nicholson, K. J., Kepler, C. K., Vaccaro, A. R., & Schroeder, G. D. (2020). Propensity score matching: a statistical method. Clinical Spine Surgery, 33(3), 120-122. https://doi.org/10.1097/BSD.0000000000000932
Kao, C.-Y. (2014). Exploring the relationships between analogical, analytical, and creative thinking. Thinking Skills and Creativity, 13, 80-88. https://doi.org/10.1016/j.tsc.2014.03.006
Kennedy, T. J., & Sundberg, C. W. (2020). 21st century skills. In B. Akpan, & T. J. Kennedy (Eds.), Science education in theory and practice (pp. 479-496). Springer. https://doi.org/10.1007/978-3-030-43620-9_32
Khairani, Z., Nasution, D., & Bukit, N. (2021). Analysis of science process skills using learning cycle 7e. Journal of Physics: Conference Series, 1811, Article 012085. https://doi.org/10.1088/1742-6596/1811/1/012085
Kmet, L. M., Cook, L. S., & Lee, R. C. (2004). Standard quality assessment criteria for evaluating primary research papers from a variety of fields. Alberta Heritage Foundation for Medical Research. https://bit.ly/3JuWIAU
Krathwohl, D. R. (2002). A revision of Bloom's taxonomy: An overview. Theory Into Practice, 41(4), 212-218. https://doi.org/10.1207/s15430421tip4104_2
Lambert, C. T., & Guillette, L. M. (2021). The impact of environmental and social factors on learning abilities: a meta‐analysis. Biological Reviews, 96(6), 2871-2889. https://doi.org/10.1111/brv.12783
Lampert, M., & Graziani, F. (2009). Instructional activities as a tool for teachers' and teacher educators' learning. The Elementary School Journal, 109(5), 491-509. https://doi.org/10.1086/596998
Lee, Y.-H. (2019). Strengths and limitations of meta-analysis. The Korean Journal of Medicine, 94(5), 391-395. https://doi.org/10.3904/kjm.2019.94.5.391
Lin, L., & Chu, H. (2018). Quantifying publication bias in meta‐analysis. Biometrics, 74(3), 785-794. https://doi.org/10.1111/biom.12817
Moallem, M., Hung, W., & Dabbagh, N. (Eds.). (2019). The Wiley handbook of problem-based learning. John Wiley & Sons. https://doi.org/10.1002/9781119173243
Morgan, C. J. (2018). Reducing bias using propensity score matching. Journal of Nuclear Cardiology, 25(2), 404-406. https://doi.org/10.1007/s12350-017-1012-y
Morrison, G. R., Ross, S. J., Morrison, J. R., & Kalman, H. K. (2019). Designing effective instruction. John Wiley & Sons.
Moust, J., Bouhuijs, P., & Schmidt, H. (2021). Introduction to problem-based learning (4th ed.). Routledge. https://doi.org/10.4324/9781003194187
Nakagawa, S., Lagisz, M., Jennions, M. D., Koricheva, J., Noble, D. W. A., Parker, T. H., Sánchez-Tójar, A., Yang, Y., & O'Dea, R. E. (2022). Methods for testing publication bias in ecological and evolutionary meta‐analyses. Methods in Ecology and Evolution, 13(1), 4-21. https://doi.org/10.1111/2041-210X.13724
Nicol, C. B., Gakuba, E., & Habinshuti, G. (2020). An overview of learning cycles in science inquiry-based instruction. African Journal of Educational Studies in Mathematics and Sciences, 16(2), 67-81. https://doi.org/10.4314/ajesms.v16i2.5
Niu, L., Behar-Horenstein, L. S., & Garvan, C. W. (2013). Do instructional interventions influence college students’ critical thinking skills? A meta-analysis. Educational Research Review, 9, 114-128. https://doi.org/10.1016/j.edurev.2012.12.002
Patton, M. Q. (1999). Enhancing the quality and credibility of qualitative analysis. Health Services Research, 34(5), 1189-1208. https://bit.ly/3RrxWpb
Ramadani, A. S., Supardi, Z. A. I., Tukiran, & Hariyono, E. (2021). Profile of analytical thinking skills through inquiry-based learning in science subjects. Studies in Learning and Teaching, 2(3), 45-60. https://doi.org/10.46627/silet.v2i3.83
Rannikmäe, M., Holbrook, J., & Soobard, R. (2020). Social constructivism-Jerome Bruner. In B. Akpan, & T. J. Kennedy (Eds.), Science education in theory and practice (pp. 259-275). Springer. https://doi.org/10.1007/978-3-030-43620-9_18
Rasheva-Yordanova, K., Iliev, E., & Nikolova, B. (2018). Analytical thinking as a key competence for overcoming the data science divide. In L.G. Chova, A. L. Martínez, & I. C. Torres (Eds.), EDULEARN18 Proceedings: 10th International Conference on Education and New Learning Technologies (pp. 7892-7898). IATED Academy. https://doi.org/10.21125/edulearn.2018.1833
Robbins, J. K. (2011). Problem solving, reasoning, and analytical thinking in a classroom environment. The Behavior Analyst Today, 12(1), 41-48. https://doi.org/10.1037/h0100710
Rodrangsee, B., Tuntiwongwanich, S., Pimdee, P., & Moto, S. (2022). Development of an online active learning model using the theory of multiple intelligence to encourage Thai undergraduate student analytical thinking skills. Journal of Higher Education Theory and Practice, 22(12), 63-75. https://doi.org/10.33423/jhetp.v22i12.5463
Rosenbaum, P. R., & Rubin, D. B. (2023). Propensity scores in the design of observational studies for causal effects. Biometrika, 110(1), 1-13. https://doi.org/10.1093/biomet/asac054
Rowland, C. A. (2014). The effect of testing versus restudy on retention: A meta-analytic review of the testing effect. Psychological Bulletin, 140(6), 1432–1463. https://doi.org/10.1037/a0037559
Rubin, D. B. (1997). Estimating causal effects from large data sets using propensity scores. Annals of Internal Medicine, 127(8), 757-763. https://doi.org/10.7326/0003-4819-127-8_Part_2-199710151-00064
Rücker, G., Schwarzer, G., Carpenter, J. R., & Schumacher, M. (2008). Undue reliance on /2 in assessing heterogeneity may mislead. BMC Medical Research Methodology, 8, Article 79. https://doi.org/10.1186/1471-2288-8-79
Sailer, M., & Homner, L. (2020). The gamification of learning: A meta-analysis. Educational Psychology Review, 32, 77-112. https://doi.org/10.1007/s10648-019-09498-w
Sartika, S. B. (2018). Teaching models to increase students’ analytical thinking skills. In T. W. Abadi, N. E. T. I. Santosa, M. T. Multazam, F. Megawati (Eds.), Proceedings of the 1st International Conference on Intellectuals' Global Responsibility (pp. 216-218). Atlantis Press. https://doi.org/10.2991/icigr-17.2018.52
Saylor, J. G., Alexander, W. M., & Lewis, A. J. (1981). Curriculum planning for better teaching and learning (4th ed.). Holt, Rinehart and Winston.
Sedgwick, P., & Marston, L. (2015). How to read a funnel plot in a meta-analysis. BMJ, 351, Article h4718. https://doi.org/10.1136/bmj.h4718
Sitthipon, A.-I. (2012). Development of teachers' learning management emphasizing on analytical thinking in Thailand. Procedia-Social and Behavioral Sciences, 46, 3339-3344. https://doi.org/10.1016/j.sbspro.2012.06.063
Slavin, R. E. (2014). Aprendizaje cooperativo y rendimiento académico:¿ por qué funciona el trabajo en grupo? [Cooperative learning and academic achievement: Why does groupwork work?]. Annals of Psychology/Anales De Psicología, 30(3), 785-791. https://doi.org/10.6018/analesps.30.3.201201
Smith, P. L., & Ragan, T. J. (2005). A framework for instructional strategy design. In P. L. Smith, & T. J. Ragan (Eds.), Instructional design (3rd ed., pp. 127-150). Wiley & Sons. https://bit.ly/3U79n28
Spaska, A. M., Savishchenko, V. M., Komar, O. A., Нritchenko, T. Y., & Maidanyk, O. V. (2021). Enhancing analytical thinking in tertiary students using debates. European Journal of Educational Research, 10(2), 879-889. https://doi.org/10.12973/eu-jer.10.2.879
Staffa, S. J., & Zurakowski, D. (2018). Five steps to successfully implement and evaluate propensity score matching in clinical research studies. Anesthesia and Analgesia, 127(4), 1066-1073. https://doi.org/10.1213/ANE.0000000000002787
StataCorp. (2023). Stata meta-analysis reference manual: Release 18. Stata Press. https://www.stata.com/manuals/meta.pdf
Stone, D. L., & Rosopa, P. J. (2017). The advantages and limitations of using meta-analysis in human resource management research. Human Resource Management Review, 27(1), 1-7. https://doi.org/10.1016/j.hrmr.2016.09.001
Suyatman, Saputro, S., Sunarno, W., & Sukarmin. (2021). The implementation of research-based learning model in the basic science concepts course in improving analytical thinking skills. European Journal of Educational Research, 10(3), 1051-1062. https://doi.org/10.12973/eu-jer.10.3.1051
Tan, O.-S. (2021). Problem-based learning innovation: Using problems to power learning in the 21st century. Gale Cengage Learning. https://bit.ly/3SYB9yw
Thaneerananon, T., Triampo, W., & Nokkaew, A. (2016). Development of a test to evaluate students' analytical thinking based on fact versus opinion differentiation. International Journal of Instruction, 9(2), 123-138. https://doi.org/10.12973/iji.2016.929a
Thoemmes, F. (2012). Propensity score matching in SPSS. arXiv. https://doi.org/10.48550/arXiv.1201.6385
Triantafyllou, S. A. (2022). Constructivist learning environments. In Proceedings of the 5th international conference on advanced research in teaching and education (pp. 1-6). Diamond Scientific Publishing. https://doi.org/10.33422/5th.icate.2022.04.10
Van Houwelingen, H. C., Arends, L. R., & Stijnen, T. (2002). Advanced methods in meta‐analysis: multivariate approach and meta‐regression. Statistics in Medicine, 21(4), 589-624. https://doi.org/10.1002/sim.1040
Varoglu, L., Yilmaz, A., & Sen, S. (2023). Effect of 5e learning cycle assisted with concept maps on conceptual understanding. Pedagogical Research, 8(3), Article em0161. https://doi.org/10.29333/pr/13167
Vevea, J. L., Coburn, K., & Sutton, A. (2019). Publication bias. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (3rd ed., pp. 383-429). Russell Sage Foundation. https://doi.org/10.7758/9781610448864.21
Wagner, T. (2008). The global achievement gap: Why even our best schools don't teach the new survival skills our children need, and what we can do about it. Basic Books.
World Economic Forum. (2023). Future jobs report 2023. https://bit.ly/3Hqg8Gn
Xu, E., Wang, W., & Wang, Q. (2023). The effectiveness of collaborative problem solving in promoting students’ critical thinking: A meta-analysis based on empirical literature. Humanities and Social Sciences Communications, 10, Article 16. https://doi.org/10.1057/s41599-023-01508-1