Combining Performance-Based and Self-Reported Measures of Executive Functions: Are Both Meaningful in Predicting Study Success in Higher Education Students?
Diane Marcia Manuhuwa
,
Mirjam Snel-de-Boer
,
Jan Willem de-Graaf
,
Joke Fleer
Research in higher education has revealed a significant connection between executive functions (EF) and study success. Previous investigations have ty.
- Pub. date: October 15, 2024
- Online Pub. date: June 21, 2024
- Pages: 1647-1663
- 16 Downloads
- 88 Views
- 0 Citations
Research in higher education has revealed a significant connection between executive functions (EF) and study success. Previous investigations have typically assessed EF using either neuropsychological tasks, which provide direct and objective measures of core EF such as inhibition, working memory, and cognitive flexibility, or self-report questionnaires, which offer indirect and subjective assessments. However, studies rarely utilize both assessment methods simultaneously despite their potential to offer complementary insights into EF. This study aims to evaluate the predictive capabilities of performance-based and self-reported EF measures on study success. Employing a retrospective cohort design, 748 first-year Applied Psychology students completed performance-based and self-report questionnaires to assess EF. Maximum likelihood correlations were computed for 474 students, with data from 562-586 first-year students subsequently subjected to hierarchical regression analysis, accommodating pairwise missing values. Our results demonstrate minimal overlap between performance-based and self-reported EF measures. Additionally, the model incorporating self-reported EF accounted for 13% of the variance in study success after one year, with the inclusion of performance-based EF raising this proportion to 16%. Self-reported EF assessments modestly predict study success. However, monitoring levels of self-reported EF could offer valuable insights for students and educational institutions, given that EF play a crucial role in learning. Additionally, one in five students reports experiencing significant EF difficulties, highlighting the importance of addressing EF concerns for learning and study success.
Keywords: Executive functions, higher education, performance-based measurements, self-reported measurements, study success.
References
Abeare, K., Razvi, P., Sirianni, C. D., Giromini, L., Holcomb, M., Cutler, L., Kuzmenka, P., & Erdodi, L. A. (2021). Introducing alternative validity cutoffs to improve the detection of non-credible symptom report on the BRIEF. Psychological Injury and Law, 14, 2-16. https://doi.org/10.1007/s12207-021-09402-4
Alloway, T. P., Banner, G. E., & Smith, P. (2010). Working memory and cognitive styles in adolescents’ attainment. British Journal of Educational Psychology, 80(4), 567-581. https://doi.org/10.1348/000709910X494566
Ato, M., López-García, J. J., & Benavente, A. (2013). A classification system for research designs in psychology. Anales de Psicología, 29(3), 1038-1059. https://doi.org/10.6018/analesps.29.3.178511
Baars, M. A. E., Nije Bijvank, M., Tonnaer, G. H., & Jolles, J. (2015). Self-report measures of executive functioning are a determinant of academic performance in first-year students at a University of Applied Sciences. Frontiers in Psychology, 6, Article 1131. https://doi.org/10.3389/fpsyg.2015.01131
Baddeley, A. (2010). Working memory. Current Biology, 20(4), 136-140. https://doi.org/10.1016/j.cub.2009.12.014
Baggetta, P., & Alexander, P. A. (2016). Conceptualization and operationalization of executive function. Mind, Brain, and Education, 10(1), 10-33. https://doi.org/10.1111/mbe.12100
Barkley, R. A. (2012). Executive functions. What they are, how they work and why they evolved. Guilford Press.
Barkley, R. A., & Fischer, M. (2011). Predicting impairment in major life activities and occupational functioning in hyperactive children as adults: Self-reported executive function (EF) deficits versus EF tests. Developmental Neuropsychology, 36(2), 137-161. https://doi.org/10.1080/87565641.2010.549877
Bauer, R. M., Iverson, G. L., Cernich, A. N., Binder, L. M., Ruff, R. M., & Naugle, R. I. (2012). Computerized neuropsychological assessment devices: Joint position paper of the American Academy of Clinical Neuropsychology and the National Academy of Neuropsychology. The Clinical Neuropsychologist, 26(2), 177-196. https://doi.org/10.1080/13854046.2012.663001
Bodenburg, S., Wendiggensen, J., & Kasten, E. (2022). Scores in self‑report questionnaires assessing adult ADHD can be influenced by negative response bias but are unrelated to performance on executive function and attention tests. Psychological Injury and Law, 15, 189-199. https://doi.org/10.1007/s12207-022-09448-y
Brooks, B. L., & Sherman, E. M. S. (2012). Computerized neuropsychological testing to rapidly evaluate cognition in pediatric patients with neurologic disorders. Journal of Child Neurology, 27(8), 982-991. http://doi.org/10.1177/0883073811430863
Campman, C., Van Ranst, D., Meijer, J. W., & Sitskoorn, M. (2017). Computerized screening for cognitive impairment in patients with COPD. International Journal of Chronic Obstructive Pulmonary Disease, 12, 3075-3083. https://doi.org/10.2147/COPD.S142871
Carragher, J., & McGaughey, J. (2016). The effectiveness of peer mentoring in promoting a positive transition to higher education for first-year undergraduate students: A mixed methods systematic review protocol. Systematic Reviews, 5, Article 68. https://doi.org/10.1186/s13643-016-0245-1
CNS Vital Signs. (2023). CNS Vital Signs interpretation guide https://bit.ly/3XbicLj
Cortés Pascual, A., Moyano Muñoz, N., & Quilez Robres, A. (2019). The relationship between executive functions and academic performance in primary education: Review and meta-analysis. Frontiers in Psychology, 10, Article 1582. https://doi.org/10.3389/fpsyg.2019.01582
Diamond, A. (2013). Executive functions. Annual Review of Psychology, 64, 135-168. https://doi.org/10.1146/annurev-psych-113011-143750
Effeney, G., Carroll, A., & Bahr, N. (2013). Self-regulated learning and executive function: Exploring the relationships in a sample of adolescent males. Educational Psychology, 33(7), 773-796. https://doi.org/10.1080/01443410.2013.785054
Follmer, D. J. (2021). Examining the role of calibration of executive function performance in college learners’ regulation. Applied Cognitive Psychology, 35(3), 646-658. https://doi.org/10.1002/acp.3787
Gaskin, C. J., & Happell, B. (2014). On exploratory factor analysis: A review of recent evidence, an assessment of current practice, and recommendations for future use. International Journal of Nursing Studies, 51(3), 511-521. https://doi.org/10.1016/j.ijnurstu.2013.10.005
Gathercole, S. E., Pickering, S. J., Knight, C., & Stegmann, Z. (2004). Working memory skills and educational attainment: Evidence from national curriculum assessments at 7 and 14 years of age. Applied Cognitive Psychology, 18(1), 1-16. https://doi.org/10.1002/acp.934
Giedd, J. N., & Rapoport, J. L. (2010). Structural MRI of pediatric brain development: What have we learned and where are we going? Neuron, 67(5), 728-734. https://doi.org/10.1016/j.neuron.2010.08.040
Gignac, G. E., & Szodorai, E. T. (2016). Effect size guidelines for individual researchers. Personality and Individual Differences, 102, 74-78. https://doi.org/10.1016/j.paid.2016.06.069
Gualtieri, C. T., & Johnson, L. G. (2006). Reliability and validity of a computerized neurocognitive test battery, CNS Vital Signs. Archives of Clinical Neuropsychology, 21(7), 623-643. https://doi.org/10.1016/j.acn.2006.05.007
Hair, J. F., Hult, G. T. M., Ringle, C. M., & Sarstedt, M. (2022). A primer on partial least squares structural equation modeling (PLS-SEM) (3rd ed.). SAGE Publications.
Huizinga, M., Dolan, C. V., & Van der Molen, M. W. (2006). Age-related change in executive function: Developmental trends and a latent variable analysis. Neuropsychologia, 44(11), 2017-2036. https://doi.org/10.1016/j.neuropsychologia.2006.01.010
Kamradt, J. M., Ullsperger, J. M., & Nikolas, M. A. (2014). Executive function assessment and adult attention-deficit/hyperactivity disorder: Tasks versus ratings on the Barkley Deficits in Executive Functioning Scale. Psychological Assessment, 26(4), 1095-1105. https://doi.org/10.1037/pas0000006
Knouse, L. E., Feldman, G., & Blevins, E. J. (2014). Executive functioning difficulties as predictors of academic performance: Examining the role of grade goals. Learning and Individual Differences, 36, 19-26. https://doi.org/10.1016/j.lindif.2014.07.001
Lorenzo-Seva, U., & Ferrando, P. J. (2021). Not positive definite correlation matrices in exploratory item factor analysis: Causes, consequences and a proposed solution. Structural Equation Modeling: A Multidisciplinary Journal, 28(1), 138-147. https://doi.org/10.1080/10705511.2020.1735393
Manuhuwa, D. M., Snel-De Boer, M., Jaarsma, A. D. C., Fleer, J., & De Graaf, J. W. (2023). Executive functions and self-regulated learning profiles in higher education students and their relationship with study success. In INTED2023 Proceedings: 17th International Technology, Education and Development Conference (INTED) (pp. 2507-2516). IATED. https://doi.org/10.21125/inted.2023
McDonald, J. D. (2008). Measuring personality constructs: The advantages and disadvantages of self-reports, informant reports and behavioural assessments. Enquire, 1(1), 75-94. https://bit.ly/3xo9TAW
Miller, E. K., & Cohen, J. D. (2001). An integrative theory of prefrontal cortex function. Annual Review of Neuroscience, 24, 167-202. https://doi.org/10.1146/annurev.neuro.24.1.167
Miyake, A., Friedman, N. P., Emerson, M. J., Witzki, A. H., Howerter, A., & Wager, T. D. (2000). The unity and diversity of executive functions and their contributions to complex “frontal lobe” tasks: A latent variable analysis. Cognitive Psychology, 41(1), 49-100. https://doi.org/10.1006/cogp.1999.0734
Musso, M. F., Boekaerts, M., Segers, M., & Cascallar, E. C. (2019). Individual differences in basic cognitive processes and self-regulated learning: Their interaction effects on math performance. Learning and Individual Differences, 71, 58-70. https://doi.org/10.1016/j.lindif.2019.03.003
Naglieri, J. A., & Goldstein, S. (2014). Assessment of executive function using rated scales: Psychometric considerations. In S. Goldstein, & J. A. Naglieri (Eds.), Handbook of executive functioning (pp. 159-170). Springer. https://doi.org/10.1007/978-1-4614-8106-5_10
Naglieri, J. A., & Otero, T. M. (2017). Essentials of CAS2 Assessment. John Wiley & Sons, Inc.
Nigg, J. T. (2017). Annual research review: On the relations among self-regulation, self-control, executive functioning, effortful control, cognitive control, impulsivity, risk-taking, and inhibition for developmental psychology. Journal of Child Psychology and Psychiatry, 58(4), 361-383. https://doi.org/10.1111/jcpp.12675
Plourde, V., Hrabok, M., Sherman, E. M. S., & Brooks, B. L. (2018). Validity of a computerized cognitive battery in children and adolescents with neurological diagnoses. Archives of Clinical Neuropsychology, 33(2), 247-253. https://doi.org/10.1093/arclin/acx067
Rabinovici, G. D., Stephens, M. L., & Possin, K. L. (2015). Executive dysfunction. Continuum, 21(3), 646-659. https://doi.org/10.1212/01.CON.0000466658.05156.54
Ramos-Galarza, C., Acosta-Rodas, P., Bolaňos-Pasquel, M., & Lepe-Martínez, N. (2020). The role of executive functions in academic performance and behaviour of university students. Journal of Applied Research in Higher Education, 12(3), 444-455. https://doi.org/10.1108/JARHE-10-2018-0221
Roth, R. M., Isquith, P. K., & Gioia, G. A. (2005). Behavior Rating Inventory of Executive Function®--Adult Version (BRIEF-A) [Database record]. APA PsycTests. https://doi.org/10.1037/t86244-000
Roth, R. M., Isquith, P. K., & Gioia, G. A. (2011). BRIEF-A: Vragenlijst executieve functies voor volwassenen (E. Scholte, & I. Noens, Trans.) HogrEFe. (Original work published 2005). https://bit.ly/4bYNXeO
Roth, R. M., Lance, C. E., Isquith, P. K., Fischer, A. S., & Giancola, P. R. (2013). Confirmatory factor analysis of the Behavior Rating Inventory of Executive Function-Adult Version in healthy adults and application to Attention-Deficit/Hyperactivity Disorder. Archives of Clinical Neuropsychology, 28(5), 425-434. https://doi.org/10.1093/arclin/act031
Rutherford, T., Buschkuehl, M., Jaeggi, S. M., & Farkas, G. (2018). Links between achievement, executive functions, and self-regulated learning. Applied Cognitive Psychology, 32(6), 763-774. https://doi.org/10.1002/acp.3462
Samuel, M., & Okey, L. E. (2015). The relevance and significance of correlation in social science research. International Journal of Sociology and Anthropology Research, 1(3), 22-28. https://bit.ly/3Kxq2HG
Shwartz, S. K., Roper, B. L., Arentsen, T. J., Crouse, E. M., & Adler, M. C. (2020). The Behavior Rating Inventory of Executive Function®-Adult Version is related to emotional distress, not executive dysfunction, in a veteran sample. Archives of Clinical Neuropsychology, 35(6), 701-716. https://doi.org/10.1093/arclin/acaa024
Snyder, H. R., Friedman, N. P., & Hankin, B. L. (2021). Associations between task performance and self-report measures of cognitive control: Shared versus distinct abilities. Assessment, 28(4), 1080-1096. https://doi.org/10.1177/1073191120965694
Suchy, Y., DesRuisseaux, L. A., Mora, M. G., Brothers, S. L., & Niermeyer, M. A. (2024). Conceptualization of the term “ecological validity” in neuropsychological research on executive function assessment: A systematic review and call to action. Journal of the International Neuropsychological Society, 30(5), 499-522. https://doi.org/10.1017/S1355617723000735
Suchy, Y., Mora, M. G., DesRuisseaux, L. A., Niermeyer, M. A., & Brothers, S. L. (2024). Pitfalls in research on ecological validity of novel executive function tests: A systematic review and a call to action. Psychological Assessment, 36(4), 243-261. https://doi.org/10.1037/pas0001297
Toplak, M. E., West, R. F., & Stanovich, K. E. (2013). Practitioner review: Do performance-based measures and ratings of executive function assess the same construct? Journal of Child Psychology and Psychiatry, 54(2), 131-143. https://doi.org/10.1111/jcpp.12001
Toplak, M. E., West, R. F., & Stanovich, K. E. (2016). The assessment of executive functions in Attention-Deficit/Hyperactivity Disorder: Performance-based measures versus ratings of behavior. In J. Hoskyn (Ed.), Executive functions in children’s everyday lives: A handbook for professionals in applied psychology (pp. 157-174). Oxford Academic. https://doi.org/m4mp
VanLunen, D. (2020, March 15). Get a grip! When to add covariates in a linear regression. A guide to accurately and precisely measuring effects! Towards Data Science. https://bit.ly/4amWRSO
Velichkovsky, B. B., Razvaliaeva, A. Y., Khlebnikova, A. A., Manukyan, P. A., Kasatkin, V. N., & Barmin, A. V. (2023). Systematic Review and Meta-Analysis of Clinically Relevant Executive Functions Tests Performance after COVID-19. Behavioural Neurology, 2023, Article 1094267. https://doi.org/10.1155/2023/1094267
Watkins, M. W. (2018). Exploratory factor analysis: A guide to best practice. Journal of Black Psychology, 44(3), 219-246. https://doi.org/10.1177/0095798418771807
Zelazo, P. D., & Carlson, S. M. (2020). The neurodevelopment of executive function skills: Implications for academic achievement gaps. Psychology and Neuroscience, 13(3), 273-298. https://doi.org/10.1037/pne0000208
Zelazo, P. D., & Müller, U. (2010). Executive function in typical and atypical development. In U. Goswami (Ed.), The Wiley-Blackwell handbook of childhood cognitive development (2nd ed., pp. 574-603). Wiley-Blackwell. https://doi.org/10.1002/9781444325485.ch22
Zorza, J. P., Marino, J., & Mesas, A. A. (2016). Executive functions as predictors of school performance and social relationships: Primary and secondary school students. The Spanish Journal of Psychology, 19, Article E23. https://doi.org/10.1017/sjp.2016.23