Numerical Ratings and Content Labeling of Speeches in an Educational Public Speaking Program
Education in public speaking often relies largely on qualitative feedback. In this study, we explored how a numerical feedback form based on a validat.
- Pub. date: April 15, 2023
- Pages: 825-835
- 229 Downloads
- 271 Views
- 0 Citations
Education in public speaking often relies largely on qualitative feedback. In this study, we explored how a numerical feedback form based on a validated scale could be used in an educational public speaking program. Data were collected at three clubs of Toastmasters International. Speech content was labeled to enable statistical analysis of correlations between speech content and audience ratings. 216 speeches by 59 speakers were evaluated by audience members, providing 1416 individual evaluations. All rating categories correlated strongly with each other. Speeches mentioning relationships, personal stories and positive emotions were rated more favorably. Speeches given in-person were rated more highly in several dimensions compared to speeches given through online video. There were some sex differences in choice of content and emotional expressiveness, but not in audience ratings of speech quality. Overall, the study found that the method was useful for both providing individual feedback and aggregated data for research purposes.
Keywords: Evaluations, public speaking, rhetoric, speeches.
References
Aguirre, R. F., Cerbito, A. F., & Gayod, D. H. (2022). Online learning experiences and satisfaction of students on the transition to remote learning. International Multidisciplinary Research Journal, 4(1), 144–154. https://doi.org/10.54476/iimrj18
Allen, J. A., Yoerger, M. A., Lehmann-Willenbrock, N., & Jones, J. (2015). Would you please stop that!?: The relationship between counterproductive meeting behaviors, employee voice, and trust. Journal of Management Development, 34(10), 1272–1287. https://doi.org/10.1108/JMD-02-2015-0032
Amo, E., & Jareño, F. (2011). Self, peer and teacher assessment as active learning methods. Research Journal of International Studies, 18, 41–47.
Brookhart, S. M., & Chen, F. (2015). The quality and effectiveness of descriptive rubrics. Educational Review, 67(3), 343–368. https://doi.org/10.1080/00131911.2014.929565
Cao, H., Lee, C.-J., Igbai, S., Czerwinski, M., Wong, P. N. Y., Rintel, S., Hecht, B., Teevan, J., & Yang, L. (2021). Large scale analysis of multitasking behavior during remote meetings. In Y. Kitamura, A. Quigley, P. Bjørn, & S. Drucker (Eds.), Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (pp. 1-13). Association for Computing Machinery. https://doi.org/10.1145/3411764.3445243
Chen, L., Feng, G., Joe, J., Leong, C. W., Kitchen, C., & Lee, C. H. (2014). Towards automated assessment of public speaking skills using multimodal cues. In A. A. Salah, J. Cohn, & B. Schuller (Eds.), Proceedings of the 16th International Conference on Multimodal Interaction (ICMI '14), (pp. 200–203). Association for Computing Machinery https://doi.org/10.1145/2663204.2663265
Cheng, W., & Warren, M. (2016). Peer assessment of language proficiency. Language Testing, 22(1), 93–121. https://doi.org/10.1191/0265532205lt298oa
Dalgleish, T., & Power, M. J. (Eds.). (1999). Handbook of cognition and emotion. Wiley. https://doi.org/10.1002/0470013494
De Grez, L., Valcke, M., & Berings, D. (2010). Student response system and learning oral presentation skills. Procedia - Social and Behavioral Sciences, 2(2), 1786–1789. https://doi.org/10.1016/j.sbspro.2010.03.985
De Grez, L., Valcke, M., & Roozen, I. (2012). How effective are self- and peer assessment of oral presentation skills compared with teachers’ assessments? Active Learning in Higher Education, 13(2), 129–142. https://doi.org/10.1177/1469787412441284
Friman, P. C. (2014). Behavior analysts to the front! A 15-step tutorial on public speaking. The Behavior Analyst, 37(2), 109–118. https://doi.org/10.1007/s40614-014-0009-y
Herrera-Pavo, M. A. (2021). Collaborative learning for virtual higher education. Learning, Culture and Social Interaction, 28, Article 100437. https://doi.org/10.1016/j.lcsi.2020.100437
Hofmann, J., Platt, T., Lau, C., & Torres-Marín, J. (2020). Gender differences in humor-related traits, humor appreciation, production, comprehension, (neural) responses, use, and correlates: A systematic review. Current Psychology. Advance online publication. https://doi.org/10.1007/s12144-020-00724-1
Iberri-Shea, G., & King Fai Hui, S. (2017). Adaptation and assessment of a public speaking rating scale. Cogent Education, 4(1), Article 1287390. https://doi.org/10.1080/2331186X.2017.1287390
Joe, J., Kitchen, C., Chen, L., & Feng, G. (2015). A prototype public speaking skills assessment: an evaluation of human-scoring quality. ETS Research Report Series, 2015(2), 1–21. https://doi.org/10.1002/ets2.12083
Kolber, B. J. (2011). Extended problem-based learning improves scientific communication in senior biology students. Journal of College Science Teaching, 41(1), 32–39. https://bit.ly/3ZwoumL
Loureiro, M., Loureiro, N., & Silva, R. (2020). Differences of gender in oral and written communication apprehension of university students. Education Sciences, 10(12), Article 379. https://doi.org/10.3390/educsci10120379
McCarthy, J. (2017). Enhancing feedback in higher education: Students’ attitudes towards online and in-class formative assessment feedback models. Active Learning in Higher Education, 18(2), 127–141. https://doi.org/10.1177/1469787417707615
Mejía, K., & Baena, V. (2020). MOOC and professional skills development: enhancing public speaking competence with international teams through Google Classroom. In R. Hernández Rizzardini, E. Tovar, & M. Castro (Eds.), 2020 IEEE Learning With MOOCS, (pp. 73–76). Institute of Electrical and Electronics Engineers. https://doi.org/10.1109/LWMOOCS50143.2020.9234330
Mitchell, V.-W., & Bakewell, C. (1995). Learning without doing. Management Learning, 26(3), 353–366. https://doi.org/10.1177/1350507695263005
Oeppen, R. S., Rutherford, E., Sadler, P., Isaac, R., & Brennan, P. A. (2020). Virtual ARCP assessment and trainee feedback meetings: facilitating the best experience and practice. British Journal of Oral and Maxillofacial Surgery, 58(10), 1240–1244. https://doi.org/10.1016/j.bjoms.2020.09.037
Plug, I., Stommel, W., Lucassen, P, olde Hartman, T., van Dulmen, S., & Das, E. (2021). Do women and men use language differently in spoken face-to-face interaction? A scoping review. Review of Communication Research, 9, 43-79. https://doi.org/10.12840/ISSN.2255-4165.026
Schreiber, L. M., Paul, G. D., & Shibley, L. R. (2012). The development and test of the Public Speaking Competence Rubric. Communication Education, 61(3), 205–233. https://doi.org/10.1080/03634523.2012.670709
Suzuki, W. A., Feliú-Mójer, M. I., Hasson, U., Yehuda, R., & Zarate, J. M. (2018). Dialogues: The science and power of storytelling. The Journal of Neuroscience, 38(44), 9468–9470. https://doi.org/10.1523/JNEUROSCI.1942-18.2018
Tekian, A., Watling, C. J., Roberts, T. E., Steinert, Y., & Norcini, J. (2017). Qualitative and quantitative feedback in the context of competency-based education. Medical Teacher, 39(12), 1245–1249. https://doi.org/10.1080/0142159X.2017.1372564
Toastmasters International. (n.d.). All about Toastmasters. https://www.toastmasters.org/about
van Ginkel, S., Gulikers, J., Biemans, H., & Mulder, M. (2015). Towards a set of design principles for developing oral presentation competence: A synthesis of research in higher education. Educational Research Review, 14, 62–80. https://doi.org/10.1016/j.edurev.2015.02.002
van Ginkel, S., Ruiz, D., Mononen, A., Karaman, C., de Keijzer, A., & Sitthiworachart, J. (2020). The impact of computer-mediated immediate feedback on developing oral presentation skills: An exploratory study in virtual reality. Journal of Computer Assisted Learning, 36(3), 412–422. https://doi.org/10.1111/jcal.12424
Yang, Y. T. C., Chen, Y. C., & Hung, H. T. (2022). Digital storytelling as an interdisciplinary project to improve students’ English speaking and creative thinking. Computer Assisted Language Learning, 4(35), 840–862. https://doi.org/10.1080/09588221.2020.1750431