Research-Oriented Framework of Training Philology Students ’ Research Skills Based on Corpus Analytical Software

The purpose of the study was to explore how technological advances incorporated into the Philology Studies curriculum could impact the students’ research skills and the quality of their research projects and what students’ and teachers’ impressions of the reshaped research component of the curriculum were. The study used qualitative and quantitative methods with the dominance of qualitative methods. It employed the baseline study, checklist to assess students’ research papers, assessment criteria, and the Triangular Assessment Method to assess the students’ papers. The consensus meeting was held to allow the experts to express their reasoning for the scores. The semi-structured interview was administered to the students’ and teachers’ to identify their impressions of the reshaped research component of the curriculum of philology. The technological advances incorporated into Philology Studies curriculum improve the students’ research skills and the quality of their research projects. Both students and teachers appreciated the reshaped research component of the curriculum. The analytical software can be successfully incorporated in the corpus analysispurpose student research. The students found the intervention a challenging experience that ‘pumped up’ their intellectual, research, and technical skills. They reported improvement in interpreting corpus using correlations, frequencies, distributions, and collecting information using software to organise it in a professional way. The lecturers agreed that the technology-based instructional model incorporated into Philology Studies curriculum improved both students’ research skills and the quality of their research projects.


Introduction
A research-led, research-oriented, research-based, and research-tutored approaches are quite commonly used as a framework for both teaching and curriculum design of pre-service training in higher education institutions (Haaker & Morgan-Brett, 2017;McLinden et al., 2015). The research is a significant component of the pre-service training of students majoring in philology and relies on the research methods employed in Linguistics (Fulk, 2016;Kiliańska-Przybyło, 2019;Litosseliti, 2018). These methods include qualitative ones such as an interview, discourse analytic approaches, and multimodal analysis which dominate over the mixed research methods such as a corpus analysis. This situation raises several issues such as the validity and precision of the students' and university scholars' findings in the field of philology and linguistics, research bias occurrence, and limitedness of the use of the software in processing and analysing the raw data. These created the research methodology-related and research quality gap to address through using the research-oriented approach and free corpus analysis software to train tertiary Philology students. The research-oriented approach is suitable for the study because it is intended to develop students' awareness and ability to utilise the appropriate research methodologies and methods within the context of a certain discipline, specifically linguistics and philology (Reiber, 2019).

Literature review
The literature review found that research-oriented instruction is quite commonly used to train tertiary students (Böttcher & Thiel, 2018;Shabbir & Abbas, 2019;Trapero & Stypka, 2018;Vauderwange et al., 2019). Vauderwange et al. (2019) opines that the above trend is driven by the fact that the research process is explained to them. The researchoriented teaching relies on inquiry-based learning (IBL) concept that is considered an indispensable component of higher education (Hofer & Lembens, 2019).
This instructional approach brings a number of benefits for both students and teachers because it accumulates and updates information/content, involves reliable and valid data to evaluate that information, and complies with standards of peer review (Shang & Liu, 2015).
The research-oriented teaching was found to engage the students in the various activities helping students to acquire the essential practical skills for dealing with situations requiring them to develop practical solutions to complex problems using their academic knowledge and reasoning (Freie Universität Berlin, 2020; Organisation for Economic Co-Operation and Development/Centre for Educational Research and Innovation [OECD/CERI], 2008). It was found that the research-oriented approach is seen as a multidisciplinary unique educational experience that engages students, teachers, and communities in creating common value (Trapero & Stypka, 2018). This approach was also found to align with application-oriented practice teaching that cultivates students' vocational ability and character, increases students' self-efficacy and motivation (Chen, 2017;Gupta, 2017). Harris and Jaen (2010) and Heine and Narrog (2015) revealed the insights of the integration of data-driven learning in language teaching along with different types of corpora such as pedagogical, spoken multimedia, and parallel. The authors stated that research still lacks analytics and statistics. According to Hofer and Lembens (2019), the researchoriented teaching approach needs reshaping to be more consistent with curricula, standards and final exams. Though found significant and effective, the teachers still use it rarely in their classes (DiBiase & McDonald, 2015).
Though the research-oriented approach is represented in the studies, its technological component, in particular in linguistic and philological research, necessitates investigation which created the gap for the study.
The purpose of the study was 1) to explore how technological advances incorporated into Philology Studies curriculum could effect the students' research skills and the quality of their research projects; 2) what students' and teachers' impressions of the reshaped research component of the curriculum were.

Methodology
The study used qualitative and quantitative methods with the dominance of qualitative methods (Ahmad et al., 2019). It employed the baseline study questionnaire (see Appendix A) as a primary research quantitative method to collect information from 287 randomly sampled students majoring in Philology at four universities in Ukraine. The intervention part of the research relies on the qualitative data obtained through the assessment of the students' research papers by the students and a panel of 7 experts. The students performed an assessment using the checklist to assess students' research papers (see Appendix B), and the experts employed criteria and the Triangular Assessment Method (TAM) to assess the papers (Pérez- Rodríguez & Rojo-Alboreca, 2017). The consensus meeting was held to allow the experts to express their reasoning for the scores.
The feedback part of the research also used a qualitative method such as a semi-structured interview to identify the students' and teachers' impressions of the reshaped research component of the curriculum of philology.

Research design
This was the only group study that combined exploratory and experimental research designs and was organised to be a series of steps such as a baseline study, the intervention, feedback, and data analysis. The baseline study part of the study was explorative and employed primary and secondary research methods to collect information. The objective of this phase of the study was to identify how the students majoring in Philology self-assessed their research skills, research methods awareness, and research software awareness used specifically in philology and linguistic research. This phase also involved literature review, sampling, identification of the research relevance and feasibility, research plan development.
The intervention was supposed to involve the sampled students in the carrying out the language research projects with the application of free or as trials of the corpus analytical tools such as the Voyant Tools (accessible at https://voyanttools.org/), the Linguistic Inquiry and Word Count software (accessible via the link of http://liwc.wpengine.com/), British National Corpus (BNC) web (CQP-Edition) (can be accessed via http://corpora.lancs.ac.uk/BNCweb/) (Harvey, 2010;Hoffmann et al., 2008) and Raven's Eye (can be accessed through the link https://ravens-eye.net/). The intervention lasted one year and four months, from September 2019 to the end of October 2020. It was organised to be holistic and split into modules such as training in the application of the software, essentials of the research methodology and methods in corpus analysis, data processing, and presentation and academic language. The students were organised as project teams of three people each and were supposed to spend 6 months to complete the research on one of the topics below. The students' research papers were assessed through the double-blind peer review and double-blind expert review. The review procedures were based on the checklist for students (see Appendix B) and the TAM for the experts.
The topics for the research papers were as follows: 1. Monomials as a set of term clusters in the terminology of Agricultural or Veterinary Medicine-related fields of English (German/French) language corpus 2. Matrix profiling of the semantics of Agricultural or Veterinary Medicine -related terminology of English (German/French) language corpus 3. Methodological Corpora Toolkit and its applications in Modelling of the Cognitive and Semantic Matrices for Agricultural or Veterinary Medicine-related terminology of English (German/French) language corpus 4. Componential analysis of Agricultural or Veterinary Medicine-related terminology of English language corpus 5. Linguistic analysis of the connotative meaning of words and comparisons as aspects of verbal reflection of national specificity The corpus analytical software provided the key tools and variables for the students' research projects.
The feedback stage was intended to identify the students' and teachers' impressions of the reshaped research component of the curriculum of philology. The data analysis part of the research focused on data consolidation, processing, and presentation (See Figure 1).

Figure 1. Research procedure Sampling
A simple random sampling technique was utilised in the baseline study online survey to hire 287 individuals selected from a total of 617 students of the target population majoring in Philology. The participants for the survey were hired from four universities in Ukraine such as the National University of Life and Environmental Sciences of Ukraine, Borys Grinchenko Kyiv University, National Pedagogical Dragomanov University, and National University of Kyiv Mohyla Academy.
The convenience sample technique was employed to select 15 students (10 males and 5 females, aged 21-22) from the National University of Life and Environmental Sciences of Ukraine to form the experimental group (EG). The sampled students were found mutually homogeneous as they sought the degree in the same major and studied in the same academic group. As can be seen in Table 1, the t-values varied between 3.5311 (SD=0.2198; p=0.02178) and 5.2187 (SD=0.1165; p=0.03312). The Alpha coefficient for the internal consistency calculated using Cronbach's alpha was .83, which was good according to Taber (2018). The inter-correlation values were between .371 and .452 which implied that the construct validity of the questionnaire was significant at p<.01. The content validity values drawn from the procedure of its validation by the research team members and hired experts using a 4-point rating scale were between 3.31 and 3.65, which suggested that the content of the questionnaire was relevant and appropriate. The interrater reliability of the questionnaire based on the computation of the Intraclass Correlation Coefficient (ICC), provided the mean value of 0.51 (95%; CI=0.46-0.56) which was sufficient.
Checklist to assess students' research papers by students (Appendix B): It was used for assessment of the relevance and appropriateness of the use of the software for corpus analysis and overall quality of the students' research papers by students using the double-blind review procedure. It is based on a Likert 4-point scale with 1 meaning 'not at all', 2 = 'slightly', 3 = 'somewhat', 4 = 'completely'. When validating, the item-level content validity index (IL-CVI) was suggestive as the values were higher than 0.81. The values for the Kappa coefficient were higher than 0.84. Both above meant a 'good validity' of the checklist.
Criteria & the Triangular Assessment Method (TAM): These were used by the 7 scholars to assess students' papers as a multi-criteria decision-making method to elicit unbiased experts' judgements on the relevance and appropriateness of the use of the software for corpus analysis and overall quality of the students' research papers. The criteria of the research paper assessment were grouped as follows: goal setting and methodology (specification of the research problem, scope; use of an appropriate research methodology to address the research questions; a discerning selection of linguistic cases); analysis and structure (holistic and relevant corpus analysis to address the research aims; clear and precise argumentation; relevant and illustrative examples); presentation (formatting, compliance with research ethics).
Further are the descriptors of the scores: 'high distinction' (ECTS -80-100) -the paper is original; provides reliable analytic insights; addresses the research goals through in-depth review of the literature, analysis of the texts and interpretation of findings; the paper is well structured and well written; 'good' (ECTS -70-79) -the paper corresponds the most of requirements, but is limited in research insights or intellectual evaluation of the findings; 'credit' (ECTS -60-69) -the paper, though providing analytics and interpretations, contains misunderstandings in the analysis and provides limited insights; it needs revision of structure and writing; 'pass' (ECTS -50-59) -the paper uses methodology limitedly, the sources are limitedly analysed; the structure seems defective; it contains misleading formulations and/or distracting errors in written expression; 'fail' (ECTS -below 50%) -the methodology is used inadequately, the analyses and interpretations seem limited or unsound or irrelevant to the research questions, and the findings are presented unscholarly. The scholars' final judgements drawn from the application of the criteria were then confirmed by TAM using a 5-point scale with values that tended to be closer to 1 meaning 'completely confident' to 5 'not confident at all'.
The experts' consensus meeting (n=7). This was held online via the Google Meet video conferencing app, was recorded and transcribed. The texualised data were processed manually. Its purpose was to allow the experts to express their reasoning for the scores and clarify their judgements.
A semi-structured interview (n=9). The interview used 5 open-ended questions to interview randomly hired 6 students and 3 lecturers (scientific advisors for the student teams). The interviews were conducted via the Google Meet video conferencing app. They were recorded and responses were transcribed. The latter were categorised by themes. The categories were used by the experts as codes.
1. What is your overall impression of the reshaped research component of the curriculum of philology? Why?
European Journal of Educational Research 675 2. What benefits did the intervention bring to you personally? Suggest your reasoning.
3. What caused you to feel disappointed or discouraged in the run of the intervention? Why?
4. Would you recommend the use of the analytic software to your peers when doing the research projects in linguistics and philology? Suggest your reasoning? 5. What do you see as a disadvantage of the intervention and should be coped with to make it more useful for the students?

Results
The baseline study found that research skills, research methods awareness, and research software awareness were not up to standard. It was also found that the instructional model based on the analytical software incorporated into Philology Studies curriculum positively influenced the students' research skills and the quality of their research projects, in general.

Baseline study results
The purpose of the baseline study survey was to identify how the students majoring in Philology self-assessed their research skills, research methods awareness, and research software awareness used specifically in philology and linguistic research. The mean values for the scales were quite low indicating that students were doubtful about their research potencies. The values for standard deviation and variance indicates that the respondents' answers were dispersed which suggested that the intervention model was feasible. Table 1 presents the results of a baseline study as descriptive statistics.

Results of the assessment of the students' research papers using the checklist (Appendix B)
Five research papers were assessed through the double-blind peer review by sampled students for the relevance and appropriateness of the use of the software for corpus analysis and the overall quality of the students' research papers using the checklist. As can be seen in Table 2, the students' scores were higher enough (more than 70% of ECTS) which indicated that the students were biased in their judgements.

The results of the assessment of students' research papers by 7 scholars based on criteria and TAM
The scholars were suggested two instruments to assess the students' research papers to achieve maximal objectiveness. The scholars' scores drawn from the assessment based on the criteria showed approximate unanimity in their judgements. Surprisingly, these scores correlated with the students' scores in assessments of all papers except RP 3 -the scholars' scores appeared to be dispersed (Figure 2). Even more surprising was the fact that TAM-based judgements were almost absolutely unanimous and confident only on RP 3 that scored the highest scores and met the outlined criteria. Furthermore, the scholars' judgements on the RP 5 demonstrated confidence and 'good' grades ( Figure 3).

. The results of the assessment of students' research papers by 7 scholars based on TAM
The consensus meeting that was held to allow the experts to express their reasoning for the scores clarified the above results. The experts reported the originality of papers that provided reliable analytic insights based on both in-depth reviews of the literature and findings. They also emphasised the high quality of methodology of yielding data and analysis of the texts and interpretation of findings which proved that the use of analytical software by the students majoring in Philology Studies improved the students' research skills and the quality of their research projects.

A semi-structured interview (n=9)
Question 1. The students associated the project with challenging experiences, 'pumping up' their intellectual, research, and technical skills. They reported improvement in interpreting corpus using correlations, frequencies, distributions, and collecting information using software to organise it in a professional way. The lecturers agreed that the technologybased instructional model incorporated into Philology Studies curriculum improved both students' research skills and the quality of their research projects. The model contributed to students' awareness of the linguistic research methodology. Some of the common quotes were as follows: [… the researches were unbiased and the results were not tentative…]

[… time efficiency was really noteworthy …]
[… students were enthusiastic and there were a lot of inquiries and initiatives from them…] Question 2. Four students responded that they upgraded their computer and research skills. Two students gained more confidence both as a learner and as a researcher. The lecturers reported that they updated their instruction skills and research software awareness. Their reasons were as follows: […because the instructional model raised standards of fulfilling research in linguistics and philology and it created competitive, transparent, and unbiased environments with equal opportunities for everyone…] Question 3. Students confessed that the criticisms from peers and experts were the greatest discouragements and disappointments. The lecturers confessed that they were often discouraged by the students' reactions to their feedback.

Students' research papers
[… some students did as if they were professors…] Question 4. Both students and lecturers reported that they would advise using analytic software to carry out the research in linguistics and philology as the software provides a more reliable and significant result and the use of it allows to comply with international standards in this field.
Question 5. Both students and lecturers emphasised that the way of providing feedback could be a problem that needed a solution. Both students and lecturers needed training in giving and receiving feedback either orally or in writing.

Discussions
The strength of the study is in the transformation of the research component of the curriculum of Philology Studies through the incorporation of analytical software in the corpus analysis-purpose student research. The results obtained from the baseline survey identified that the students majoring in Philology were doubtful about their research potencies and this was supported by the mean values for the scales that were quite low, between 2.1207 and 3.3638 with a maximal expected value of 5.00. The values for standard deviation and variance indicates that the respondents' answers were dispersed which suggested that the intervention model was feasible. It was found that research papers that were assessed through the double-blind peer review by sampled students for the relevance and appropriateness of the use of the software for corpus analysis and the overall quality of the students' research papers using the checklist were high enough (more than 70% of ECTS). This could indicate that the students were biased in their judgements. The scholars' scores drawn from the assessment based on the criteria showed approximate unanimity in their judgements. Surprisingly, these scores correlated with the students' scores in assessments of all papers except RP 3 -the scholars' scores appeared to be dispersed. The findings form this phase are consistent with Klomsri and Tedre (2016) who found that surveyed students performed poorly in searching and processing information which was connected to ineffective information literacy training as knowledge and skills were imparted.
The results of the intervention go in line with Rodríguez et al. (2019) who found that inquiry-based learning -in our case, it is a research-oriented approach -develops the research and creative thinking skills. Additionally, it generally improves the students' academic results. These are consistent with Missingham et al. (2018) who proved that fulfilling real-case-based projects promotes students' metacognition capacities which are crucial in both research and learning.
The TAM-based judgements were almost absolutely unanimous and confident only on RP 3 that scored the highest scores and met the outlined criteria. Furthermore, the scholars' judgements on the RP 5 demonstrated confidence and 'good' grades. The consensus meeting found that the papers were original and provided reliable analytic insights based on both in-depth reviews of the literature and findings. They also emphasised the high quality of methodology of yielding data and analysis of the texts and interpretation of findings which proved that the use of analytical software by the students majoring in Philology Studies improved the students' research skills and the quality of their research projects. In the interview, the students responded that the intervention was a challenging experience for them and due to it, they 'pumped up' their intellectual, research, and technical skills. They reported improvement in interpreting corpus using correlations, frequencies, distributions, and collecting information using software to organise it in a professional way. The lecturers agreed that the technology-based instructional model incorporated into Philology Studies curriculum improved both students' research skills and the quality of their research projects. The model contributed to students' awareness of the linguistic research methodology. Both students and lecturers benefited from the intervention as they upgraded their computer and research skills, gained more confidence both as a learner and as a researcher, and updated their instruction skills and research software awareness. The results agree with Mahani and Molki (2012) who found that research is used as a method of intellectual inquiry in universities to improve or test/evaluate or reshape or update/upgrade the teaching. They also emphasis that the research helps students increase their academics potential. The findings align with Giraud and Saulpic (2019) who showed how the reshaped course to be a form of research benefited both students and teachers as it produced new knowledge and upgraded instruction.
The findings drawn from the baseline study and the intervention contributed to the previous research. These go with Litosseliti (2018) who revealed the research methods used in linguistics and philology from the perspective of the principles, key disputes, and contributions these methods make to the linguistic/philology research. The study aligns with Mickan (2017) who claims that text-based research, despite there is a lot of linguistic research, is often misunderstood and lacks reliable interpretation of the results.

Conclusion
The technological advances incorporated into Philology Studies curriculum improve the students' research skills and the quality of their research projects. Both students and teachers appreciated the reshaped research component of the curriculum. The analytical software can be successfully incorporated in the corpus analysis-purpose student research. The students found the intervention a challenging experience that 'pumped up' their intellectual, research, and technical skills. They reported improvement in interpreting corpus using correlations, frequencies, distributions, and collecting information using software to organise it in a professional way. The lecturers agreed that the technologybased instructional model incorporated into Philology Studies curriculum improved both students' research skills and the quality of their research projects. The model contributed to students' awareness of the linguistic research methodology. Both students and lecturers benefited from the intervention as they upgraded their computer and research skills, gained more confidence both as a learner and as a researcher, and updated their instruction skills and research software awareness. Further research in the use of the analytic software in student research in the field of Philology and Linguistics is feasible.

Recommendations
The researchers should study how the feedback could impact the development of the students' research skills and the quality of their research projects. They should investigate how the training in giving feedback could influence the student research, in general. The instructors should select the topics for student research projects considering their applied value such as marketing, psychology, medicine, or others.

Limitations
Three main limitations can be considered in the study. First, the number of sampled students, which is purposefully formed to be manageable can be viewed as a limitation. Second, one institution involved in the experiment can also be a limitation. Thirdly, qualitative data leaves the possibility of unidentified confounding variables and can be regarded to be a limitation, as well.