To assess students’ source language skills, relevant parts of a language test based on Cambridge Proficiency Tests were used (Harrison, 2001). The sub-tests that students had to fill in were a Reading test and a Use of English (grammar test). The Reading test consisted of two multiple-choice tasks and a paragraph gap-filling task. The Use of English test included two gap-filling exercises and one sentence transformation task (the tasks can be found in Appendix 2).

Although it is obvious that target language skills are as important as source language skills, the lack of standardised Hungarian language tests for adult native speakers prevented us from assessing L1 (A language) competence. In 2022, four research group members started to work on devising a Hungarian language test. The test is in the pilot phase, and in its present form, it includes a gap-filling task, a synonym identification task and a re-writing task.

Extra-linguistic competence (also referred to as domain knowledge or thematic knowledge) was assessed by multiple-choice tests. As two different texts were used in the project, two tests had to be prepared. The Legal background knowledge test focused on copyright law and was compiled by a lawyer at Szeged University specialising in copyright law and holding a PhD degree in law. The Legal test is not relevant for the present paper; accordingly, it is not attached in the Appendix. The Bilingualism background knowledge test was devised by the author, who has taught courses on bilingualism and published a study on it. The Bilingualism test contains three items on pragmatics, too, as it was a key topic in the source text (see Appendix 3 for both the Hungarian and English versions). Nevertheless, the test will be referred to as the “Bilingualism test”. Both the Copyright and the Bilingualism tests are in Hungarian, and they include 10 multiple-choice test items.

To evaluate students’ knowledge of translation, a questionnaire was developed based on the PACTE group’s 2008 questionnaire (Pacte Group, 2008b). The PACTE questionnaire consisted of 27 items and aimed at measuring the declarative knowledge of translation. After translating the questionnaire into Hungarian and piloting it in a Hungarian context, it became clear that the instrument needed major restructuring. In consequence, a new questionnaire was developed by the Szeged research group, the process of which is reported elsewhere (M. Lesznyák, Sermann, et al., 2023; M. Lesznyák et al., 2025b). The new questionnaire bears the name “Beliefs about translation” as both the original PACTE and the new Szeged questionnaire contain items that can be considered more beliefs than knowledge. It assesses how closely students’ beliefs, principles, and translation-related mindset align with current professional views. The 18-item instrument is in Hungarian and cover five areas of beliefs about translation: the concept of translation and translation competence, the identification and solution of translation problems, translation procedures and methods, the translation context/setting, and post-editing of machine translation (M. Lesznyák et al., 2025b) (see Appendix 4 for both the original Hungarian version and its English translation).

Students’ translation and post-editing competencies were assessed by a translation and a post-editing task. Two texts, a legal text (part of a copyright agreement, 350 words, not attached) and an academic text (an abstract of a study on bilingualism, 127 words, see Appendix 5), were chosen for the research. There were two main reasons for the choice of texts: on the one hand, familiarity with the topic of the text was one factor that we wanted to vary (law vs. linguistics). As 94% of the participants earned their first degree in modern languages (see Table 1), it was assumed that the abstract of an academic paper on linguistics would be familiar to them both in terms of content and genre. In contrast, they had no experience with legal texts and had not studied law previously; as a result, we hypothesised that the translation of a legal text would present a major difficulty for them.

On the other hand, we wanted to select source texts for which the MT output clearly needed post-editing. Therefore, we decided to choose texts from the pool we had used for teaching before the experiment, as we were familiar with those texts. This familiarity probably explains why the first two texts we selected fulfilled the criterion of the need for post-editing. This was also supported by expert opinions (see next paragraph).

The MT output was generated with DeepL for the bilingualism text (see Appendix 6) and with eTranslate for the copyright agreement (not attached). At the time of the data collection, both engines used NMT.

To ensure that the source texts fulfilled the criteria of (a) being different in terms of difficulty, (b) requiring post-editing, we asked 12 experts (translation trainers) to assess the difficulty of the two selected texts for (a) translation and (b) post-editing (M. Lesznyák et al., 2025a). Difficulty was assessed on a five-point scale, 5 indicating “very difficult” and 1 representing “very easy”. The experts were also invited to share their comments on the texts. The responses indicated that the legal text was seen as considerably more challenging both to translate and post-edit than the academic abstract (see Table 4).

 

These comments, together with numeric evaluations, indicate that the academic abstract and the copyright agreement were appropriate texts for the investigation.

Time on task, research behaviour and other strategic behaviour were studied with the help of screen recording and keylogging (see details about data collection below).

The quality of the translations was evaluated by an MQM-based error category system developed by the Szeged Translation and Post-Editing Research Group (for the original MQM, see Mariana et al., 2015). MQM has the advantage that it can be tailored to the needs of users. Thus, the Szeged research group decided to include the error categories Accuracy, Fluency, Style and Terminology (for sub-categories see Table 5) in the project. Accuracy errors occur “when the target text does not accurately correspond to the propositional content of the source text” (https://themqm.org/error-types-2/typology/). The name „Fluency” has been changed to „Linguistic conventions” recently, but in this paper, we will stick to the term we started to use at the beginning of the investigation. Fluency errors violate the language rules of the target text and include both grammar and spelling errors. Style errors refer to instances when grammar is correct, but the formulation of a linguistic element is inappropriate. Terminology errors are defined as “errors arising when a term does not conform to normative domain or organisational terminology standards or when a term in the target text is not the correct, normative equivalent of the corresponding term in the source text” (https://themqm.org/error-types-2/typology/).

At the beginning of the research, it was decided that the same terminology error would be counted only once. This decision can be justified by claiming that the „error of choice” is only committed once, and it would be relatively easy to correct (with a global “change” command) in a real-life setting. Nevertheless, this decision had far-reaching consequences for terminology error numbers, as they are relatively underrepresented in the target texts.

The idea of weighing errors was also dismissed, because it could have introduced one more criterion, on which raters could have disagreed and because of the researchers’ experiences that weighing may change raw error numbers but will not change the rank order of students. The lack of error weights had no apparent effect on quality assessment.

 

To check inter-rater reliability, ICC (intraclass correlation) was computed for all error types. Because the aim was to evaluate agreement among the specific raters who participated in the study rather than to generalise to a larger population of raters, a two-way mixed-effects model with single measures and consistency definition was applied (Koo & Li, 2016). ICC values, 95% confidence intervals, F-statistics, and p-values for the longitudinal subsample are reported in Table 6, together with qualitative interpretations of reliability levels.

After checking inter-rater reliabilities, mean values of the raters’ error counts were calculated. These means were used in all subsequent analyses.

After finishing the translation or the post-editing task, students were asked to complete a follow-up questionnaire that collected data on students’ subjective experiences with the task. The questionnaire had an HT and a PE version (see Appendices 7 and 8 for the Hungarian versions of the questionnaires and their English translations), the PE version being longer with some extra, special questions. The items on the questionnaire focused on the perceived difficulty of the translation and the post-editing task. Both open-ended and closed (rating scale) questions were included in the instrument. The PE version had some extra questions focusing on the specific features of post-editing that may help or hinder the translators’ work.