Reliability of the analytical process is partly ensured by the procedure itself. Having two approaches, hence two analyses to each text (a non-cohesive and a cohesive analysis, supplemented with a list of errors) it is easy to account for all the items in the text in one way or another. Still, it cannot be sure that everyone would categorize non-cohesive items or cohesive chains in the same way. While it is acknowledged that some errors are inherent in the analysis, based on previous reliability checks we assume that there should be at least a 90% overlap between two analyses of the same coder or between the results of the analyses of two coders. For this reliability analysis I asked an advanced-level English speaker friend, Judit Andruskó, to analyze an RA based on the guidelines I had sent her. For it to be a pedagogical tool besides being an analytical tool, Referential Cohesion Analysis should be such that it can be carried out by anyone with sufficient English language knowledge to understand the particular text. The analysis should be simple enough to be performed by students without extensive training. Judit received a five-page description of the instructions (see Appendix A) and was asked to report any difficulties or problems during the analysis. She reported that she had found the task quite enjoyable, though it took her a bit more than an hour to finish the analysis of 15–20 sentences, depending on the length of the sentences and the number of items in them. With some practice the analysis gets much easier, and it is actually possible to reach 50–60 sentences per hour (which still means that it takes about 12 hours to analyze an MA thesis, which is not really the efficiency one would dream of). She learned the method very quickly and her analysis was very helpful, and her feedback on the tool was informative.

 

All the items above (and the others not listed here) contain some academic vocabulary that may have led to uncertainty on the part of the coder and to the exclusion of the items from the analysis. What we need to consider here is how much of the academic vocabulary university students are familiar with and whether it causes problems in discourse production and processing.

 

 

The second part of Table 20 gives general information about the cohesive ties and chains found in the corpora. Percentages of cohesive and non-cohesive items out of the total number of items searched were counted, which produced rather unexpected results. Apparently, the highest ratio of cohesive items characterized low rated theses, and not RAs, with high-rated theses in between these results. While this phenomenon is very interesting, there might be a wide range of possible reasons behind it. It might be hypothesized that a higher ratio of cohesive items is related to text length, but it could also be proficiency-related or even accidental. What suggests that it is more of an expert-novice difference is that a comparison between the specific types of cohesive referring items shows systematic differences between item frequencies. For example, the pronoun she as part of a cohesive tie appears on average once in RAs, 7.9 times in high-rated and 21.7 times in low-rated papers; in addition, all the other pronouns display a similar frequency pattern.