Before discussing the validity of the instrument its reliability needs to be discussed to see if “the data collection procedure” – here, the analysis of reference – “is consistent and accurate” (Seliger & Shohamy, 1989, 185). Inter- and intra-coder reliability was used: Krisztina Zsova has also participated in a course on discourse analysis and had previous experience in various methods of text analysis, including the instrument used here. The instructions she received included the coding scheme, the layout of the table (as in Table 14) to be filled in, a sample analysis and an outline of the instructions provided by Halliday and Hasan (1976, 328–358: see Appendices A and B) and of course, Abstracts 1–10. Further instructions were not given for two main reasons; first, the coder was already familiar with the taxonomy; second, because the taxonomy should be usable without a full theoretical background if it is to be used as a pedagogical tool as well. In order to check intra-coder reliability, the other coder was asked to do the analysis twice, with a three-week break between the two analyses; I, on the other hand, compared the results of an earlier analysis conducted approximately three years earlier. This difference in the time elapsed might be one of the reasons why in the case of the other coder the intra-coder reliability was higher; between her two analyses the time was shorter, thus she might have remembered the logic she had used previously.

 

 

The percentages in Figure 5 indicate that there are considerable differences between the results, which means that the instrument probably cannot be regarded as reliable and that in its present form it is not suitable for the analysis of longer texts.

 

Halliday and Hasan (1976) do not specify whether in this case only this cohort forms a cohesive tie, or they would establish a separate tie pointing to the same item via referring to this cohort first, and if it is the latter, how it should be coded. This results in inconsistencies in the analysis concerning both the categories and the number of cohesive ties. It is difficult to say anything conclusive about construct validity based on such a small sample; still, it shows whether the components coded are relevant to reference or whether there are any types that could be ambiguous. The identification of the most problematic items in the analysis should be very useful in this process; therefore, the analysis was complemented with a think-aloud-like process (in written form) including analytical problems and decisions. The following subsections discuss the extent to which Halliday and Hasan’s (1976) taxonomy can be used to draw conclusions from the results concerning the cohesion of the abstracts.