Summative assessment of students' writing predicts not only the extent to which the course learning objectives have been achieved but also reveals the relevance of the assessment design with the construct of writing being assessed. Any dichotomy between the assessment criteria and the construct of writing or between the assessment criteria and test scoring procedures can produce unreliable and invalid interpretations of the students' writing proficiency. Assuming cohesion as a measure of writing quality, the present study chose samples of academic writing which did not specify cohesion as a descriptor in the assessment scale. A cohesion index was, therefore, developed to investigate how cohesive devices created texture in the sample texts and correlated with the test scores. Descriptive statistics and non-parametric correlation analysis revealed that cohesive devices were positively associated with appropriate use to create texture; however, they only weakly correlated with the test scores. The findings imply that there is the need for developing assessment criteria which consistently measures the text-forming resources to reliably ascertain the writing proficiency of the students. The study recommends a research initiative based on an analytical assessment criteria to ensure a more accurate analysis of the role of cohesion in text-formation and writing quality.