In this study I compare 81 students’ performances on and perceptions of two different writing exams: an impromptu timed-writing (TW) exam and a process-based timed-writing (PBTW) exam. The students had 45 minutes to write an essay for the impromptu TW exam. For the PBTW exam, the same participants read an article and watched two short videos about a topic, discussed the topic in small groups, and planned their essays. This took approximately 45 minutes. After that, they had 45 minutes to write their essays. Thus, the PBTW exam lasted 90 minutes. After taking both exams, the participants answered a short post-writing questionnaire about their perceptions of the two exams. Eighteen participants were randomly selected or volunteered to participate in a semi-structured interview in groups to receive more detailed information about their opinions regarding the exams. My secondary aim was to investigate what raters think of the two exams. Two raters scored the essays using an analytic rubric and participated in a semi-structured interview after they scored the essays. Furthermore, the raters participated in two training and norming sessions before they began rating the essays, both of which were audio-recorded to gather more information about the raters’ perceptions of the exams. To explore the results, I correlated the scores that the students received in the exams in SPSS and performed a t-test to determine whether there were significant differences between the scores. I also examined the essays to investigate accuracy, lexical and syntactic complexity, and fluency. I investigated the students’ perceptions of the exams by analyzing their answers in the post-writing questionnaire and the transcripts of the semi-structured interviews. Finally, I analyzed the training and norming sessions, as well as the semi-structured interviews with the raters to explore their perceptions of the exams. The results of the study revealed that, although the learners’ performance did not significantly differ in the two exams, the participants expressed a clear preference for the PBTW exam because they had time to learn about the topic through the article, videos, and discussion, they had time to plan their writing, and they could use the ideas in the source materials to support their opinions in the essays. The scores that the test takers received for content and punctuation were significantly higher in the PBTW exam, while the scores that they received for spelling were significantly higher in the TW exam. The participants also wrote significantly longer essays and significantly more words per minute in the PBTW exam. In addition, they used more sophisticated vocabulary and a wider variety of nouns in the PBTW exam. The students’ essays did not vary in terms of syntactic complexity or grammatical accuracy, however. The scores that students received in the two exams correlated only moderately (.391), which suggests that the two exams measure different constructs, with the PBTW exam measuring reading, listening, and source integration, among other skills, in addition to writing. Both the learners and the raters mentioned that the learners had difficulties integrating sources, which could be a result of the negative washback of TW exams in their classes. The inter-rater reliability coefficient for all of the exams was high, but the inter and intra-reliability coefficients for the TW exam were higher. The results of this study, combined with the results of other similar studies and the skills that L2 learners need to succeed in college-level classes suggest that the PBTW exam may be a better tool to evaluate the construct of academic writing.
Read
