Nonverbal behavior is essential in human interaction (Gullberg, de Bot, & Volterra, 2008; McNeill, 1992, 2005). For second language speakers, nonverbal features can be helpful for successful and efficient communication (e.g., Dahl & Ludvigsen, 2014). However, due to the complexity of nonverbal features, language testing institutions have conventionally employed an asynchronous format (e.g., audio-only mode), focusing on the test-taker's verbal features. Recently, the advance in technology, and most importantly, the coronavirus pandemic (COVID-19) outbreak in late 2019 and early 2020, have accelerated the use of video-conferencing applications in educational contexts, including speaking tests (Nakatsuhara, Inoue, Berry, & Galaczi, 2021). Accordingly, the synchronous test delivery mode (video-conferencing), which presents test-takers' visual information, is a timely and necessary approach to addressing the impact of nonverbal features on L2 oral communication. In response to these issues, I investigated the comparability of different delivery modes of speaking proficiency assessment. This study aimed to understand the dimensionality of the measured speaking construct and the impact of test-takers' visual information on rater behavior. Two datasets were used to address the research goals: first dataset (i.e., dataset 1) included scores of 110 test-takers, assessed by eight trained raters; second dataset (i.e., dataset 2) included scores of 284 prospective international teaching assistants (ITAs), assessed by 12 professional and certified raters. I collected dataset 1, and English language learning and testing program at a Midwest University provided dataset 2. I used two quantitative techniques to analyze both datasets: confirmatory factor analysis (CFA) and Multifaceted Rasch model (MFRM) analysis. For dataset 1, I qualitatively analyzed the raters' verbal reports.Dataset 1 had an asynchronous format; all test-takers' performances were recorded for rating. Eight trained raters gave scores to the audio-recorded and video 1 mode (test-taker and test giver were displayed). Three weeks later, the raters assessed video 2 mode (only test-taker's visual information was displayed). Within one month, raters participated in a one-on-one semi-structured interview. Dataset 2 stems from an operational testing context. This dataset has only scores, as I borrowed the data from the operational English testing program at the university. The scores were first given by examiners in a synchronous format (Live mode) and later by raters in an asynchronous format (Recorded mode). CFA findings indicated the multi-dimensional aspect of the underlying construct of speaking for both datasets, but the high inter-correlations showed that these are associated. Findings of MFRM revealed that raters showed leniency when rating (a) video mode over audio-only mode (dataset 1) and (b) synchronous mode over asynchronous mode (dataset 2). Findings suggest that using the video-conferenced delivery mode may be beneficial. However, the degree of usefulness across video modes differed, and how the raters utilized test-takers' nonverbal behaviors (e.g., gaze) varied. Thus, I decided that further investigation is needed to sufficiently support the use of video-conferencing applications to complement the physical face-to-face delivery mode. Overall, future research is highly recommended regarding the standardization of scoring of nonverbal features about the types of video mode, which would assist with the practical and valid application of virtual speaking tests.
Read
