THE ROLE OF NONVERBAL BEHAVIOR AND AFFECT ON RATINGS OF SECOND LANGUAGE PROFICIENCY

A long-standing problem in applied linguistics is how to account for nonverbal behavior in models of second language (L2) communicative ability (Canale, 1983; Canale & Swain, 1980; Celce-Murcia, 2007; Galaczi & Taylor, 2018; Hymes, 1972). Attempts have been made to incorporate some nonverbal behavior into some of these models, but they generally only account for strategic and interactional competences rather than the full range of information these behaviors can convey. It is well-established, however, that nonverbal behavior is fundamental to spoken, face-to-face communication (Hall et al., 2019; Hall & Knapp, 2013; Matsumoto et al., 2016), conveying semantic, cognitive, affective, and social-interactional information. Affect is one of the most important signals of nonverbal behavior, especially in facial movements (Knapp et al., 2013), conveying a range of emotive, orientational, and stance-related information. Nonetheless, language tests rarely account for this vital visual realm of information in their constructs or rating scales, despite research showing that it is meaningful to raters when formulating impressions of language proficiency (Choi, 2022; Ducasse & Brown, 2009; Jenkins & Parra, 2003; May, 2009, 2011; Nakatsuhara et al., 2021a; Nambiar & Goon, 1993; Neu, 1990; Orr, 2002; Sato & McNamara, 2019; Thompson, 2016). To date, few studies have observed measurable effects of nonverbal behavior or affect on impressions of language proficiency (Chong & Aryadoust, 2022; Kim et al., 2023; Nagle, 2022; Trofimovich et al., 2021; Tsunemoto et al., 2022). To address this research gap, I designed a research study to triangulate ratings of affect, measurements of nonverbal behavior, and cognitive interviews to determine the role of nonverbal behavior and its affective information when listeners formulate impressions of language proficiency when observing L2 speakers’ performances in a language test. I recruited 100 naïve, untrained raters to listen and watch short video recordings of 30 test takers interacting with an examiner during an oral proficiency interview in a high stakes speaking test. While listening and watching, the raters scored each speech sample on four categories of language—fluency, vocabulary, grammar, and comprehensibility—and ten categories of affect, covering dimensions of assuredness, involvement, and positivity. Following the rating activity, 20 of the raters took part in stimulated verbal recall sessions, which captured the raters’ thought processes while formulating their evaluations of L2 proficiency. In addition, I used automated, machine-learning software, iMotions, to extract measurements of nonverbal behavior in the speaking test samples in the form of engagement, attention, and valence. I also manually extracted speech and nonverbal behavior using multimodal annotations in ELAN. The study broadly found that nonverbal behaviors and affect can impact proficiency outcomes in different ways. Desirable, communicatively-oriented behaviors such as mutual gaze, nodding, leaning forward posture, and representational gestures can convey confidence, engagement, and positive affect, which lead to differential outcomes in fluency, vocabulary, grammar, and comprehensibility. Comprehensibility, for example, was most impacted by raters’ impressions of test taker engagement and also through behaviors that conveyed approachability, such as smiling and nodding. Fluency, vocabulary, and grammar were most impacted by impressions of confidence and low anxiety, as well as more target-like attentional focus. The raters were especially attuned to detecting listening comprehension, and the negative impact of comprehension breakdowns could be moderated when test takers took an adaptable stance that showed a desire to communicate. Overall, nonverbal behavior was found to affect perceptions of language proficiency in complex, dynamic ways that were mediated by interactions with the social context, requiring holistic interpretations of their impact. By using naïve, untrained raters, this study has offered a glimpse into how non-linguists perceive language in real world settings. It thus has implications for language testing practice, as L2 speaking tests generally ask raters largely to ignore what they see and award scores based on what they hear. Speaking tests need to account for the nonverbal and affective repertoires of test takers in their constructs, rating scales, and rater training in order to capture test takers’ full range of language ability. The raters’ focus on listening comprehension also has implications for a broader adoption of integrated speaking assessments, where listening and speaking are assessed together. Finally, the study has important implications for how applied linguists conceptualize L2 communicative competence, as nonverbal behavior plays a much more important role than is currently ascribed.