IDENTIFYING SIGNATURES OF PERCEIVED INTERPERSONAL SYNCHRONY By Eric Novotny A DISSERTATION Submitted to Michigan State University in partial fulfillment of the requirements for the degree of Communication – Doctor of Philosophy 2020 ABSTRACT IDENTIFYING SIGNATURES OF PERCEIVED INTERPERSONAL SYNCHRONY By Eric Novotny Interpersonal synchrony, or the temporal alignment of behaviors between communicators, forms a basis for social bonding in humans. Synchrony is a phenomenon that people can evidently see and feel on a Gestalt level, but the phenomenon itself is intricate. Several qualities of a dyadic interaction can modify its manifestation and effects, including complexity, entrainment, periodicity, and intentionality of the synchronous interaction. To date, it is unclear which of these qualities drive perceptions of synchrony and its corresponding effects. The lack of attention to synchrony’s components results in a potential over- generalization of the concept, which is compounded by a surplus of measurement techniques. As an initial attempt to address these concerns, the current study centers on a specific type of synchrony (complex, reciprocal, repetitive, and purposeful), its correlation with perceived synchrony, and its relationship to a previously identified social outcome variable: outgroup trust. Using full-body motion capture of dyadic partners performing a Tai-Chi routine, three- dimensional movement data were collected and several objective synchrony measures were applied. Then, by overlaying neutral computer avatars onto the motion data, stimulus videos showcasing dyads’ movements were created for an online survey. One-hundred fifteen participants judged synchrony and the leader/follower relationship in the videos. These perception ratings provided a comparison for convergent validity with objective measures. Findings suggested that most objective measures significantly correlated with perceived synchrony, though to different magnitudes. No measures correlated with outgroup trust directly, but when comparing ingroup to outgroup dyads, synchrony correlated with outgroup trust positively for ingroup participants and negatively for outgroup participants. Results indicate that for a complex, reciprocal, repetitive, and purposeful type of synchrony, several measures of synchrony relate to perceptions. More spontaneous/irregular forms of synchrony may require more selectivity in measurement. ACKNOWLEDGEMENTS This work is the culmination of seven years of interest in nonverbal communication, and specifically, interpersonal synchrony. Something about the way that simply moving together leads to all kinds of positive social outcomes gripped my interest early on. One day along this journey, I was reading one of my favorite books, J.R.R Tolkien’s The Fellowship of the Ring, when a line jumped out at me: “A star came out above the trees in the darkening East before them. They went abreast and in step, to keep up their spirits.” I was reminded how human (or Hobbit-like, in this case) it is to enjoy the basic jubilation of moving in unison. At the risk of sounding corny, it also reminded me how my own support network has been a sort of Fellowship throughout graduate school. First, I would like to thank my family. My parents Linda and Jiri, brothers Alex and Gary, as well as their spouses Blair and Allison are always a phone call or a 5-7 hour drive away, which has been a great comfort. Second, I’d like to thank my friends, especially those at MSU. As we all know, graduate school can be overwhelming at times. Identity crises can become a nightly ritual, and constant social comparison can leave one feeling lost. Being able to unwind by playing video games, going out for a beer, or just sitting on the couch and watching terrible movies with friends has made these challenges much less imposing. Thanks to Kevin, Jeff, Clare, Lindsay, Matthew, Adam, and Joomi especially for being my closest friends from the program for the past few years. Third, I have to thank my fiancée Claire. Five years ago, I moved to Michigan with no concept of where I would be at the end of my stay. I had no clue I would meet the love of my life halfway through the voyage. Claire, I don’t need to tell you how invaluable you have been to me iv since we’ve met. We are pandemic buddies, and while other relationships fell apart from cabin fever, ours has only grown stronger. I also want to acknowledge the wonderful second family I have gained from you. I can’t wait to marry you in a year and start our new lives in Georgia this summer with our trusty companion, Peanut. Lastly, I must thank my committee. Rene, I knew from our first class together that you had a verve and rigor in your work that I wished to emulate, as well as a kind and jovial personality. Ralf, you share these traits with Rene. I enjoyed our time building VR brains together and you giving me Vitamin-C in your office. Ron, you were my original advisor, and your meticulousness and underrated personality have been obvious to me all along. Keep being you. Finally, Gary, your passion for science and generosity toward your graduate students has not been overlooked. By encouraging me to reach for the not-so “low hanging fruits,” you have instilled in me a thirst for sound research I won’t soon forget. Thank you also for adapting the Python program for this project. I hope to continue working with this Fellowship of Germans + Ron in the future. v TABLE OF CONTENTS LIST OF TABLES ......................................................................................................... viii LIST OF FIGURES ........................................................................................................... ix INTRODUCTION ...............................................................................................................1 THE ESSENCE OF INTERPERSONAL SYNCHRONY ..................................................3 Definitions ....................................................................................................................3 Functions: Why Synchronize? ......................................................................................4 Measurement .................................................................................................................6 Behavioral coding ..................................................................................................6 Pearson correlation.................................................................................................7 Mutual Information (MI) .......................................................................................8 Dynamic Time Warping (DTW) ............................................................................9 Phase synchrony...................................................................................................10 Rolling Window Time-Lagged Cross Correlations (RWTLCC) .........................11 Dynamic mimicry ................................................................................................14 Outcomes ..................................................................................................................17 Qualities of Synchrony ...............................................................................................17 Complexity ...........................................................................................................17 Entrainment .........................................................................................................20 Periodicity ...........................................................................................................21 Intentionality ........................................................................................................23 STUDY OVERVIEW .......................................................................................................24 METHOD ..........................................................................................................................27 Generation of Movement Database ............................................................................27 Motion capture procedure ...................................................................................27 Spatial normalization ..........................................................................................29 Motion data export ..............................................................................................30 Generation of Video Stimuli .......................................................................................31 Characterization procedure ................................................................................31 Video production procedure ..............................................................................33 Measures ....................................................................................................................34 Perceived synchrony .........................................................................................34 Perceived LFR ..................................................................................................34 Two variables for quantifying synchrony .........................................................35 Overall movement activity ......................................................................35 Positional difference ...............................................................................36 Behavioral data-based synchrony measures .....................................................36 Pearson correlation..................................................................................36 Mutual information .................................................................................36 Dynamic time warping ............................................................................36 vi Phase synchrony......................................................................................37 Rolling window time-lagged cross correlations .....................................37 Dynamic mimicry ...................................................................................37 Outgroup trust ..................................................................................................38 Observer Survey..........................................................................................................39 Participants ..........................................................................................................39 Survey .................................................................................................................39 Procedure ............................................................................................................39 Programming script .............................................................................................40 RESULTS ..........................................................................................................................40 Observer Judgments ....................................................................................................40 Synchrony ratings ...............................................................................................40 Intraclass correlations .........................................................................................41 Perceived leader-follower relationships ..............................................................41 Behavioral Data .........................................................................................................42 Pearson correlations ............................................................................................42 Mutual information .............................................................................................43 Dynamic time warping ........................................................................................44 Phase synchrony..................................................................................................46 Lag offset (from RWTLCC) ...............................................................................48 Dynamic mimicry ...............................................................................................51 Research Questions .....................................................................................................53 Relationship between Synchrony and Outgroup Trust ...............................................55 DISCUSSION ....................................................................................................................56 Findings Pertaining to Research Questions ................................................................56 Implications.................................................................................................................60 Limitations ..................................................................................................................61 Conclusion ..................................................................................................................62 APPENDIX ........................................................................................................................63 REFERENCES ..................................................................................................................86 vii LIST OF TABLES Table 1. Qualities of interpersonal synchrony. ...................................................................2 Table 2. Summary of synchrony measures. .......................................................................16 Table 3. Levels of each quality of synchrony demonstrated in the current study. .............24 Table 4. Percentage breakdowns of leader-follower relationship ratings across all stimuli. ..................................................................................................................42 Table 5. Correlations of major variables. .........................................................................54 viii LIST OF FIGURES Figure 1. Windowed time-lagged cross correlation visualization from Boker et al. (2002) . ............................................................................................................................................12 Figure 2. Motion capture to character animation procedure............................................29 Figure 3. Visualization of two participants’ movement data. ............................................30 Figure 4. Standard wooden mannequin avatar from Bente et al. (2010). .........................31 Figure 5. Still shot of a stimulus video from the observer survey......................................34 Figure 6a. Pearson correlations over time for Dyad 11. .................................................43 Figure 6b. Pearson correlations over time for Dyad 38 ...................................................43 Figure 7a. Dynamic time warping distance matrix for Dyad 11 .......................................45 Figure 7b. Dynamic time warping distance matrix for Dyad 38 .......................................46 Figure 8a. Phase synchrony for Dyad 11 ..........................................................................47 Figure 8b. Phase synchrony for Dyad 38 ..........................................................................48 Figure 9a. RWTLCCs for Dyad 11 ....................................................................................49 Figure 9b. RWTLCCs for Dyad 38 ....................................................................................50 Figure 10a. Lagged synchrony for Dyad 11. .....................................................................50 Figure 10b. Lagged synchrony for Dyad 38. .....................................................................51 Figure 11a. Dynamic mimicry for Dyad 11 .......................................................................52 Figure 11b. Dynamic mimicry for Dyad 38 .......................................................................52 Figure 12a. Offset of dynamic mimicry for Dyad 11 .........................................................53 Figure 12b. Offset of dynamic mimicry for Dyad 38 .........................................................53 ix INTRODUCTION Interpersonal synchrony, or the temporal coordination of behavior between interactants, is a common means of affiliation and bonding among humans (Hove & Risen, 2009; Launay, Tarr, & Dunbar, 2016). In both an historical and developmental sense, synchrony has been identified as a deeply rooted tendency by which humans create emotional and psychological connections. Historically, groups of humans have long performed rituals involving simultaneous, rhythmic movements to increase group cohesion or work toward a shared goal (McNeill, 1995; Wiltermuth & Heath, 2009). Developmentally, infants and their mothers show synchronization of physiological and emotional variables, including heart rate, breathing rate, and mood (Feldman, 2007). In modern and adult life, synchrony has been well-substantiated as a predictor of several social outcomes, including cooperation, rapport, and trust (Delaherche et al., 2012). It has been studied across a range of applied contexts, including psychiatrist-patient relationships (Ramseyer & Tschacher, 2011), teacher-student interactions (Bernieri, 1988), as well as sports (Cohen, Ejsmond-Frey, Knight, & Dunbar, 2010). Throughout these studies, a common theme has emerged: Synchrony is a powerful predictor of human bonding. Despite the solidarity of these findings, another axiom remains evident: Not all synchronous interactions are created equal. Several qualities, or characteristic features, of a synchronous interaction can affect the appearance and outcomes of the specific version of synchrony that emerges. First is the complexity, or possible degrees of freedom, of the behavior. For example, dyadic partners may perform synchronization of simple lateral movements (e.g., Noy, Dekel, & Alon, 2011) or complex movements in a three-dimensional space (Slawinski et al., 2013). Similarly, the number of communication channels being synchronized can vary widely between interactions; some may 1 involve the coordination of eye gaze only (e.g., Harel, Gordon, Geva, & Feldman, 2011), whereas others can involve full-body synchronization (Niewiadomski et al., 2019). Second is the type of entrainment, or the one-way or two-way adaptation of rhythms between actors (Bernieri, Reznick, Rosenthal, 1988; Cacioppo et al., 2014; Konvalinka, Vuust, Roepstorff, & Frith, 2010) that brings about synchrony. Third, the interaction might follow a steady beat (a rhythm with equal intervals between events) or be more chaotic in its cadence. Fourth, coordination behaviors might be purposeful/deliberative or spontaneous/automatic (Koban, Ramamoorthy, & Konvalinka, 2019). See Table 1 for a summary of these qualities. Table 1. Qualities of interpersonal synchrony. Characteristic Example Levels Complexity Entrainment Periodicity Gaze direction vs. full body motion Unilateral, orchestral, reciprocal Repetitive vs. chaotic Intentionality Purposeful vs. spontaneous Each of the above qualities may lead to distinct perceivable properties of the behavior or resulting social outcomes. For instance, eye gaze synchrony and body posture synchrony would likely appear differently to observers and may induce correspondingly varying results. However, to date, few studies have acknowledged how these properties differ or how they may influence perceptions and outcomes. Without understanding these qualities, researchers might treat the multidimensional construct of interpersonal synchrony as a unidimensional one, thereby potentially reducing predictive power (Roznowski & Hanisch, 1990). The “watering down of a powerful concept” (Bente & Novotny, in press, p. 5) has trickle-down effects unto its 2 measurement. Indeed, it is well-known that conceptual definitions drive the measurement of those concepts, but the reverse is also arguable: that the operationalization of phenomena define the concept one is observing. Conceding this assertion, it is important to acknowledge which aspects of synchrony the various available techniques actually measure. For instance, measuring the correlation of postures over time may not assess the latent variable of ‘synchrony’ in the same way as measuring the alignment of phase angles between actors’ movements (Cheong, 2019). To address these issues, the goals of the current research are to (a) identify which objective measures best detect synchrony in an interaction featuring a specific level of each quality, (b) examine how these measures relate to human perceptions of synchrony, and (c) observe which measures of synchrony predict outgroup trust – a previously identified outcome variable (Tamborini et al., 2018). The implications are to enhance understanding about how the qualities of synchrony, their various levels, the collection of available measures, and social outcomes align with global perceptions of synchrony. In the following, I discuss interpersonal synchrony broadly, including its associated definitions, functions, key measures, and outcomes. Second, I discuss in more depth the qualities of synchrony highlighted above. Third, I present a study that compared objective synchrony measures of full-body motion capture data with observer ratings of synchrony. This will allow for examination of which aspects of synchrony truly relate to its perception. THE ESSENCE OF INTERPERSONAL SYNCHRONY Definitions Broadly, the concept of interpersonal or behavioral synchrony has been used to describe the mutual attunement of biological and behavioral rhythms between interactants (Bernieri, 3 Reznick, & Rosenthal, 1988; Burgoon, Stern, & Dillman, 2007). Evidence for synchrony is found in the alignment of the amplitude (strength) and frequency (rate) of bio/behavioral cycles such as heart rate (Mitkidis, McGraw, Roepstorff, & Wallot, 2015), breathing rate (Muller & Lindenberger, 2011), affect (Rafaeli, Rogers, & Revelle, 2007), speech and other expressive behaviors (Cappella, 1981), as well as body movements (Wiltermuth & Heath, 2009). Restricting the current research’s consideration of synchrony to the nonverbal domain, interpersonal synchrony is defined as the temporal coordination of motor behavior rhythms between interaction partners (Bente & Novotny, in press; Bernieri, Reznick, & Rosenthal, 1988; Delaherche et al., 2012). Beyond timing, the form of interactants’ movements may also be similar, though this is not a requirement. Interpersonal coordination types characterized by occasional matching of postures or movements are better subsumed by the term mimicry (Chartrand & Bargh, 1999). Unlike synchrony, mimicry often involves a static match between movement forms, rather than a dynamic sharing of movement timing. A combination of rhythmic matching and form matching has been dubbed ‘perfect synchrony’ (e.g., perfect unison of a marching band), whereas general synchrony only requires a match in timing (e.g., an orchestra; Hale, 2017). Functions: Why Synchronize? Numerous explanations exist regarding the ubiquity of synchrony in human interaction. The first account treats synchronous behavior as an evolutionarily functional behavior (McNeill, 1995; Wiltermuth & Heath, 2009). Throughout human history, countless cultures have developed rituals that foster motor synchrony: From tribal dances around the fire, to religious practices involving simultaneous bowing and rising, to vibrant dancing at modern rave festivals. Such activities are thought to increase cooperation and bonding among group members, as well as 4 identify potential “free-riders,” or members of the group who do not pull their weight in terms of coordinating toward group goals (Wiltermuth & Heath, 2009). In the evolutionary perspective, movement synchrony is thus a way of enhancing group entitativity, or the degree to which a collection of entities is perceived as a unit (Lakens, 2010). The second perspective is not at odds with the first, but instead focuses on synchrony as a perceptual phenomenon that enhances social bonding (Hove & Risen, 2009; Lakens, Schubert, & Paladino, 2016). Here synchronous movement functions to blur self-other perceptual boundaries in the mind. This means that when a person witnesses another individual moving in the same rhythm as his/herself, the neural representation of ‘self’ and ‘other’ becomes almost indistinguishable (Paladino, Mazzurega, Pavani, & Schubert, 2010). Moreover, as Aron, Aron, Tudor, and Nelson (1991) write: “…to the extent a partner is perceived as part of one's self, allocation of resources is communal (because benefiting other is benefiting self)” (p. 242). As such, the self-other merging created from synchrony fosters cooperation and social coordination (Galinsky, Ku, & Wang, 2005), positive outcomes that could explain our propensity to synchronize. A third explanation for synchrony is the brain optimization principle (Koban Ramamoorthy, & Konvalinka, 2019). This recent account implicates the reduced neural energy involved in synchrony (as opposed to out of sync motion) as a reason for its prominence in human behavior. Optimization of brain functionality is founded on the free energy principle, which refers to the brain’s tendency to minimize coding costs when predicting and representing environmental stimuli (Friston, 2010). Neural networks have been compared to man-made electronic devices, in that they are constructed to facilitate minimization of energy cost (Laughlin & Sejnowski, 2003). The optimization principle proposes that during an interaction where two 5 people’s perceptual systems are linked (i.e., they can see or hear each other) synchronization is likely to develop because the brain requires less effort to represent the other’s behavior if it is similar to that of the self. As such, an implicit desire for less mental energy stimulates synchronized movements, and subsequently, through properties of dynamic systems (Schmidt, Carello, & Turvey, 1990), a stable state can emerge where interactants’ behaviors remain in synchrony. Moreover, Koban et al. posit that the reduced effort involved in synchrony is experienced as rewarding. The desirable emotional states deriving from synchrony become associated with the interaction partner, leading to positive bonding variables such as rapport and cooperation. Measurement Interpersonal synchrony has spawned a wealth of measures over the course of its study, ranging from the most basic (simple human coding of the behavior; Bernieri, 1988) to the most complex measures assessing the intricate dynamics of dyadic interactions. In the following sections, I focus on behavioral coding as a basic measure, followed by Pearson correlations, mutual information, dynamic time warping, phase synchrony, and time-lagged cross correlations, (see Cheong, 2019). This range of measures addresses different ways to look at synchrony, from an overall aggregation of similarity to fine pattern recognition. Behavioral coding. A basic measure of interpersonal synchrony is conducted through human observation and identification (Bernieri, 1988). This method approaches synchrony as a Gestalt-level behavior, identifiable not from specific movements per se, but from the degree to which an interacting dyad generally shares tempos, meshes behaviors smoothly, performs movements simultaneously, and assumes similar postures (Bernieri, 1988). As Bernieri (1988; Bernieri, Davis, Rosenthal, & Knee, 1994) contends, synchrony can be faithfully captured from 6 observations of dyadic video, thus not requiring rigorous movement coding or computational analyses. It remains to be seen, though, which aspects of synchronous movement drive these perceptions. This method typically involves observers watching videos of real interactants in conversation or some other dyadic activity. The videos are muted, and observers are instructed to judge nonverbal components of rapport, a psychological construct partially embodied by the physical expression of motor coordination. In fact, rapport is thought to consist of mutual attentiveness, positivity, and coordination of behaviors in interaction (Tickle-Degnen & Rosenthal, 1990). Like synchrony, rapport is characterized as readily observable from nonverbal behavior (Grahe & Bernieri, 1999). A clear limitation with behavioral coding of synchrony is the subjugation of measurement precision for more abstract examination. Bernieri argues that synchrony can be observed from an abstract viewpoint, but this approach does not answer questions pertaining to specific movement patterns (in timing or form) that drive perceptions of synchrony or rapport. Thus, the explanatory power of this method regarding parameters that drive synchrony is relatively limited. Further, this method is confounded by appearance-based variables of the stimulus dyads (see Bente, 2019, p. 11). Bernieri, Davis, Rosenthal, and Knee (1994) created a video mosaic method to account for a different appearance-based confound (smiling behaviors being linked to positivity, thus disrupting measures of synchrony per se), but it is evident from viewing these stimuli that gender and race are still interpretable (Bente, 2019). As such, we again turn to measurement procedures that avoid these confounds and enhance precision. Pearson correlation. The Pearson product-moment correlation, or simply Pearson’s r, is a widely used and relatively simplistic measure of the strength of association between two 7 continuous variables (Puth, Neuhauser, & Ruxton, 2014). It assesses the covariation between variables without making predictions about causal direction. In synchrony research, the correlation between time series can be calculated to give a measure of covariation between two actors’ movement activity. This measure is easy to interpret but is limited in (a) its susceptibility to outliers and (b) its assumption that data are stationary across a time series (Cheong, 2019). To account for these issues, extensions of the correlation, such as cross-correlations and windowed cross-lagged correlations, have been developed (Boker, Xu, Rotondo, & King, 2002; Coco & Dale, 2014). Still, the basic Pearson r is advantageous as a straightforward first look at association between systems. Mutual Information (MI). Mutual information (MI, Shannon, 1948; Moddemeijer, 1989) is a measure of statistical dependence conducted between two discrete or continuous variables (Ince et al., 2017). The formula for MI is quite straightforward, though the calculations for its components are more complex (see Hershey & Movellan, 2000, for example, for more information): MI(x,y) = H(x) + H(y) - H(x,y) MI represents mutual information, H represents entropy, and x and y are the two systems being compared. Entropy is a fundamental measure in Shannon’s mathematical theory of communication (1948), and indicates the amount of information (in Shannons or bits, typically) provided by an event in relation to all other possible events. All else equal, more possibilities in terms of outcomes equals higher entropy. In other words, entropy is the degree of uncertainty regarding an outcome of an event (Shannon, 1948). MI is a measure of comparisons of entropy between two variables. As seen in the formula, MI consists of the addition of the independent entropies of variable x and y, and subtracts from this the joint entropy, or the combined entropy 8 of both events occurring simultaneously. The resulting MI measure refers to how aligned two systems are temporally. A higher MI score indicates higher synchrony; lower MI indicates lower synchrony (Prince et al., 2004). Mutual information has been utilized as a measure of synchrony, mainly within the psychophysiological literature as an indicator of audio-visual synchrony (Hershey & Movellan, 2000; Prince et al., 2004). In the case of Hershey and Movellan (2000), MI was calculated for the synchrony between an audio signal and a spatially localized video signal. As Prince et al. (2004) note, “The HM [i.e., Hershey & Movellan] algorithm is relatively general, detecting temporal synchrony between two time-based input streams” (p. 89). Though little research has used mutual information to measure interpersonal motor synchrony, the generality in this respect gives it potential. In sum, MI is a previously established method with possible application to different synchrony scenarios. One could use this measure to provide an aggregate measure of total alignment in time of two motor systems, though it is not useful for uncovering specific dynamic patterns in the data (e.g., leader-follower relationships). Dynamic Time Warping (DTW). Dynamic time warping is a test that measures similarity between two time series while accounting for time shifts and speed differences (Sakoe & Chiba, 1978). DTW realigns two time series by plotting their data arrays against each other in a matrix and comparing each time series’ data points to those of the other (Mueen & Keogh, 2016; Pouw & Dixon, 2020). It involves calculation of a warp line, or a path through the matrix that connects all the lowest values (i.e., smallest distances between data points). Starting with the upper right cell, which is the last time point for both time series, the DTW procedure checks for the minimum value among the adjacent cells: one cell to the left, one to the diagonal lower left, and one to the lower. Whichever value is the lowest, the warp line is traced to that cell. The 9 drawing of this line continues until it reaches the lower left cell of the matrix. The resulting warp line can be compared to the ideal diagonal to indicate how closely the two time-series are aligned and visualizes any temporal differences or time shifts between the two. Data from two people who were perfectly synced would generate a warp line that was very close to the ideal diagonal. A final distance value can be computed that sums all the minimum values, providing an aggregate representation of overall difference between the two time series. Phase synchrony. Derived from dynamic systems research (Rosenblum, Pikovsky, Kurths, Schafer, & Tass, 2001; Schmidt & O’Brien, 1997), phase synchrony measures the relationship between two time series in terms of their phase. Along with period, frequency, and amplitude, phase is a feature of an oscillating system’s (a system whose parts show a periodic behavior) cycle that defines the dynamic behavior. For an exemplary oscillating system, consider two people each swinging a pendulum next to one another (cf. Schmidt & O’Brien, 1997). The system has two oscillators (each of the swinging pendulums) and each of these oscillators exhibits a periodic behavior. The period is the length of time it takes for the pendulum to complete one movement cycle (i.e., starting from the left, swinging to the right, and reaching the left again). The frequency is the inverse of the period and represents the rate of the behavior. In the pendulum example, this would be complete pendulum swings per time unit. The amplitude is the magnitude of the behavior, or the y-axis in a time series graph. In our example the amplitude is the physical distance the pendulum swings laterally. Finally, the phase is point in the cycle at which the oscillator operates at a given time. A pendulum’s cycle could be thought to start at 0º on the left endpoint, swing to 180 º on the right endpoint, and then restart the cycle at the left again. 10 This explanation of phase relates to the oscillatory behavior of one system. Phase synchrony, however, represents the relation between phase angles of two oscillating systems. Considering the pendulum example again, phase synchrony represents the alignment of the phases of each pendulum over time. Coupling in turn relates to the entrainment of two systems; in the case of interpersonal synchrony, coupling refers to an interdependent relationship facilitated through a shared visual or auditory space (Oullier et al., 2008; Schmidt, Bienvenu, Fitzpatrick, & Amazeen, 1998). It has been shown that once in action, coupled systems stabilize to either an in-phase (same phase angle) or anti-phase (opposite angles; e.g., 0 and 180) angle, and remain in this state robustly (Schmidt et al., 1990). Phase synchrony is a useful measure when researchers are interested in the alignment of rhythms between two systems. It is advantageous in that it can identify synchronous rhythms between even noisy and nonstationary systems (Rosenblum et al., 2001). For example, in a conversation in which movements are not repetitive or cyclical, phase synchrony can still identify interdependencies of phases. However, phase synchrony operates independently of the amplitude of the systems, thus not giving meaningful information about the magnitude of behaviors. Rolling Window Time-Lagged Cross Correlations (RWTLCC). One popular time series method is the windowed time-lagged cross correlation (WTLCCs; Boker, Xu, Rotondo, & King, 2002; Cheong, 2019), which involves calculation of correlations of a given parameter between two time series, like a standard Pearson correlation, but additionally provides correlations between the two series at different time lags. Rather than just calculating the movement similarity between person A and person B at an intersubject lag of 0 (‘on the spot’), the WTLCC provides correlations for each of a range (specified by the researcher) of lags (see 11 Figure 1, from Boker et al., 2002; p. 9). Additionally, WTLCCs improve over correlations alone in that the latter assumes the time series data are stationary – that is, that the mean and variance of a parameter is relatively stable throughout an interaction (Hendry & Juselius, 2000, 2001; Jebb, Tay, Wang, & Huang, 2015; Moulder & Boker, 2018). As many unstructured dyadic interactions are not stable in this regard, the WTLCC addresses this lack of stationarity by using small windows of time rather than producing correlations that cover a whole time series. Figure 1. Windowed time-lagged cross correlation visualization from Boker et al. (2002). From p. 9: “Four pairs of windows selected from two data vectors, X and Y. Results of correlating each pair of windows is stored into the results matrix whose columns represent the relative lag of the two windows and whose rows represent the starting time of the window selected from X.” With WTLCCs researchers can obtain information about the relative covariation between events, whether the parameter represents overall movement activity (e.g., the change in movement from time one to two) or position similarity. Further, when aggregating correlations, one can either calculate the average or maximal correlation between two time series (Coco & 12 Dale, 2014), and with these single metrics, perform basic cross-sectional statistical tests like ANOVA or regression. WTLCC thus has utility for both static and dynamic analyses. The formula for WTLCCs is as follows (see Boker, Xu, Rotondo, & King, 2002): , where Tw is the number of observations in each window, Wxt and Wyt are elements of two time series for t ∈{1...Tw}, X and Y within the windows Wx and Wy, W̄ x and W̄ y are the mean values of each window, and sd(Wx) and sd(Wy) are the standard deviations of each window. Prior to running WTLCCs, the researcher must specify four parameters: window size, window increment, maximum lag, and lag increment (Boker et al., 2002). First, window size is the number of data points in each window. If the window size is too short, the WTLCC measure cannot capture enough information to describe faithfully a relationship. If it is too long, shifting leads or lags can cancel one another out, leading to low correlations. Second, window increment is the amount of time that elapses from one window to the next. If this is too short, successive data rows may show too little variation; too long, and there may be too much variation, resulting in apparently unrelated successive observations. Third, the maximum lag determines the maximum difference between starting points of two windows (each from a different time series). Fourth, the lag increment determines the interval of time between each successive lag observed (Boker et al., 2002). Each of these four elements should be selected by the researcher in a manner appropriate for the phenomenon at hand. For example, one would not use a maximum lag of 2 minutes in an economics study forecasting stock values by decade. The rationale for using WTLCCs over aggregate measures like standard correlations is that it provides a more precise representation of dynamic data patterns. Between the cross- 13 correlations themselves and the resulting heat maps, one can identify (both statistically and visually) similarities and differences between dyad members’ nonverbal behavior throughout an interaction. One can contrast this with basic human coding, which might be useful for having a broad sense of how well a dyad is moving in unison, but is arguably less functional for identifying rapid leader and follower fluctuations or onset/offset patterns of synchrony. A disadvantage of WTLCCs is the difficulty or arbitrariness of selecting values for the four parameters. As Boker et al. (2002) advise, researchers should conduct pilot tests on data to see which parameter sizes fit best. Another potential disadvantage is that the WTLCC’s assumption of local stationarity (i.e., that the mean and variance is stable throughout a window) can be violated, driving down estimates of correlations. In the current study, I leverage a continuous form of the WTLCC, the rolling WTLCC, or RWTLCC (see Cheong, 2019). With this version, the windows overlap to form a higher-resolution graph, depicting smoother changes in leader and follower dynamics. Dynamic mimicry. For the current study, a custom measure was created that compares the positions of each of 15 joints over time. Whereas other measures here make use of overall movement activity (i.e., changes in position), this measure serves as a dynamic comparison of the specific locations of two actors’ body parts in a 3-D space. In this way, it can be thought of as a measure of the ‘perfect synchrony’ (rhythmic matching as well as form matching) discussed earlier in this manuscript (Hale, 2017). In addition, the output of this measure gives a lag offset measure similar to that of the RWTLCC. Dynamic mimicry is useful for any researcher interested in both rhythm and form of synchronous dyads. However, for a researcher who is only interested in rhythm/timing of movements (such as the timing of overall movement activity shifts), this measure would not be 14 the optimal first choice. Given its recent creation, it has not been applied in other synchrony research to date. In the current study, this measure provides the only instance of form-similarity. As such, if only this measure relates to perceived synchrony, these findings would suggest that similar movement form is indeed vital to people’s perceptions of synchrony. Table 2 summarizes the measures covered in this section. 15 Table 2. Summary of synchrony measures. Synchrony Measure Behavioral coding Movement activity Component Targeted Output Type Aggregate Advantages Disadvantages Relatively simple Coarse/imprecise similarity to enact measurement; human error Pearson Movement activity Aggregate Simple and an Ignores dynamic Correlation similarity easy-to-interpret information; overview of inappropriate for synchrony nonstationary data Mutual Shared entropy Aggregate Easy to understand Misses Information leader/follower info Dynamic Time Movement activity Dynamic or Works for signals Computationally Warping similarity Aggregate of varying lengths complex; abstract for Phase Synchrony Phase angle Dynamic or Allows Ignores understanding similarity Aggregate identification of amplitude/magnitude cycle patterns of movements Windowed Time- Movement activity Dynamic or Increases Window size is Lagged Cross similarity Aggregate precision/resolution potentially arbitrary Correlation over Pearson r Dynamic Positional similarity Dynamic or Features both Ignores changes in Mimicry over time Aggregate position and timing overall movement information activity 16 Outcomes Interpersonal synchrony has been found to predict many social outcomes, mostly related to affiliation and/or bonding. In a meta-analysis on the prosocial effects of interpersonal synchrony, Rennung and Goritz (2016) discovered a moderate effect for synchrony on prosocial attitudes and behaviors. Attitudes included entitativity, liking, similarity, and trust (Launay, Dean, Bailes, 2013). Another notable attitude stemming from synchrony not mentioned in that meta-analysis is rapport (Bernieri, 1988; Bernieri, Davis, Rosenthal, & Knee, 1994), a dyadic construct reflecting positivity, attention, and coordination in an interaction (Tickle-Degnen & Rosenthal, 1990). Behavioral outcomes of synchrony included cooperation (Wiltermuth & Heath, 2009), conformity (Wiltermuth, 2012), helping behavior (Cirelli & Einarson, & Trainor, 2014), other-related attention (Miles, Nind, Henderson, & Macrae, 2010), and trust toward outgroup members when the synchrony partner is also an outgroup member (Tamborini et al., 2018). Though most of the effects of synchrony are beneficial in nature (Rennung & Goritz, 2016), effects of the “dark side” of synchrony have been found, including destructive disobedience (Wiltermuth, 2012) and reduced trust toward outgroup members when the synchrony partner is an ingroup member (Tamborini et al., 2018). This outgroup trust measure will be the focal outcome variable in the current research. Qualities of Synchrony Complexity. The phenomenon of interpersonal synchrony can be quite complex, both in terms of the number of channels that synchronize (e.g., facial expressions, movements, gaze) as well as the degrees of freedom of the movements themselves (Bente & Novotny, in press; Poyatos, 1983). Research has been conducted on the full range of complexity of synchronous movements, from the simplest (left-to-right slider movements; Noy, Dekel, & Alon, 2011) to the 17 most complex (full-body motion routines captures in a three-dimensional space; Novotny, Tamborini, & Bente, 2019). Each level of complexity (low versus high; or unidimensional versus multidimensional) offers advantages and disadvantages for study. The benefit of the reductionist approach of measuring synchrony is that it restricts all other variables except a single dimension of movement, allowing researchers to isolate the temporal dynamics of the exercise (Richardson et al., 2007). The resulting time-series graphs show the position of the oscillator on the y-axis and time on the x-axis, providing a clear indication of where the oscillator was in space and time throughout an interaction. Then, researchers can compare the time series of two actors’ movements with various time series methods, such as cross-lag correlations or co-confident motion analysis, a measure of smoothness not used in the current study (Noy, Dekel, & Alon, 2011). The disadvantage of this method, of course, is that it is not necessarily reflective of the bodily synchrony that occurs in the real world. For example, consider the complexity of two figure skaters attempting to synchronize in their routine. Their dance is one of many parts and directions, showcasing the maximal degrees of freedom. To capture fully the intricacies of this interaction, one would need to consider multiple body parts (usually major joints or limbs) as well as their positions in six degrees (three directions of translation in x, y, and z planes; as well as three directions of rotation of pitch, yaw, and roll). As such, multidimensional measures of synchrony are more advantageous in capturing the true nature of full-body synchrony. However, like the reductionist approach, the multidimensional approach has its drawbacks. Namely, the capture of all oscillators and directions has yet to be fully realized in a single method. Currently, two techniques that attempt to capture the complexity of body synchrony are in use: motion energy analysis and motion capture. The former (MEA; Ramseyer 18 & Tschacher, 2011) is a video-based technique that segments frames into cells, and quantifies the amount of pixel change within each cell from one frame to the next. The technique thus provides a frame-by-frame illustration of the amount of movement activity a person shows in each area of the two-dimensional video space, which can then be compared to that of another actor to approximate synchrony. Though this technique is innovative in demonstrating full-body synchrony, it lacks the specificity to show where individual body parts are in space and time; rather it is an aggregate picture of movement activity. More recent versions of MEA involve segmenting cells into head versus body areas, but this approach is still not comprehensive (see Bente & Novotny, in press). Motion capture offers an alternative to MEA and is advantageous in that it captures movements (a) of individual body parts and (b) in all directions, providing the most comprehensive means of assessing full-body synchrony that is currently available. Motion capture is a technique in which participants wear special suits with reflective markers placed throughout and uses unique tracking cameras (typically infrared) to track the position of these nodes over time. The tracking data are transmitted to a software (e.g., Optitrack Motive) which displays the data as a moving avatar for visualization. The data can also provide source material for spreadsheets that display body marker names along the x-axis, time frames along the y-axis, and global position/rotation data in the cells. Arguably, the complexity of this technique most closely approaches that of real-life synchrony, but like the other techniques, motion capture is not flawless. Major drawbacks noted in the literature are the expense as well as the potential for obtrusiveness of the equipment. As Paxton and Dale (2013) note: “Once these systems become cheaper and less restrictive, motion tracking may become a standard tool for bodily synchrony research. Nevertheless, for researchers facing limitations in funding and for those whose questions are not compatible with the high- 19 tech motion capture requirements, body-suit motion capture still poses significant challenges” (p. 331). This assertion is seven years old as of the writing of this paper and is likely accurate that advances will make the technology more cost-effective and seamless. In fact, these advances have already begun to occur since their claim. This lends credence to our use of motion capture in the current research, which captures the motion of 15 key body parts per participant in a 3-D space. Entrainment. Interpersonal synchrony can arise through different types of entrainment (an adjustment to an external rhythm) between interaction partners (Bernieri, Reznick, & Rosenthal, 1988; Schmidt & O’Brien, 1997). Entrainment is accomplished in human interaction through one-way or two-way adaptation within shared visual or auditory environments (Oullier et al., 2008; Schmidt, Bienvenu, Fitzpatrick, & Amazeen, 1998). As Cacioppo et al. (2014) describe, there are three types of entrainment, any of which can precede synchrony. First, unilateral entrainment is one-way entrainment that signifies a strict leader-follower relationship. The resulting behavior can be thought of as temporal mimicry; two people become unilaterally entrained when one actor perceives and follows the rhythmic behavior of another. In this case, the adaptation is said to be one-way because while the follower adjusts his/her rhythms to follow the leader, the leader need not adjust his/her rhythms in return. Second, orchestral entrainment entails multiple actors becoming synchronized indirectly with each other through some zeitgeber (external pace-making entity; Strogatz, 2003). The term orchestral is appropriate here, as a musical orchestra is a quintessential example of the phenomenon. Though the rhythms of the individual players are synchronized, the master rhythm is set by the motions of the conductor. Thus, there is not (necessarily) mutual entrainment between players, but rather, 20 several adjacent instances of unilateral entrainment between each player and the conductor. Lastly, reciprocal entrainment involves the mutual adaptation of two or more individuals’ behavioral rhythms. In this case, there is no strict leader or follower; rather, the dyad or group is engaged in a bi- or poly-directional joint action that is typically spontaneous (Noy, Dekel, & Alon, 2011; Oullier et al., 2008). An improvised dance where the rhythmic movements are unplanned, yet harmonious and co-created (generated by both people rather than only one), would fit as an example of synchrony attained through reciprocal entrainment. Some evidence suggests that the manifestation of entrainment types, or the leader- follower relationship (LFR) can affect the smoothness and performance of the resulting interaction. Noy, Dekel, and Alon (2011) showed that joint improvisers who mutually adapted to one another’s motions performed more confident and smooth movements compared to dyads in a leader-follower condition. Thus, the presence of a true leader/follower versus a more balanced approach has implications for the dynamic patterns embedded within the interaction. If a researcher is interested in the LFR, he or she should employ measures that enable its observance. Static/aggregate measures such as the Pearson correlation give overall information about the similarity between two time series, but provide no insight into the dynamic LFR. Techniques such as windowed time-lagged cross correlations, dynamic time warping, phase synchrony, and dynamic mimicry allow for the visualization and quantification of the LFR. These techniques will be used in the current study to provide evidence of LFR among reciprocally-entrained dyads. Periodicity. Periodicity refers to the regularity of intervals between events over time. It is a concept frequently used in sciences like ecology (Carrero-Colon, Nakatsu, Konpka, & 2006) and geology (Kvet, 1990), often to track and forecast the recurrence of target events. In the case 21 of interpersonal synchrony, periodicity can be thought of as the ‘beat’ of an interaction, or how regularly a person/dyad’s behaviors return to a specific state (Coco & Dale, 2014). As one can imagine, the periodicity of a synchronous interaction is highly variable; periodicity may differ between or within dyadic partners’ movements overall and can even change over the course of an interaction. To understand the concept of periodicity, it is helpful to consider its poles: on one end is a perfectly regular, fixed rhythm. A clear example of this is a well-maintained clock. One can expect that exactly every second, the second hand will move another six-degree tick around the circle. Quite literally, one can set one’s watch to this regularity. In human movement research, Richardson et al. (2007) has shown that human dyads keep a steady rhythm with one another when performing regular movements (both swinging pendulums and rocking in chairs), and tend to stabilize to either an in-phase or anti-phase state. At the other end of the spectrum is a completely irregular rhythm. This type of rhythm (or lack thereof) can often be seen in the erratic cycles of the stock market, which though occasionally exhibiting some trends, can hardly be called regular. Focusing again on the body movements, Fujiwara and Daibo (2016) demonstrated the irregularity of behavior that occurs within unscripted dyadic conversations. In such conversations, there is a lack of steady structure; instead, synchrony can be found as more of an alignment of overall frequencies of movements. The degree of periodicity may determine the measures one can appropriately use to assess synchrony. For example, a highly regular or stationary time series can be represented in terms of synchrony by a simple correlation. However, unstructured conversations or less regular interactions will not allow for simple correlation to precisely measure the dynamics of an erratic interaction. As such, one may turn to measures that account this non-stationarity, such as 22 windowed time-lagged cross correlations or dynamic time warping analyses (Cheong, 2019). In the current study, participants perform a movement routine that lacks a regular and fixed rhythm, separate from the fact that the interaction type is indeed repetitive. Intentionality. Means of entering synchrony can be classified as either purposeful or spontaneous (Bente & Novotny, in press; Koban et al., 2019). Purposeful synchrony occurs when two or more individuals become entrained in service of shared goals (Keller, Novembre, & Hove, 2014), which could feasibly be overt or covert. Overt goals in this sense include sports in which synchronized movements facilitate success, such as rowing (Cohen et al., 2009), whereas covert goals constitute for example religious or cultural rituals that build unity through shared movement (McNeill, 1995; Wiltermuth & Heath, 2009). Some evidence suggests that shared intentionality of the coordination behavior can enhance cooperation and trust more than just matched behavior alone (Reddish, Fischer, & Bulbulia, 2013). Conversely, spontaneous synchrony has no overarching goals, and typically arises through purely physical/dynamical or perceptual means (Oullier et al., 2008; Richardson, Marsh, Isenhower, Goodman, & Schmidt, 2007; Schmidt & O’Brien, 1997). Examples of this include the rocking chairs of adjacent sitters becoming synchronized through an adjustment to shared visual inputs (Richardson et al., 2007), or the claps of a theatre audience becoming coordinated following a performance (Neda, Revasz, Brechet, Vicsek, & Barabasi, 2000). In the current study, participants will purposefully synchronize their movements to one another in the form of a jointly-performed martial arts routine. Each of the above qualities can be altered to form a unique synchronous experience. To begin understanding how altering levels of the qualities can impact (a) synchrony’s perception and (b) its associated outcomes, the current research leverages a previous dataset, which 23 involved an interaction type that exhibited one level of each quality. The previous sections outlined how each quality is represented by that dataset. To review: Regarding complexity, the interactions in this study provided high degrees of freedom – 15 key body parts in a three- dimensional space. Regarding entrainment, the interaction was based on reciprocal entrainment, wherein two participants mutually aligned their behaviors without a present conductor or designated leader/follower. Regarding periodicity, we induced a routine that, while repetitive in the sense that the routine recurred, did not feature a steady pulse or beat. Regarding intentionality, we induced a purposeful type of synchrony that was repetitive; participants were instructed to memorize and re-enact a movement routine with a partner five times. A summation of these levels can be seen in Table 3. Table 3. Levels of each quality of synchrony demonstrated in the current study. Quality Complexity Entrainment Periodicity Intentionality Level High (full-body capture) Reciprocal (mutual adaptation) Repetitive (irregular rhythm, but repeats) Purposeful STUDY OVERVIEW The following study examines how objective measures of a given type of synchrony relate to its perceptions. As a source of full-body motion capture data, we refer to a previous unpublished experiment (Novotny, Tamborini, & Bente, 2019) that induced synchrony in dyads performing a Tai-Chi routine (a rhythmic martial artform), and subsequently tested its impact on 24 trust toward racial outgroup members. The resulting motion data allowed for (a) calculations of various objective synchrony measures and (b) the creation of stimulus videos depicting the movements via neutral computer characters (i.e., characters whose appearance lacked age, race, gender, or cultural cues, which can confound judgments; see Bente, 2019). A series of these stimulus clips was presented to a sample of participant observers, who judged both synchrony and the leader-follower relationship of each dyad. The ratings generated from this study provided a comparison measure against which to judge the objective operationalizations (Bernieri, Reznick, & Rosenthal, 1988; Cappella 1981). If synchrony is a readily perceivable phenomenon at the Gestalt-level, and currently available measures capture synchrony validly, we should see a high correlation between subjective observer ratings and the various objective synchrony measures. Attempting to find just this, researchers (Schoenherr et al., 2019) conducted a study to validate various time series analytic methods by comparing them to human coder ratings. Using a therapist-patient context, they found that only in an artificial condition (comparing person A’s movements with a time lagged version of his/her own movements) were time series methods reliably correlated with human ratings. Conversely, in more naturalistic conditions (where person A’s movements were compared with Person B’s), the algorithms did not agree highly with raters in terms of identifying synchrony. As the authors explain: “Our study revealed that a lot of algorithms with very high identification quality in the artificial configuration failed in the naturally embedded configuration. This could mean that the algorithms had another synchrony concept than the human raters in our study” (p. 17). This comparison between algorithms and coders will be retested in the current study, though with an enhanced means of measuring movements. Notably, Schoenherr’s study used motion energy analysis (Ramseyer & Tschacher, 25 2011) as the technique to extract time series measures. This method, though popular, evidently lacks precision with respect to analyzing specific body part locations throughout an interaction (Bente, 2019). The use of full-body motion capture in the current study may further illuminate the relationship between objective synchrony measures and human observer ratings. To address this possibility, we ask: RQ1: Which objective measures of synchrony predict perceived synchrony? Second, beyond capturing the degree of synchrony, I am also interested in the role of the leader- follower relationship (LFR) in a synchronous interaction. This is often the product of the entrainment of the relationship as outlined earlier. In a leader-follower type interaction, one person mimics the behavior of another with some delay, whereas in a reciprocally adaptive interaction, each person synchronizes through mutual prediction and reaction in real-time (Konvalinka et al., 2010). The nature of this relationship has been shown to impact the smoothness or performance of the involved partners (Noy, Dekel, & Alon, 2011). If LFR is a central defining factor of a synchronous interaction, and synchrony can ostensibly be perceived by observers, then objective measures that can accurately identify leader-follower patterns should align with observers’ ability to detect these same patterns: RQ2: Which objective measures of synchrony predict observer ratings of leader-follower relationships (LFRs)? Lastly, I am interested in whether the outcomes of these various measures can predict a previously identified outcome variable of outgroup trust (Tamborini et al., 2018). Interpersonal synchrony has previously been linked to a reduction in outgroup bias generally. Inzlicht et al. (2012) found that mimicking an African-American actor’s motions improved implicit attitudes (e.g., lowered bias) toward African-American people compared to passive viewing of an 26 African-American actor’s movements or mimicry of a Caucasian actor’s movements. Similarly, Tuncgenc and Cohen (2016) found that for children assigned to minimal groups, those who synchronized with outgroup members exhibited higher intergroup bonds compared to those who were asynchronous with respect to outgroup members. Finally, we have elaborated previously on the findings of Tamborini et al. (2018), which showed a marginal increase in outgroup trust given synchrony with an outgroup member. An enhanced sense of affiliation is often cited as the mechanism through which synchrony breaks down intergroup barriers. Coordination in general functions as a social binding variable, increasing a group’s sense of unity (Wiltermuth & Heath, 2009). This enhanced unity can attenuate previously established biases toward outgroup members, facilitating trust. If the measures we feature in the current research faithfully capture synchrony in its essence, we would expect to see these measures correlate with outgroup trust. RQ3: Which synchrony measures correlate significantly with outgroup trust? Generation of Movement Database METHOD Motion capture procedure. The OptiTrack Motion Capture system (NaturalPoint) was used to collect full-body motion data. Motion capture took place in two divided square cells (15’ x 15’) in a laboratory. Twelve optical cameras were suspended from a truss system in each cell. These cameras detect motion through transmission of infrared light from reflective markers on the participants’ body suits. The suits are composed of tight-fitting black Nylon, and feature 37 passive Velcro markers placed throughout the participant’s body. Motive, the software that operates the OptiTrack system, recorded and stored the motion tracking time series data. 27 The motion capture procedure was divided into four phases. In phase one, participant dyad members individually entered separate rooms in a laboratory and donned motion capture outfits before completing a pre-test outgroup trust measure at separate computer stations. Next, in phase two, they separately learned and mimicked a Tai-Chi routine from a virtual avatar appearing as a gender- and race-neutral wooden mannequin. This instructor, who appeared on a large wall projection, performed five repetitions of a 30-second routine, thus providing the training necessary for the next phase. In phase three, participants were instructed to perform the same routine they just learned, but now with a black or white virtual partner (the main manipulation) appearing on the screen, whose movements were generated in real-time by their real dyadic partner. The avatar movements were created by relaying the movement data in real time to an animation software that displayed a black or white avatar (matched to the dyad’s gender; Figure 2 demonstrates the routine in phase three). Notably, this was the stage in which the participants’ movement data (body part locations in 3-D space at each time frame) were collected via motion capture. Finally, in stage four participants completed a post-test outgroup trust measure to assess the effect of partner group and synchrony on this outcome. 28 Figure 2. Motion capture to character animation procedure. Spatial normalization. A spatial normalization procedure of motion capture data was performed as recommended by Poppe et al. (2014). This is advised for comparing motion capture data between actors of different sizes and with different starting positions. To begin, I merged the motion capture files of two dyadic partners using Motionbuilder 2018 (Autodesk). I then applied the a pre-rendered character (described below) to each actor’s motion capture data for visualization purposes. Once characterized, I scaled uniformly each character according to the average size of a male (1.75m or 5’9”) and female (1.62m or 5’4”) in the U.S. After scaling, I translated each actor’s root node (the hip joint) to the origin of the scene: the point where x, y, and z are all set to 0 in Motionbuilder’s viewer window. Next, I ‘snapped’ the two actors’ hips to this origin; that is, throughout the scene, the translation both actors’ hips were constrained to the origin point while the rest of their bodies moved freely as in real life. The last step here was to 29 set the starting orientation (at frame 0) of each actor to the front of the scene by rotating the Woody’s hip joint to 0º around the y-axis. The resulting scene shows two identically sized characters, both facing forward, and their hips fixed together. See Figure 3 for a still shot of this result. Figure 3. Visualization of two participants’ movement data. Figures are snapped by the hips and standardized in size. The orange actor is Participant A, and the blue actor is Participant B. Motion data export. The movement data were exported (one data file per dyadic partner) via the tool Export Global Data for Motionbuilder 2018 (Leuschner, 2010). This tool outputs the movement data as a spreadsheet in which the rows are time frames (at 25Hz) and the columns are the movement translation in x, y, and z dimensions of 15 key body parts as advised by Poppe et al. (2014). Given a dyadic routine lasting 2.5 minutes, this would result in a rich dataset of 168,750 cells (3750 frames x 45 body part translation columns) per partner. 30 Generation of Video Stimuli Using the motion database, I created stimulus videos of dyadic partners performing the Tai-Chi routine side by side. This process involved rendering the motion capture data as standard virtual characters and producing a video for embedding into the final survey. Characterization procedure. First, the motion capture take data (time frames x body part locations) of the first participant in a dyad (Participant A) were exported from Motive as an FBX file. FBXs are animation files that operate within Motionbuilder, which features a plugin for Motive. Using the Motive plugin, I overlaid a neutral character, appearing as a wooden mannequin, onto the motion capture data for the first participant in a dyad. This wooden character was adopted from previous research (Bente, Leuschner, Al-Issa, & Blascovich, 2010), and can be seen in Figure 4. The purpose of using a neutral character such as this was to disguise the identities of participants in a controlled manner while preserving the fidelity of the human movement (cf. Bente, 2019). 31 Figure 4. Standard wooden mannequin avatar from Bente et al. (2010). Once the character was applied, I merged Participant A’s dyadic partner’s (Participant B) movement data into the same FBX file. Because these two participants originally performed the Tai-Chi routine at the same time, I produced a file that shows both participants performing the routine with the same start/end frames and side-by-side – even though in real life, they were physically separated. After merging the partner’s data, I applied the character to Participant B’s movements as well. The next step was to align the two characters so that they were are facing forward at frame one and were each centered on their half of the screen. To do so, I set the global rotational angle (the angle of a given body part with respect to the scene’s origin point) to 0º at the hip joint (the body hierarchy’s root node). Following this, I checked that the two actor’s movements were generally going in the same direction throughout their interaction. In the dyadic interactions, 21 pairs performed opposite movements (i.e., mirror mimicry) whereas 17 performed same- direction movements (i.e., rotational mimicry). If they were mirrored rather than rotational, I corrected this by mirroring Participant B’s movements across the y-axis. For instance, if Participant A typically swung her arm to the left and Participant B swung hers to the right, I flipped B’s movements so that both swung to the left. While it is an empirical question whether the direction of imitation matters for perceptions of synchrony, we did not wish to test this variable in the current research; feasibly, observers could witness a highly syncing dyad who was mirrored (rather than rotational), and this could impact the synchrony ratings differently compared to a highly synchronizing dyad who mimicked rotationally. In sum, control of the visual stimuli was more important in the current research than testing the effect of movement direction. 32 Video production procedure. Once the characterization process was complete, the scene was rendered as an AVI file in Motionbuilder. The frames were set to PAL (25Hz, or 25 frames per second) and the video was compressed to the highest quality available within Motionbuilder. The resulting files averaged 3668.87 frames, or about 2 minutes and 27 seconds. The next step was to segment each AVI file into the first three cycles of the Tai-Chi routine. This was done to provide more stimuli for the survey, as well as to provide more appropriate time segments for observers. Segments were created by noting the time frame at which the Tai-Chi cycle restarted; that is, the point at which both participants had their arms down at the starting point at the same time. If a simultaneous restarting of both participants did not occur, I noted when just one participant restarted. AVI cutting was performed in the program Bandicut, and the files were compressed to MP4 files using the program VLC Media Player. Because one dyad had an erroneous third segment resulting from a capture error, the final stimulus pool featured 113 videos (38 dyads x 3 segments, minus 1 faulty segment), with an average segment length of 24 seconds. An example of the final stimulus video participants would view is demonstrated in Figure 5. 33 Figure 5. Still shot of a stimulus video from the observer survey. Measures Perceived synchrony. Perceived synchrony was measured on a slider scale from 0 (no synchrony) to 100 (perfect synchrony) for each video. Participants received the following instruction: “After each video, we will ask you (a) how "in sync" the pairs were, and (b) whether one person led the interaction (versus a more balanced interaction). "In sync" just refers to how smoothly and similarly the two moved together in time ('high coordination'.) On our slider scale, 100 = perfect sync. The opposite of "in sync" would be clumsy, out of tune, or awkward ('poor coordination'). On our scale, 0 = no sync.” Perceived LFR. The perceptions of the leader-follower relationship of the dyad was judged for each video through the following multiple-choice item: “Was one person leading the 34 interaction, or was it fairly balanced?” Possible responses to this question were: “Person A (on the left) led mostly,” “Person B (on the right) led mostly”, “It was fairly balanced”, and “Not sure.” Responses to this item were tallied for each stimulus and divided by the total number of occurrences of that stimulus to provide the proportion A led per stimulus, the proportion B led per stimulus, and the proportion of balanced ratings per stimulus. Two variables for quantifying synchrony. Two variables were used to quantify synchrony: the similarity of shifts in overall movement activity (used for Pearson correlations, MI, DTW, Phase Synchrony, and RWTLCC) and the similarity of position (used in Dynamic Mimicry). Overall movement activity. For most of the various synchrony measures outlined below, the variable of interest is the overall movement activity exhibited by a single dyad member with respect to his/her partner. Rather than focusing on the form of movements, for example in behavioral mimicry research (Lakin & Chartrand, 2003), this approach focuses on the timing of general movement activity, the variable more central in the concept of synchrony (Hove & Risen, 2009). Overall movement activity of each participant was calculated in the following way. First, taking a file from each dyadic partner in one dyad, the x, y, and z translation of 15 primary body locations joints were targeted as suggested by Poppe et al. (2014). These include: Chest, left arm, left forearm, left hand, right arm, right forearm, right hand, head, right upper leg, light leg, right foot, left upper leg, left leg, and left foot. Second, we performed a standard Z- transformation vertically (over time) to normalize the data for 14 of these 15 joints (the hips were locked at 0, 0, 0 throughout the interaction). Lastly, an average was conducted laterally (across joints) to give a single “movement activity” score for each time frame. This score served as the 35 y-axis variable that fluctuates in the various time series measures used in this study (except for dynamic mimicry). Positional difference. Rather than implementing the changes in overall movement as a measure of synchrony, this variable represents a comparison of specific positions of two actors’ body parts in a shared global space. Specifically, it is the difference in Euclidean distance (x, y, and z translation) between all 14 joints of two actors. A lower difference of positions indicates higher mimicry, which, when looked at over time and with different lags, gives us the dynamic mimicry measure detailed below. Behavioral data-based synchrony measures. The following measures were collected using a combination of Python codes, mainly using a synchrony suite created by Cheong (2019). The final program script for all analyses can be found in Appendix A. Pearson correlation. To compute a Pearson correlation between two time series, the Python program takes the average value of a given time series and correlates it with the average value of a second time series. Mutual information. Mutual information (MI) was calculated by a custom Python program (https://stackoverflow.com/questions/20491028/optimal-way-to-compute-pairwise- mutual-information-using-numpy) that was appended to the original program by Cheong (2019). Uusing the formula mentioned earlier: MI(x,y) = H(x) + H(y) - H(x,y), where MI is the mutual information, and H(x) is the entropy of time series x, H(y) is the entropy of time series y, and H(x,y) is the joint entropy (shared by both systems). Dynamic time warping. Dynamic time warping (DTW) is computed by minimizing the distance between two time series’ data points in a matrix, and comparing the resulting diagonal line to an ideal diagonal. The package dtw (https://github.com/pierre-rouanet/dtw) was used to 36 visualize the DTW matrix, and provide overall distance scores for each dyad. These distance scores indicate the distance of the diagonal to the ideal line; a smaller distance indicates higher synchrony. Phase synchrony. The phase angles of two time series can then be compared for a measure of interpersonal synchrony. First, one must transform the movement data using a Hilbert transform, which separates a time series signal into its phase and power (Zayed, 1998). Then, the phase angles are plotted along a time series and inter-subject comparisons can be made. To obtain a score of phase synchrony, the program compares the phase angles by the following: PS = 1 - sin(|al1-al2/2|), where PS is phase synchrony, al1 is the phase angle of time series A at a given point, al2 is the phase angle of time series B at a given time point. Finally, this PS score is averaged over a whole time series to give a measure of overall phase synchrony, to be used for correlations with other variables. Rolling windowed time-lagged cross correlation. The program executes Pearson correlations between two time series over given windows of time, and smoothing this process out with a more continuously sliding window. In the current study, we use a window size of 125 frames (4 seconds) for correlations, but a window increment of only 5 frames (.2 seconds). In this way, the resulting time series graph gives a smoothly rolling output that is more visually interpretable. Dynamic mimicry. This measure aggregates the position (rather than overall movement activity) of one participant’s joints limbs in x, y, and z directions of translation, and compares these values with those of his/her dyadic partner. The difference in position is then plotted over a time series with a range of time lags along the y-axis (as with RWTLCCs). The lag offset at 37 which positional differences are the smallest (i.e., most similar) is plotted in a separate graph, and will be used as the aggregate measure of positional similarity for correlation with other variables. Outgroup trust. Outgroup trust was assessed during collection of the initial movement database. It was measured using a custom computer game called “Will they Ship?”, adapted from Bente et al. (2014.; original game developed by Bolton, Katok, & Ockenfels, 2004). The game involves choosing to buy or not buy textbooks from 32 (16 pretest, 16 posttest) virtual salespeople who would ship or not ship the textbook. A payoff matrix similar to the famous Prisoner’s Dilemma game, with potential risks and rewards of making a deal with another player, was used to motivate the actions of players. These sellers were depicted as White or Black static avatars (16 of each), pretested by Tamborini et al. (2018) to appear neutrally trustworthy and natural looking. Outgroup trust was calculated by first tallying the number of ‘outgroup buys’ for pretest and posttest rounds. Outgroup buys were instances when a player selected ‘buy’ from an outgroup salesperson (for example, when a White participant bought from a Black salesperson). Next, I aggregated outgroup buys into a pretest proportion (outgroup buys/total encounters with outgroup salesperson during pretest) and posttest proportion (outgroup buys/total encounters with outgroup salesperson during pretest). Following this, the pretest proportion was subtracted from the posttest proportion to get an outgroup trust change score for one participant. Finally, the change scores were averaged between dyad members to generate a dyadic average for outgroup trust change. 38 Observer Survey Participants. Participants were 115 individuals (MAge = 23.3, SDAge = 10.92, 54% female, 78% White) collected from two sources. The first set of participants (n = 13, MAge = 46.0, SDAge = 17.62, 54% female, 85% White) consisted of family and friends of the researcher, who were provided a survey link via email. These participants were blind to the research questions of the researcher, and received only thanks for participation. The second set (n = 102, MAge = 19.89, SDAge = 1.50, 50% female, 74% White, 14% Asian, 12% other races) consisted of undergraduates from a large public university in the midwestern United States. This group participated to fulfill optional research credits for a communication course of their choice. All procedures were approved by the institutional review board at the university from which the second sample was drawn. Survey. A survey was created in Qualtrics Survey Software. The survey presented to participants a pseudo-random series of 30 videos to view and rate. This stimulus sampling method was chosen because it selected videos at random while still ensuring each video was viewed the same amount. The survey asked participants to watch each video until it auto- advanced to the next page, which asked them to rate both synchrony and LFR. Demographic questions, which appeared at the end of the survey, asked for age, race/ethnicity, and gender. Procedure. A link to an online consent form was distributed to friends and family via email, and to undergraduates through a participant pool management software. Upon consenting to participation, the consent page rerouted participants to the observer survey. Due to coronavirus-related quarantine procedures, participants filled out the survey from a location of their choosing rather than a computer laboratory. The survey guided participants through viewing and rating of 30 stimulus clips. 39 Programming script. A custom Python program was adapted from Cheong (2019). In the original code, this program calculates Pearson correlations, dynamic time warping, phase synchrony, and windowed time-lagged cross correlations. A custom script that computes Z- transformations of selected joints, and which additionally calculates mutual information, was appended to this code (see Appendix A). The script begins firstly by importing necessary packages and defining the variables to be measured. Secondly, one selects the variables for which he/she would like to view figures and descriptive statistics. Thirdly, once the measures are selected, the program looks for a list of CSV files (described in the previous section) from which to derive data. In the current study, this list includes 38 files for participant As and 38 files for participant Bs. Fourthly, the “movement activity” score for each time frame is conducted in the means described in the Overall movement activity section. Fifthly, a filter of the user’s choosing is applied. In the current study, we used a Butterworth bandpass filter, which normalizes the ‘cutoff’ frequency, at which data finer than a certain threshold are smoothed (Paxton & Dale, 2013). Finally, when the program runs, it outputs text files of descriptive data for selected measures (e.g., distance scores for DTW for each dyad) as well as a PDF of all figures. Observer Judgments RESULTS Synchrony ratings. An average synchrony score (perceived sync) between 0 and 100 was calculated for each of the 113 stimulus videos: M PerceivedSync = 44.47, SD PerceivedSync = 16.18, Max PerceivedSync = 73.84 (Dyad 11), Min PerceivedSync = 21.20 (Dyad 30). The synchrony means were also broken down by averaging synchrony across the three segments for each dyad: M Segment1PerceivedSync = 44.49, SDSegment1PerceivedSync = 16.48; MSegment2PerceivedSync = 44.26, SDSegment2PerceivedSync = 18.23; MSegment3PerceivedSync = 43.64, SDSegment3PerceivedSync = 17.15. An 40 ANOVA was conducted to test whether these three segments differed statistically from each other (in other words, to see if there was a change in synchrony ratings over time). The difference between time segments was non-significant, F(2, 112) = .054, p = .947, partial eta2 = .001, suggesting no change in ratings over the course of the three segments of dyadic interaction. Table 4 shows the breakdown of synchrony ratings from the mostly highly- to lowly-rated dyad. Intraclass correlations. Intraclass correlations (ICCs) were conducted across all stimuli to check for inter-rater reliability in judging perceived sync. A matrix was created with videos as rows and raters as columns. Because each rater viewed only a random 30-video subset out of the entire 113, the matrix featured a high number of empty cells – making a normal ICC calculation unfeasible. As such, I deleted all empty cells by pushing filled cells to the left. While this procedure changes the columns (i.e., distorts which rater made which rating), it preserves the rows (i.e., the ratings each stimuli received). The resulting measure is thus an indicator of the consistency with which stimuli were rated, not the consistency between raters per se. Using this strategy, consistency between 10 ratings for each video (the minimum number of columns in which all videos were rated) was analyzed for reliability. The average ICC was high at .908, 95% CI (.881, .932), F (111, 999) = 11.09, p < .01, suggesting strong agreement within stimuli ratings. Perceived leader-follower relationships. Three average proportions for each stimulus were calculated: (a) the percentage A was rated as leading, (b) the percentage B was rated as leading, and (c) the percentage of ratings indicating a balance of leadership/followership. The breakdowns are summarized in Table 4. Pearson correlations were conducted to check for covariation between perceived sync and the perceived proportion that A led (LFR_A), B led (LFR_B), or that there was an even LFR (LFR_E). Perceived sync was positively correlated with 41 LFR_E, r = .576, p < .001, negatively correlated with LFR_A, r = -.421, p = .009, and had no relationship with LFR_B, r = .126, p = .45. This suggests that the perception of a balanced LFR is related to perceived synchrony. Table 4. Percentage breakdowns of leader-follower relationship ratings across all stimuli. Mean % A Led 34.14 B Led 43.42 Even LFR 22.45 Maximum % 86.07 (Dyad 38) 77.59 (Dyad 23) 49.11 (Dyad 11) Minimum % 4.89 (Dyad 12) 6.91 (Dyad 38) 7.02 (Dyad 38) Note. Consistent with the idea that synchrony involves a reciprocal leader-follower relationship, Dyad 11 had the highest even LFR as well as the highest average synchrony ratings. Behavioral Data For each of the objective synchrony measures, I provide a comparison of figures between Dyad 11 (the highest LFR_E, or most balanced dyad) and Dyad 38 (the lowest LFR_E, or least balanced dyad). A juxtaposition of these figures demonstrates how each measure showcases the range of synchrony from high to low (see Figures 6 through 12). Pearson correlations. The average correlation of movement activity between participant A and B was calculated for each dyad with an intersubject lag of 0 frames, MPearson = 0.28, SDPearson = 0.27. The correlation between participants in Dyad 11 was r = 0.66, p < .01. The correlation for Dyad 38 was r = -0.02, p = 0.23. The time series of general movement activity featuring these correlations can be seen in Figures 6a and 6b. A higher correlation indicates higher synchrony. 42 (6a) Dyad 11: Pearson r = 0.66 Figure 6a. Pearson correlations over time for Dyad 11. Higher correlations indicate higher synchrony, which is indicated by a close matching of the orange (Participant A) and blue (Participant B) lines. Dyad 11’s lines overlap more often than Dyad 38 (see Figure 6b). (6b) Dyad 38: Pearson r = -0.02 Figure 6b. Pearson correlations over time for Dyad 38. Mutual Information (MI). MI indicates the information we can predict from one system based on observations of another system. The average MI for all dyads was MMI = 10.68, SDMI = 0.15. For Dyad 11, MI = 10.84, and for Dyad 38, MI = 10.86. Interestingly, Dyad 11 had a 43 slightly lower MI score than Dyad 38, despite Dyad 11 generally appearing higher across other synchrony measures. This suggests that MI may not be strongly aligned with other measures of interpersonal synchrony. Dynamic Time Warping (DTW). The DTW score indicates synchrony of overall movement activity irrespective of the length of an interaction. The average distance (DTW) for all dyads was MDTW = 6221.92, SDDTW = 1132.20. For Dyad 11, DTW = 5674.25, and for Dyad 38, DTW = 6356.17. A comparison of these dyads’ DTW scores can be found in Figures 7a and 7b. The lower distance score of Dyad 11 indicates that this dyad exhibited more similarities in overall movement activity over time compared to Dyad 38. 44 (7a) Dyad 11: DTW distance = 5674.25 Figure 7a. Dynamic time warping distance matrix for Dyad 11. The x-axis is one participant’s timeline whereas the y-axis is his/her partner’s timeline. The white line traces the minimum distance between participants’ movement activity at each time point. A white line more approximating a perfect diagonal represents a smaller distance, or higher synchrony. Dyad 11’s distance is less than that of Dyad 38 (see Figure 7b). 45 (7b) Dyad 38: DTW distance = 6356.17 Figure 7b. Dynamic time warping distance matrix for Dyad 38. Phase synchrony. This measure indicated the degree to which the phase angles of two participants’ overall movement activity were aligned. The average phase synchrony was MPhaseSync = 0.48, SDPhaseSync = 0.10. For Dyad 11, PhaseSync = 0.69, and for Dyad 38, PhaseSync = 0.38. A comparison of these dyads’ phase synchrony can be found in Figures 8a and 8b. The higher PhaseSync score of Dyad 11 indicates that Dyad 11 was more temporally aligned in the cycles of their movements compared to Dyad 38. 46 (8a) Dyad 11: Angle at each Timepoint and Instantaneous Phase Synchrony (bottom) Figure 8a. Phase synchrony for Dyad 11. The top graph shows the angle at each timepoint, with red representing one actor and blue the other. The bottom graph shows phase synchrony (from 0 to 1) continuously throughout the interaction. Dyad 11 more often shows phase synchrony scores approximating 1.0, whereas Dyad 38 shows this less often (see Figure 8b). 47 (8b) Dyad 38: Angle at each Timepoint (top) and Instantaneous Phase Synchrony (bottom) Figure 8b. Phase synchrony for Dyad 38. Lag offset (from RWTLCC). This measure indicates the amount of lag between participants at which synchrony of movement activity was typically the highest. The average lag offset in frames, given by the RWTLCC, was MLag = 18.42, SDLag = 13.58. For Dyad 11, Lag = 2 frames, and for Dyad 38, Lag = 43 frames. The RWTLCC graphs can be found in Figures 9a and 9b, and a comparison of these dyads’ lag offsets can be found in Figures 10a and 10b. 48 (9a) Dyad 11: Rolling Window Time-Lagged Cross Correlations Figure 9a. RWTLCCs for Dyad 11. Blue indicates highly negative correlations, red indicates highly positive correlations, and white indicates no association. The midline for Dyad 11 (0 offeset) shows that the highest positive correlation (i.e., synchrony) occurred almost on the spot. This is not true for Dyad 38 (see Figure 9b). 49 (9b) Dyad 38: Rolling Window Time-Lagged Cross Correlations Figure 9b. RWTLCCs for Dyad 38. (10a) Dyad 11: Offset (RWTLCCs) Figure 10a. Lagged synchrony for Dyad 11. Optimal offset for Dyad 11 is 2 frames, and offset for Dyad 38 is 43 frames (indicated by the red dotted lines; see Figure 10b). Left of the black dotted line indicates that Subject A leads, and right of the black dotted line indicates that Subject B leads. The smaller optimal offset for Dyad 11 compared to Dyad 38 suggests a more temporally aligned dyad with respect to overall movement activity. 50 (10b) Dyad 38: Offset (RWTLCCs) Figure 10b. Lagged synchrony for Dyad 38. Dynamic mimicry. Dynamic mimicry represents the similarity of positions of two actors’ joints over time. Here we show calculations of dynamic mimicry for Dyads 11 and 38 (see Figures 11a and 11b). Further, the optimal offset of dynamic mimicry (i.e., the time lag value at which positional differences were smallest) was also calculated. For Dyad 11, offset = 7 frames; for Dyad 38, offset = -41 frames. The absolute values of these lags were used in correlations, as the sign should not impact the strength of association. 51 (11a) Dyad 11: Dynamic Mimicry Figure 11a. Dynamic mimicry for Dyad 11. The blue coloring around the midline for Dyad 11 indicates a low difference in positions between the two actors when lag = 0. This is less evident for Dyad 38, whose coloring was less consistent in this regard (see Figure 11b). This suggests that the positions of Dyad 11’s actors’ body parts were more aligned when lag = 0 compared to those of Dyad 38. (11b) Dyad 38: Dynamic Mimicry Figure 11b. Dynamic mimicry for Dyad 38. 52 (12a) Dyad 11: Offset (Dynamic Mimicry) Figure 12a. Offset of dynamic mimicry for Dyad 11. Dyad 11 shows a lag closer to 0 compared to Dyad 38, indicating less of a delay when movement similarity was at its peak (see Figure 12b) (12b) Dyad 38: Offset (Dynamic Mimicry) Figure 12b. Offset of dynamic mimicry for Dyad 38. Research Questions To address the research questions, first, correlations were run among the among the various objective and subjective synchrony measures as well as outgroup trust and LFR_E. The results of these correlations can be seen in Table 5. To refresh, we would expect high correlations among all synchrony variables, though the sign depends on the measures. For 53 Perceived Sync, Pearson r, MI, PhaseSync, and LFR_E, a higher score indicates more synchrony. For DTW, RWTLCC, and Dynamic Mimicry, a lower score indicates synchrony. Table 5. Correlations of major variables. Perceived Pearson r MI DTW PhaseSync RWTLCC Dyn. Outgroup LFR_E Sync Mimicry Trust 1.Perceived Sync 2.Pearson r .828** 3.Mutual Info .337* .207 4.DTW -.292 -.468** .297 5.PhaseSync .849** .967** .260 -.372* 6.RWTLCC -.512** -.586** -.428** -.125 -.569** 7.Dyn. Mimicry -.717** -.639** -.269 .286 -.648** .583** 8.Outgroup Trust -.243 .052 -.037 -.269 -.001 -.074 .049 9.LFR_E .576** .516** .318 .103 .581** -.441** -.397* -.006 **. Correlation is significant at the 0.01 level (2-tailed). *. Correlation is significant at the 0.05 level (2-tailed). In response to RQ1, which asked which measures of synchrony predict perceived synchrony, perceived sync correlated significantly with phase synchrony, r = 0.85, p < .001, followed by Pearson r, r = 0.83, p < .001, dynamic mimicry, r = -0.72, p < .01, and mutual information, r = 0.34, p = .04. A regression was run to check for a causal linear relationship between the objective measures and perceived sync. Diagnostics revealed that multicollinearity was not a concern, as only one variable pairing showed variance proportions higher than .90 (see Hair, Black, Babin, & 54 Anderson, 2013). The regression revealed strong fit of the predicted model, F(6, 31) = 19.60, p < .01, R2 = .74. Dynamic mimicry was the only individual variable that significantly predicted perceived sync, standardized beta = .34, t = -2.81, p = .01. Regarding RQ2, which asked which objective measures of synchrony predict observer ratings of leader-follower relationships (LFRs), the balance of the leader-follower relationship (LFR_E) was correlated significantly with Pearson r (r = 0.58, p < .01), phase synchrony (r = 0.58, p <.01), lag offset (r = -0.44, p < .01) and dynamic mimicry (r = -0.40, p = .014). A regression checking for the effect of the objective measures on ratings of LFR was conducted. The regression showed moderate fit, F(6, 31) = 4.42, p < .01, R2 = .46. No individual variables significantly predicted LFR_E, all p > .05. Relationship between Synchrony and Outgroup Trust Research question 3 inquired whether any of the synchrony measures would predict outgroup trust. No significant relationships were found between synchrony variables and outgroup trust (all correlations between synchrony variables and outgroup trust > p = .10.). However, in the original study by Tamborini et al. (2018), the effect of synchrony on outgroup trust was only found when moderated by the group membership (ingroup versus outgroup) of the virtual partner. As such, separate correlations were calculated for those who had ingroup versus outgroup partners. For those with ingroup partners, Pearson r (r = 0.52, p = .03) and DTW (r = - 0.59, p = .01) were both significantly correlated with outgroup trust. For those with outgroup partners, perceived sync (r = -0.54, p = .02) alone correlated significantly with outgroup trust. These findings are inconsistent with those of Tamborini et al. (2018), though in that study, the nature of the synchronous routine was different (i.e., more of a mimicked interaction compared to reciprocal), which may account for the difference in results. 55 DISCUSSION The goal of this research was to illuminate which objective measures of interpersonal synchrony best relate with global perceptions of synchrony, while taking into consideration that synchronous interactions differ in several qualities. A type of interaction that was complex, reciprocally entrained, repetitive, and purposeful was used as a first example with which to apply these measures. Results indicated that numerous measures including phase synchrony, Pearson correlations, mutual information, and dynamic mimicry are all linked to global synchrony perceptions for this interaction type. Moreover, the reliability of synchrony judgments was high when comparing ratings within stimuli. Next, a balance in LFR was related to Pearson r, phase synchrony, lag offset (RWTLCC), and dynamic mimicry. Interestingly, only a few synchrony measures related to the findings about outgroup trust as previously discovered by Tamborini et al. (2018), and these were in the opposite direction than expected. In the following discussion, I first remark on the findings pertaining to the research questions, speculating on how these findings could change with interaction types featuring different levels of synchrony qualities. Following this, I discuss implications and limitations of this research. Findings Pertaining to Research Questions The first research question inquired which measures of synchrony would predict the subjective measure perceived sync. In order of correlation strength from strongest to weakest, phase synchrony, Pearson r, dynamic mimicry, and MI all were significantly related to perceived sync. Beginning with phase synchrony, the strength of this measure’s association with perceived sync may stem from the repetitive nature of this study’s interaction routine. The phase of an interaction is a feature of its periodic or cyclic nature; the more aligned two systems’ phases are, the more rhythmic they can be said to be. Despite the fact that the interaction type in the current 56 study was not regular (in the sense that it did not feature a steady pulse of movements), the repetitive and scripted nature of the Tai-Chi routine likely improved participants’ ability to achieve phase synchrony. In more spontaneous interactions, such as free-flowing conversations, it might be more difficult for the phase synchrony measure to identify rhythmic regularities like this. Accordingly, this measure is often used for scripted or regular interactions (Ouwehand & Peper, 2015; see for an exception Schmidt & Morr, 2012). Next, dynamic mimicry was associated with perceived sync. The negative direction of the correlation indicates that a smaller difference between positions of dyadic partners equates to higher perceived sync ratings. This shows that the position of the limbs in space, not just the timing alone, could be related to perceptions that a dyad is in synchrony. The strength of this measure’s correlation with perceived sync, and the fact that it was the only significant individual predictor of perceived sync in the regression, show that perhaps people look for ‘perfect synchrony’ (timing and form matching; Hale, 2017) when making judgments. In sum, in the type of synchronous interaction shown in this study, the form of the movements evidently played some role in shaping judgments. Moving to Pearson r and MI, these aggregate measures were also associated with perceived sync. For this type of interaction, these measures serve as strong indicators of global synchrony, and are a good starting point for synchrony research involving relatively stationary data. The fact that they showed association with perceived synchrony in a highly complex dataset such as this one points to their robustness in identifying synchrony. However, for researchers interested in (a) non-stationary data types or (b) the dynamic patterns in a dataset, these measures simply will not suffice. As we saw from this study, the leader-follower relationship in a synchronous interaction ties in closely to perceptions of global synchrony, so 57 researchers interested in the LFR would require more dynamic measures. Further, examination of figures produced by dynamic measures, such as the RWTLCC chart, can reveal patterns in the data that may be otherwise missed by aggregate measures. For instance, imagine a dyad who, visibly, was highly coordinated in their movement dynamics, but had one participant leading the other by 5 frames. If correlations were conducted only “on the spot” (i.e., with no inter-subject lag), the result may indicate that there was an absence of synchrony. By looking at the patterns throughout the range of time lags, though, a strong association could be found at beyond the on- the-spot portion of the interaction graph. Regardless of definition of synchrony as simultaneous or simply coordinated, many researchers would likely still be interested in the alignment of this dyad. As such, aggregate measures are advisable, but not sufficient in cases where dynamics are of interest. The second research question asked which measures would correlate with a balance in leader-follower relationship, as measured by the item LFR_E. The balance of LFR correlated with several measures including Pearson r, phase synchrony, lag offset (RWTLCC), and dynamic mimicry. Many synchrony ratings and measures thus seem to be inextricably related to a balance of leadership and followership in an interaction, even though there are types of synchrony in which leader and follower roles are not balanced (i.e., unilaterally entrained synchrony). When leader and follower roles are fixed and there is an accompanying delay in the follower’s movements (i.e., mimicry), LFR is not balanced – though the movements themselves are still somehow coordinated in timing. Future studies should continue to investigate the role of balance in perceptions of synchrony – is it an essential component, or just something that enhances the synchronous experience? 58 The final research question inquired which measures of synchrony would relate to outgroup trust. No synchrony measures were significantly associated with outgroup trust in a bivariate sense. However, I then sorted participant groups by ingroup and outgroup conditions as performed by Tamborini et al. (2018). For dyads in the ingroup condition, Pearson r and DTW were both positively correlated with outgroup trust, whereas for dyads in the outgroup condition, perceived sync negatively correlated with outgroup trust. These findings are inconsistent with Tamborini et al., as in their study they found that for ingroup condition participants, synchrony decreased outgroup trust, whereas for outgroup condition participants, synchrony marginally increased outgroup trust. The distinction between these findings feasibly stems from the difference in measures used between Tamborini et al.’s study and the current Study 2. The former study used a simple video game score of movement similarity as an indicator, whereas the latter used more complex motion-capture-based methods and advanced measures of synchrony. Another possible reason for the inconsistent findings is difference in the nature of the synchronous activities; Tamborini et al. used a one-way interaction between human and computer character, more akin to a mimicked interaction than a synchronous one. The current study used an interaction type that was reciprocal. More research is needed to disentangle how synchrony types and measures can influence social outcomes. Given the relative ubiquity of findings stating that synchrony improves social outcomes, it remains to be seen which types/qualities of synchrony drive these improvements. Is it the simultaneity of movements? Or the rhythmic aspect? Does the form of movements matter at all? These questions cannot be ignored by lumping all interaction types together and dubbing them “synchrony.” The current research brought these issues to the forefront so they may be addressed going forward. Future research would ideally compare these aspects of synchrony in terms of 59 their outcomes; for instance, one might expect perfect synchrony, compared to general synchrony, to produce stronger social effects, given that shared timing and form have been shown to contribute independently to social outcomes. Implications The first major implication of this research is to establish and compare the validity of different measurement techniques for assessing interpersonal synchrony. Many synchrony measures, perhaps predictably, were related to global perceptions of synchrony. This study thus demonstrated the convergent validity between several of the available synchrony measures and perceptions. Another key finding here was that the reliability of synchrony ratings was high, consistent with prior claims that synchrony can be reliably observed without complex measurement techniques (Bernieri, 1988; Bernieri et al., 1994). Notably, however, these findings may be peculiar to the type of interaction used in the present research. I encourage other synchrony researchers to differentiate between interaction types, and to justify their use of measures over others accordingly. Another implication pertains to the methodology used in this study as a recommended protocol for measuring synchrony. Several aspects of this methodology render it an improvement over other extant methods. First, the use of character animation allows researchers to either alter or control the appearance of stimuli, while preserving the fidelity of the real human movements. This balance between control and realism is ideal. Second, the use of full-body motion capture is relatively rare in synchrony research. Many studies in this domain rely on motion energy analysis (Ramseyer & Tschacher, 2011), which leverages changes in video pixels as a measure of broad movement activity shifts. As noted earlier, this technique lacks the precision and granularity of the current method, which locates the movements of specific joints on the human 60 body, which can subsequently be aggregated. Thus, this research is a showcase of the power of combining character animation and motion capture in nonverbal communication research involving observations of movement parameters (see Bente, 2019). Limitations The first limitation of this study was that it did not compare multiple types of synchronous interactions (i.e., did not look at comparisons between the various levels of each synchrony quality). A direct comparison between reciprocal and unilateral interactions, or between regular versus irregular routines, for example, could be useful in further uncovering the utility of the various available measures. Still, this study was a first step toward establishing the need for further research on this topic. By pointing to the need to distinguish synchrony types by their qualities, and by testing one type’s relationship to global synchrony perceptions, this first step leaves to future research the task of comparing more types. A second limitation was the exploratory nature of this research. Strong theoretical background warranting the use of certain measures over others is lacking in the communication science literature, as well as in other domains that study synchrony. As such, addressing research questions instead of hypotheses seemed more appropriate for the current study. As differences among measures and their relationships to qualities of synchrony continue to be discovered, the grounding for theoretical advancement will become more plausible. A third limitation was that this study did not encapsulate all available measures of synchrony. Other methods such as cross-recurrence quantification analysis (Coco & Dale, 2014; Shockley, Butwill, Zbilut, & Webber Jr., 2002) and spectral approaches like the cross-wavelet analysis (Fujiwara & Daibo, 2016; Schmidt, Nie, Franco, & Richardson, 2014) are available, which look at frequencies of events. Conversely, the current study focused on time-domain 61 methods to get a first look at how this range of measures compared with perceptions. Future studies should incorporate alternative measurements to see how they align with the current findings. A final limitation noted here is the relatively small dyadic sample from the original database, especially when divided into ingroup versus outgroup conditions. The conclusions made regarding the relationship between synchrony and outgroup trust should thus be taken with caution. For the current dissertation, the trust variable was of interest to see if synchrony measures related to a social outcome, but in future research, more focus could be dedicated to delving deeper into the theoretical relationship between synchrony and outgroup trust. Conclusion Interpersonal synchrony can be found in different shapes and scopes throughout the natural world. Disentangling how a metronome differs from a human, how a religious ritual differs from a conversation, and how instruction differs from spontaneity are all key questions for synchrony researchers. Understanding these differences enables scholars to better understand which tools are right for the job, reducing confusion and promoting clarity. This research was a first action toward identifying these differences, and showcasing how perceptions of one type of synchrony (complex, reciprocal, repetitive, and intentional) relate to various available measures. Several measures were able to detect synchrony differences that corresponded with variation in general perceptions. Future research may find that more unintentional and spontaneous interactions show different results with respect to synchrony measures. Indeed, we may wonder, does synchrony in a free-flowing conversation even exist in the same vein as two partners rocking back in forth in chairs stably? Answers to such questions must wait for the next wave of synchrony research. 62 APPENDIX 63 APPENDIX Python Script for Synchrony Analyses # -*- coding: utf-8 -*- """ Created on Mon May 18 10:24:32 2020 @author: novot """ # -*- coding: utf-8 -*- """ Created on Sun Apr 19 05:17:47 2020 @author: gabente """ from scipy.stats import zscore from os import listdir import os from os.path import isfile, join import pandas as pd import numpy as np import matplotlib.pyplot as plt import seaborn as sns import scipy.stats as stats from scipy import signal from dtw import accelerated_dtw,multi_dtw,dtw from scipy.signal import hilbert, butter, filtfilt from scipy.fftpack import fft,fftfreq,rfft,irfft,ifft from math import sqrt from numpy import inf import math from PIL import Image import statistics def Save_PDF(): images=[] #fpath='C:/Users/gabente/Desktop/ERIC_data/new_matrices/' onlyfiles = [f for f in listdir(fpath) if isfile(join(fpath, f))and f.endswith('png')] for f in onlyfiles: im = Image.open(fpath+f).convert("RGB") images.append (im) 64 rf=fpath+f os.remove(rf) images[0].save(fpath+'Eric_Plots.PDF', save_all=True, append_images=images[1:]) def resample (s1,ss): nrow=len(s1) j=-1 for name in list(s1): j+=1 for ii in range (0,nrow-ss,ss): x=0 y=0 for k in range (0,ss): x+=s1.iloc[ii+k,j] y+=s1.iloc[ii+k,j] x=x/5 s1.iloc[ii,j]=x drops=int(nrow-(nrow/ss)) s2 = s1.drop(s1.tail(drops).index) return s2 def z_trans(df): d1=(df-df.mean())/df.std(ddof=0) return d1 def butter_bandpass(lowcut, highcut, fs, order=5): nyq = 0.5 * fs low = lowcut / nyq high = highcut / nyq b, a = butter(order, [low, high], btype='band') return b, a def butter_bandpass_filter(data, lowcut, highcut, fs, order=5): b, a = butter_bandpass(lowcut, highcut, fs, order=order) y = filtfilt(b, a, data) return y def low_pass_filter(x,fs,fc): w = fc / (fs / 2) # Normalize the frequency b, a = signal.butter(5, w, 'low') x = signal.filtfilt(b, a,x,axis=0) #x= pd.DataFrame(x) #data=x.flatten()) 65 return x def high_pass_filter(x,fs,fc): w = fc / (fs / 2) # Normalize the frequency b, a = signal.butter(5, w, 'high') x = signal.filtfilt(b, a,x,axis=0) #x= pd.DataFrame(x) #data=x.flatten()) return x def median_filter(df,wi): df.rolling(window=wi,center=True).median() return df def crosscorr(datax, datay, lag, wrap=False): if wrap: shiftedy = datay.shift(lag) shiftedy.iloc[:lag] = datay.iloc[-lag:].values return datax.corr(shiftedy) else: return datax.corr(datay.shift(lag)) def shannon_entropy(A, mode="auto", verbose=False): """ https://stackoverflow.com/questions/42683287/python-numpy-shannon-entropy-array """ A = np.asarray(A) # Determine distribution type if mode == "auto": condition = np.all(A.astype(float) == A.astype(int)) if condition: mode = "discrete" else: mode = "continuous" mode="discrete" if verbose: print(mode, sys.stderr) # Compute shannon entropy pA = A / A.sum() # Remove zeros pA = pA[np.nonzero(pA)[0]] #print (pA) #print (A) if mode == "continuous": return -np.sum(pA*np.log2(A)) if mode == "discrete": 66 return -np.sum(pA*np.log2 (pA)) def mutual_information(df, mode="auto", normalized=False): """ I(X, Y) = H(X) + H(Y) - H(X,Y) https://stackoverflow.com/questions/20491028/optimal-way-to-compute-pairwise-mutual- information-using-numpy """ x=df['DataA'] y=df['DataB'] x= x+ abs(x.min()) y= y+ abs(y.min()) # Determine distribution type if mode == "auto": condition_1 = np.all(x.astype(float) == x.astype(int)) condition_2 = np.all(y.astype(float) == y.astype(int)) if all([condition_1, condition_2]): mode = "discrete" else: mode = "continuous" mode="continuous" H_x = shannon_entropy(x, mode=mode) H_y = shannon_entropy(y, mode=mode) H_xy = shannon_entropy(np.concatenate([x,y]), mode=mode) # Mutual Information I_xy = H_x + H_y - H_xy if normalized: MI= I_xy/np.sqrt(H_x*H_y) else: MI= I_xy MIlist.append(str(round(MI,5))) print (MI) def corr(df): #overall_pearson_r = df.corr().iloc[0,1] #print(f"Pandas computed Pearson r: {overall_pearson_r}") # out: Pandas computed Pearson r: 0.2058774513561943 r, p = stats.pearsonr(df.dropna()['DataA'], df.dropna()['DataB']) #print(f"Scipy computed Pearson r: {r} and p-value: {p}") 67 # out: Scipy computed Pearson r: 0.20587745135619354 and p-value: 3.7902989479463397e- 51 # Compute rolling window synchrony f,ax=plt.subplots(figsize=(14,4)) df.rolling(window=25,center=True).median().plot(ax=ax) ax.set(xlabel='Time',ylabel='Motion') ax.set(title=f"{id} Pearson r = {np.round(r,2)} p= {np.round(p,4)}"); plt.show f.savefig(fpath+ id + '_' +'Pearson.png', bbox_inches = "tight") r = .5*(math.log(1+r)-math.log(1-r)) a=str(round(r,3))+'\t'+str(round(p,4)) corrlist.append (a) def RWS(df): # Rolling Window Synchrony r_window_size = 25 #125 #300 rolling_r = df['DataA'].rolling(window=r_window_size, center=True).corr(df['DataB']) f,ax=plt.subplots(2,1,figsize=(14,6),sharex=True) df.rolling(window=25,center=True).median().plot(ax=ax[0]) ax[0].set(xlabel='Frame',ylabel='Motion') rolling_r.plot(ax=ax[1]) ax[1].set(xlabel='Frame',ylabel='Pearson r') plt.suptitle("Rolling Window Correlation: "+id) plt.show f.savefig(fpath+ id + '_' +'RWS.png', bbox_inches = "tight") def TLCC(df): d1=df['DataA'] d2=df['DataB'] rs = [crosscorr(d1,d2, lag) for lag in range(-int(seconds*fps),int(seconds*fps+1))] offset = np.ceil(len(rs)/2)-np.argmax(rs) #print (len(rs),np.ceil(len(rs)/2) , np.argmax(rs,0)) xs=0 xe=2*int(seconds*fps)+1 f,ax=plt.subplots(figsize=(15,3)) ax.plot(rs) ax.axvline(np.ceil(len(rs)/2),color='k',linestyle='--',label='Center') 68 ax.axvline(np.argmax(rs),color='r',linestyle='--',label='Peak synchrony') ax.set(title=f'Lagged Cross Correlation\n{id}\nOffset = {offset} frames\nS1 leads <> S2 leads',xlim=[xs,xe], xlabel='Offset',ylabel='Pearson r') ax.set_xticks(np.arange(xs,xe, fps)) ax.set_xticklabels(np.arange(-(xe-1)/2, (xe-1)/2+1 , fps)) #ax.set_xticks(np.arange(0, int(seconds*fps)+1, 25.0)) #ax.set_xticklabels(np.arange(-int(seconds*fps),int(seconds*fps), 25)) #ax.set_xticks([0, 50, 100, 151, 201, 251, 301]) #ax.set_xticklabels([-150, -100, -50, 0, 50, 100, 150]); plt.legend() plt.show f.savefig(fpath+ id + '_' +'TLCC.png', bbox_inches = "tight") RSneg=(rs[0:int(seconds*fps)+1]) RSpos=(rs[int(seconds*fps)+1:int(seconds*fps)*2+1]) rsumA=max(RSneg) #(RSneg.max(axis=0)) rsumB=max(RSpos) #(RSpos.max(axis=0)) rs1=str(round(offset,3)) rs2=str(round(abs(offset),3)) rx=(rsumB/rsumA) if rx>1: rx=(rsumA/rsumB) rs3=str(round(abs(rx),3)) rs4=str(round(abs(rsumB-rsumA),3)) out1=rs1+'\t'+rs2+'\t'+rs3+'\t'+rs4 TLCClist.append(out1) def WTLCC(df): # Windowed Time-lagged Cross Correlation no_splits = 100 #int (len(df)/25) #20 #clip = 60 seconds = 1500 frames samples_per_split = int(df.shape[0]/no_splits) #print (samples_per_split) rss=[] for t in range(0, no_splits): t_start=t*samples_per_split t_end=(t+1)*samples_per_split d1 = df['DataA'].iloc[t_start:t_end] 69 d2 = df['DataB'].iloc[t_start:t_end] rs = [crosscorr(d1,d2, lag,wrap=False) for lag in range(- int(seconds*fps),int(seconds*fps+1))] rss.append(rs) rss = pd.DataFrame(rss) f,ax = plt.subplots(figsize=(12,10)) sns.heatmap(rss,cmap='RdBu_r',ax=ax) #xlim=[75, 176], ax.set(title=f'Windowed Time Lagged Cross Correlation: '+id, xlabel='Offset',ylabel='Window epochs') """ ax.set_xticks(np.arange(75, 176, 100/10)) ax.set_xticklabels(np.arange(-5, 6, 1)) ax.set_yticks(np.arange(0,31, 5)) ax.set_yticklabels(np.arange(0, 31,5)) """ plt.show rss2=rss.transpose() f,ax = plt.subplots(figsize=(18,6)) sns.heatmap(rss2,cmap='RdBu_r',ax=ax) #xlim=[-2,31],ylim=[75, 176], #ax.set(title=f'Windowed Time Lagged Cross Correlation: '+id,ylim=[75, 176], ylabel='Offset',xlabel='Window epochs') ax.set(title=f'Windowed Time Lagged Cross Correlation: '+id, ylabel='Offset',xlabel='Time Line (epochs=1 sec)') """ ax.set_yticks(np.arange(75, 176, 100/10)) ax.set_yticklabels(np.arange(-5, 6, 1)) ax.set_xticks(np.arange(-2,34, 5.65)) ax.set_xticklabels(np.arange(0, 71,10)) """ ax.spines['top'].set_visible(True) ax.spines['right'].set_visible(True) ax.spines['bottom'].set_visible(True) ax.spines['left'].set_visible(True) ax.spines['top'].set_linewidth(0.5) ax.spines['right'].set_linewidth(0.5) ax.spines['bottom'].set_linewidth(0.5) ax.spines['left'].set_linewidth(0.5) plt.show f.savefig(fpath+ id + '_' +'TLCC.png', bbox_inches = "tight") 70 def RWTLCC(df): # Rolling window Time-lagged Cross Correlation #seconds = 5 #rapport ==== lag time wondow to right and left #frames per second = 25 Hz window_size = 125 #00 #300 #samples = secs * fps * 2 ==== number of samples for correlation step_size = 5 #frames ==== advancements after each sample t_start = 0 t_end = t_start + window_size #print (df) rss=[] le=len(df) while t_end < le: #1500: #5400: d1= df['DataA'].iloc[t_start:t_end] d2= df['DataB'].iloc[t_start:t_end] rs = [crosscorr(d1,d2, lag, wrap=False) for lag in range(- int(seconds*fps),int(seconds*fps+1))] rss.append(rs) t_start = t_start + step_size t_end = t_end + step_size rss = pd.DataFrame(rss) # either standard display time vertical rss2=rss.transpose() # or transposed time line horizontal rss2=rss2.dropna(axis=0, how='all') f,ax = plt.subplots(figsize=(18,6)) sns.heatmap(rss2,cmap='RdBu_r',ax=ax) ax.set(title=f'Rolling Windowed Time Lagged Cross Correlation: '+id, ylabel='Offset',xlabel='Time Line (epochs = 1/5 sec)') ymin, ymax = ax.get_ylim() xmin, xmax = ax.get_xlim() #print (ymin,ymax,xmin,xmax) #ax.set(title=f'Rolling Windowed Time Lagged Cross Correlation: '+id,ylim=[0,window_size+1], xlim=[0,(le/25)],ylabel='Offset (seconds)',xlabel='Time Line (seconds, epoch=1/5 secs)') ax.set_yticks(np.arange(ymin, ymax-1,-ymin/2)) ax.set_yticklabels(np.arange(-seconds,seconds+1,seconds)) 71 ax.set_xticks(np.arange(xmin,xmax+1, 10.0)) ax.set_xticklabels(np.arange(xmin,xmax+1, 10.0)) #plt.savefig(fpath+id) plt.show() f.savefig(fpath+ id + '_' +'RWTLCC.png', bbox_inches = "tight") rss2=(np.array(rss2).mean(axis=1)) #rss2[0]=rss2[1] #correct for 1. row offset = np.ceil(len(rss2)/2)-np.argmax(rss2) #print (np.ceil(len(rss2)/2),np.argmin(rss2)) f,ax=plt.subplots(figsize=(16,4)) ax.plot(rss2) xlim=[0,251] #or eric [0,101] ax.axvline(np.ceil(len(rss2)/2+1),color='k',linestyle='--',label='Center') ax.axvline(np.argmax(rss2),color='r',linestyle='--',label='Peak Corr') ax.set(title=f'RWTLCC Pearson Mean\n{id}\nOffset = {offset} frames\nS1 leads <> S2 leads', xlabel='Offset',ylabel='Pearson_r') ax.set_xticks(np.arange(0,251, 25.0)) ax.set_xticklabels(np.arange(-125, 126, 25)) plt.legend() plt.show f.savefig(fpath+ id + '_' +'RWTLCCCorr.png', bbox_inches = "tight") RSneg=(rss2[0:int(seconds*fps)+1]) RSpos=(rss2[int(seconds*fps)+1:int(seconds*fps)*2+1]) rsumA=max(RSneg) #(RSneg.max(axis=0)) rsumB=max(RSpos) #(RSpos.max(axis=0)) rs1=str(round(offset,3)) rs2=str(round(abs(offset),3)) rx=(rsumB/rsumA) if rx>1: rx=(rsumA/rsumB) rs3=str(round(abs(rx),3)) rs4=str(round(abs(rsumB-rsumA),3)) out1=rs1+'\t'+rs2+'\t'+rs3+'\t'+rs4 RWTLCClist.append(out1) 72 def DTW(df): #Dynamic Time Warping d1 = df['DataA'] #.interpolate().values d2 = df['DataB'] #.interpolate().values #d, cost_matrix, acc_cost_matrix, path = accelerated_dtw(d1,d2, dist='euclidean') #d, cost_matrix, acc_cost_matrix, path = accelerated_dtw(d1,d2, dist='euclidean') w= inf s=1.0 l2_norm = lambda x, y: (x - y)**2 dist, cost_matrix, acc_cost_matrix, path = dtw(d1, d2, dist=l2_norm,w=w,s=s) f,ax = plt.subplots(figsize=(12,10)) plt.imshow(acc_cost_matrix.T, origin='lower', cmap='jet', interpolation='nearest') plt.plot(path[0], path[1], 'w') plt.xlabel('Subject1') plt.ylabel('Subject2') plt.title(f'DTW Minimum Path with minimum distance: {np.round(dist,2)} : {id}') plt.colorbar(fraction=0.046, pad=0.04) plt.show() f.savefig(fpath+ id + '_' +'fastDTW.png') distlist.append(str(round(dist,3))) def fastDTW(df): #Dynamic Time Warping d1 = df['DataA'].interpolate().values d2 = df['DataB'].interpolate().values d, cost_matrix, acc_cost_matrix, path = accelerated_dtw(d1,d2, dist='euclidean') f,ax = plt.subplots(figsize=(12,10)) plt.imshow(acc_cost_matrix.T, origin='lower', cmap='jet', interpolation='nearest') plt.plot(path[0], path[1], 'w') plt.xlabel('Subject1') plt.ylabel('Subject2') plt.title(f'DTW Path with minimum distance: {np.round(d,2)}: {id}') plt.colorbar(fraction=0.046, pad=0.04) 73 xx=str(round(100* ( abs(len(d1)-d) /len(d1) ),3)) dd=str(round(d,3)) distlist.append(dd+'\t'+xx) plt.show() f.savefig(fpath+ id + '_' +'fastDTW.png') def matDTW(s1,s2,Typ): """ 2 dummy arrays original 1D vectors :param array x: now len(s1) :param array y: now len(s2) 2 3 DoF arrays for distance measures :param array s1 N1*M*dims array :param array s2: N2*M*dims array :param func dist: distance used as cost measure :param int warp: how many shifts are computed. :param int w: window size limiting the maximal distance between indices of matched entries |i,j|. :param float s: weight applied on off-diagonal moves of the path. As s gets larger, the warping path is increasingly biased towards the diagonal Returns the minimum distance, the cost matrix, the accumulated cost matrix, and the wrap path. """ ste=5 nrow=len(s1) #print (nrow) df1=[] df2=[] new_s1=[] new_s2=[] #news2=[] global fpath k=0 #resample 25 Hz to 5 Hz either mean of 5 (postion data) or sum of 5 (dynamic data) if Typ=='pos': for ii in range (0,int(nrow/ste),ste): st=ii*ste en=st+ste df1=s1[st:en] df2=s2[st:en] k+=1 new_s1.append (df1.mean(axis=0)) new_s2.append (df2.mean(axis=0)) #print(k,df1.mean(axis=0)) 74 else: for ii in range (0,int(nrow/ste),ste): st=ii*ste en=st+ste df1=s1[st:en] df2=s2[st:en] k+=1 new_s1.append (df1.sum(axis=0)) new_s2.append (df2.sum(axis=0)) s1 = pd.DataFrame(new_s1) s2 = pd.DataFrame(new_s2) x=len(s1) y=len(s2) w=inf s=1.0 dist, cost_matrix, acc_cost_matrix, path = multi_dtw(x, y,s1,s2,Typ, warp=1,w=w,s=s) #print (round(dist,3)) f,ax = plt.subplots(figsize=(12,10)) plt.imshow(acc_cost_matrix.T, origin='lower', cmap='jet', interpolation='nearest') plt.plot(path[0], path[1], 'w',linewidth=3) plt.xlabel('Subject1') plt.ylabel('Subject2') plt.title(f'DTW Path with minimum distance: {np.round(dist,2)}: {id}') plt.title('Matrix-DTW ('+Typ+'): '+id+' (dist=)'+str(round(dist,3))) plt.colorbar(fraction=0.046, pad=0.04) #ymin, ymax = ax.get_ylim() #xmin, xmax = ax.get_xlim() #plt.plot(x, label='x') #plt.plot(y, label='y') distlist.append(str(round(dist,3))) plt.show() f.savefig(fpath+ id +'_' +'MatDTW_'+Typ+'.png', bbox_inches = "tight") def PhS(dfRaw): #Phase Synchrony y1 = dfRaw['DataA'].interpolate().values 75 y2 = dfRaw['DataB'].interpolate().values #y1 = butter_bandpass_filter(y1,lowcut=lowcut,highcut=highcut,fs=fs,order=order) #y2 = butter_bandpass_filter(y2,lowcut=lowcut,highcut=highcut,fs=fs,order=order) al1 = np.angle(hilbert(y1),deg=False) al2 = np.angle(hilbert(y2),deg=False) phase_synchrony = 1-np.sin(np.abs(al1-al2)/2) N = len(al1) # Plot results f,ax = plt.subplots(3,1,figsize=(14,7),sharex=True) ax[0].plot(y1,color='r',label='y1') ax[0].plot(y2,color='b',label='y2') ax[0].legend(bbox_to_anchor=(0., 1.02, 1., .102),ncol=2) ax[0].set(xlim=[0,N], title='Filtered Timeseries Data') ax[1].plot(al1,color='r') ax[1].plot(al2,color='b') ax[1].set(ylabel='Angle',title='Angle at each Timepoint',xlim=[0,N]) phase_synchrony = 1-np.sin(np.abs(al1-al2)/2) ax[2].plot(phase_synchrony) ax[2].set(ylim=[0,1.1],xlim=[0,N],title='Instantaneous Phase Synchrony: '+id,xlabel='Time',ylabel='Phase Synchrony') plt.tight_layout() f.savefig(fpath+ id + '_' +'PhS.png') print(statistics.mean(phase_synchrony)) def TLMimDyn(s1,s2): #(s1,s2): swapped persons to be compatible with RWTLCC fps=25 lag=int(seconds*fps) N=len (s2) #1500 stp=5 lst=[] diff_all=[] for i in range(lag,N-lag,stp): #step forward d1= s2.iloc[i] lst=[] dda=0 for j in range (i-lag+1,i+lag): d2=s1.iloc[j] dda=0 for k in range(14): #difference in dyad of Eukledian diffrences between all joints abs (Xtn - Xtn+1) 76 x1=d1.iloc[k] x2=d2.iloc[k] dd=sqrt((x1-x2)**2) dda+=dd lst.append(dda) diff_all.append (lst) rss = pd.DataFrame(diff_all) # either standard display time vertical rss2=rss.transpose() #rss2=(rss2-rss2.mean())/rss2.std(ddof=0) #rss2 = (rss2 - rss2.mean())/rss2.std(ddof=0)*-1 f,ax = plt.subplots(figsize=(16,4)) #vmin, vmax = 0, 10 #sns.heatmap(rss2,cmap='RdBu_r',ax=ax,center=(vmin + vmax) / 2., vmax=vmax) stdev=rss2.std(ddof=0) stdev=2*(stdev.std(ddof=0)) m=rss2.mean() m=m.mean() vmin=m-2*stdev vmax=m+2*stdev sns.heatmap(rss2,cmap='RdBu_r',ax=ax,vmin=vmin,vmax=vmax) ymin, ymax = ax.get_ylim() xmin, xmax = ax.get_xlim() #print (ymin,ymax,xmin,xmax) #ax.set_yticks(np.arange(100,-1,-(100/4))) #ymin, ymax,(abs(ymax-ymin)+1)/5)) #ax.set_yticklabels(np.arange(-2, 2.1, 1)) ax.set_yticks(np.arange(0, 2*lag+1,int((2*lag+1)/4))) ax.set_yticklabels(np.arange(-2, 2.1,1)) ax.set_xticks(np.arange(xmin,xmax+1, 25.0)) ax.set_xticklabels(np.arange(xmin,xmax+1, 5)) #ax.set(title=f'Rolling Windowed Time Lagged Cross Correlation',xlim=[0,301], xlabel='Offset',ylabel='Epochs (1 sec)') ax.set(title=f'Time-Lagged Movement Difference: '+id,ylabel='Offset (seconds)',xlabel='Time Line (seconds)') #plt.savefig(path+id) plt.show() 77 f.savefig(fpath+ id + '_' +'MimDyn.png', bbox_inches = "tight") rss2=(np.array(rss2).mean(axis=1)) offset = np.ceil(len(rss2)/2)-np.argmin(rss2) f,ax=plt.subplots(figsize=(16,4)) ax.plot(rss2) ax.axvline(np.ceil(len(rss2)/2),color='k',linestyle='--',label='Center') ax.axvline(np.argmin(rss2),color='r',linestyle='--',label='Peak synchrony') ax.set(title=f'Lag Distribution Synchrony (motion)\n{id}\nOffset = {offset} frames\nS1 leads <> S2 leads',xlim=[0,101], xlabel='Offset',ylabel='Distance') ax.set_xticks(np.arange(0, 101, 5.0)) ax.set_xticklabels(np.arange(-50, 51, 5)) #ax.set_xticks(np.arange(0, 251, 25.0)) #ax.set_xticklabels(np.arange(-125, 126, 25)) plt.legend() plt.show f.savefig(fpath+ id + '_' +'MimPos2.png', bbox_inches = "tight") RSneg=(rss2[0:int(seconds*fps)+1]) RSpos=(rss2[int(seconds*fps)+1:int(seconds*fps)*2+1]) rsumA=max(RSneg) #(RSneg.max(axis=0)) rsumB=max(RSpos) #(RSpos.max(axis=0)) rs1=str(round(offset,3)) rs2=str(round(abs(offset),3)) rx=(rsumB/rsumA) if rx>1: rx=(rsumA/rsumB) rs3=str(round(abs(rx),3)) rs4=str(round(abs(rsumB-rsumA),3)) out1=rs1+'\t'+rs2+'\t'+rs3+'\t'+rs4 mimdynlist.append(out1) def TLMimPos(s1,s2): #(s1,s2): swapped persons to be compatible with RWTLCC seconds=3 lag=int(seconds*fps) N=len(s1) #1500 78 stp=75 lst=[] diff_all=[] for i in range(lag,N-lag,stp): #step forward d1= s1.iloc[i] lst=[] for j in range (i-lag,i+lag): d2=s2.iloc[j] dda=0 for k in range(14): p=k*3 x1=d1.iloc[p] y1=d1.iloc[p+1] z1=d1.iloc[p+2] x2=d2.iloc[p] y2=d2.iloc[p+1] z2=d2.iloc[p+2] dd=sqrt((x1-x2)**2+(y1-y2)**2+(z1-z2)**2) dda+=dd lst.append(dda) diff_all.append (lst) rss = pd.DataFrame(diff_all) # either standard display time vertical rss2=rss.transpose() rss2=rss2.dropna(axis=0,how='all') #print (N,(rss2)) f,ax = plt.subplots(figsize=(16,4)) #vmin, vmax = 0, 500 sns.heatmap(rss2,cmap='RdBu_r',ax=ax) #,center=(vmin + vmax) / 2., vmax=vmax) ymin, ymax = ax.get_ylim() xmin, xmax = ax.get_xlim() #print (ymin,ymax,xmin,xmax) ax.set(title=f'Time-Lagged Posture Difference: '+id,ylabel='Lag (seconds)',xlabel='Time Line (seconds)') """ ax.set_yticks(np.arange(0, 251, 128)) #2*lag+1,int((2*lag+1)/2))) ax.set_yticklabels(np.arange(-5, 5.1,5)) ax.set_xticks(np.arange(xmin,xmax+1, 25.0)) ax.set_xticklabels(np.arange(xmin,xmax+1, 25.0)) """ plt.show() f.savefig(fpath+ id + '_' +'MimPos1.png', bbox_inches = "tight") 79 rss2=(np.array(rss2).mean(axis=1)) rss2[0]=rss2[1] #correct for 1. row #rss2[0]=rss2[1] #correct for 1. row offset = np.ceil(len(rss2)/2)-np.argmin(rss2) #print (rss2) #print (np.ceil(len(rss2)/2),np.argmin(rss2)) f,ax=plt.subplots(figsize=(16,4)) ax.plot(rss2) ax.axvline(np.ceil(len(rss2)/2),color='k',linestyle='--',label='Center') ax.axvline(np.argmin(rss2),color='r',linestyle='--',label='Peak mimicry') ax.set(title=f'Lag Distribution Mimicry (position) \n{id}\nOffset = {offset} frames\nS1 leads <> S2 leads', xlabel='Offset',ylabel='Distance') #ax.set_xticks(np.arange(0, 256, 25.0)) #ax.set_xticklabels(np.arange(-125, 126, 25)) plt.legend() plt.show f.savefig(fpath+ id + '_' +'MimPos2.png', bbox_inches = "tight") RSneg=(rss2[0:int(seconds*fps)+1]) RSpos=(rss2[int(seconds*fps)+1:int(seconds*fps)*2+1]) rsumA=max(RSneg) #(RSneg.max(axis=0)) rsumB=max(RSpos) #(RSpos.max(axis=0)) rs1=str(round(offset,3)) rs2=str(round(abs(offset),3)) rx=(rsumB/rsumA) if rx>1: rx=(rsumA/rsumB) rs3=str(round(abs(rx),3)) rs4=str(round(abs(rsumB-rsumA),3)) out1=rs1+'\t'+rs2+'\t'+rs3+'\t'+rs4 mimiclist.append(out1) def show_it(df): m1=np.amax(df['DataA']) m2=np.amax(df['DataB']) m=max(m1,m2) f,ax = plt.subplots(figsize=(16,4)) #plt.title(id+' RawData (z-transformed)') 80 plt.plot( df['DataA'], label='Person A') plt.plot( df['DataB'], label='Person B') plt.ylim((-m*1.2,m*1.2)) #plt.yticks([-1,1]) plt.show() #main program id='' fpath='' fileMovA=[] fileMovB=[] filePosA=[] filePosB=[] distlist=[] corrlist=[] TLCClist=[] gazelist=[] mimiclist=[] mimdynlist=[] MIlist=[] RWTLCClist=[] #study ID: #case_ID='Rapport' case_ID = 'Eric' #data type: mode='rawmat' #mode='vectors' #mode='SRL' #headers for parameter outputs RWTLCClist.append('offset_signed'+'\t'+'off_absolut'+'\t'+'AB_Proportion'+'\t'+'AB_Difference' ) mimiclist.append('offset_signed'+'\t'+'off_absolut'+'\t'+'AB_Proportion'+'\t'+'AB_Difference') mimdynlist.append('offset_signed'+'\t'+'off_absolut'+'\t'+'AB_Proportion'+'\t'+'AB_Difference') TLCClist.append('offset_signed'+'\t'+'off_absolut'+'\t'+'AB_Proportion'+'\t'+'AB_Difference') distlist.append('DTW_dist'+'\t'+'Dist_Percent') corrlist.append('Pearson_r'+'\t'+'p_value') gazelist.append ('direct_perc') 81 org_samp_rate=25 fs=org_samp_rate new_samp_rate= 25 re_samp=int(org_samp_rate/new_samp_rate) lowcut = .05 #.01 #band pass low highcut = .5 #bad pass high order = 1 fc_low = .5 # Cut-off frequency of the low pass filter fc_high = .3 # Cut-off frequency of the high pass filter mF=50 # median filter constant seconds=5 #2 # size off lag for lagged analyses fps = new_samp_rate if mode== 'rawmat': #Eric Data if case_ID=='Eric': fpath='C:/Motion Data/' onlyfiles = [f for f in listdir(fpath) if isfile(join(fpath, f))and f.endswith('0_mov_A.txt')] elif case_ID=='Rapport': #rapport data fpath='C:/Users/gabente/Desktop/Rapport/SnapData/' onlyfiles = [f for f in listdir(fpath) if isfile(join(fpath, f))and f.endswith('Mov_A.txt')] for l in onlyfiles: ll=l.upper() fileMovA.append(fpath+ll) fileMovB.append(fpath+ll.replace('A','B')) filePosA.append(fpath+ll.replace('MOV_A','POS_A')) filePosB.append(fpath+ll.replace('MOV_A','POS_B')) l=-1 for ff in onlyfiles: if case_ID=='Eric': id="DYAD"+ff[0:2] #Eric elif case_ID=='Rapport': id="DYAD"+ff[5:7] #Rapport l+=1 #Read movemnt data 82 fid1=fileMovA[l] fid2=fileMovB[l] d1=[] d2=[] hvalid=['Chest','L_Arm','L_ForeArm','L_Hand','R_Arm','R_ForeArm','R_Hand','Head','R_UpLeg ','R_Leg','R_Foot','L_UpLeg','L_Leg','L_Foot'] if case_ID=='Rapport': hvalid=list(map(lambda x: x.upper(), hvalid)) #hips if in data are skipped as they set to zero in the snap mode dfMovA = pd.read_csv(fid1,sep='\t',usecols=lambda column : column in hvalid) dfMovB = pd.read_csv(fid2,sep='\t',usecols=lambda column : column in hvalid) #Read postion data fid1=filePosA[l] fid2=filePosB[l] dfPosA = pd.read_csv(fid1,sep='\t',usecols=lambda column : column not in ['HIPS:TX', 'HIPS:TY', 'HIPS:TZ']) dfPosB = pd.read_csv(fid2,sep='\t',usecols=lambda column : column not in ['HIPS:TX', 'HIPS:TY', 'HIPS:TZ']) #z-transform motion data in all joints separetly cols = list(dfMovA.columns) #identical for A and B for col in cols: dfMovA[col] = (dfMovA[col] - dfMovA[col].mean())/dfMovA[col].std(ddof=0) dfMovB[col] = (dfMovB[col] - dfMovB[col].mean())/dfMovB[col].std(ddof=0) #correct first row of data dfMovA.iloc[0:] = dfMovA.iloc[1:] dfMovB.iloc[0:] = dfMovB.iloc[1:] dfPosA.iloc[0:] = dfPosA.iloc[1:] dfPosB.iloc[0:] = dfPosB.iloc[1:] #aggregate motion data across joints for A and B d1= dfMovA.sum(axis=1) d2= dfMovB.sum(axis=1) #combine A and B in one file dfx = pd.concat([d1, d2], axis=1, sort=False) #replace variable names names=[] 83 names.append('DataA') names.append('DataB') dfx.columns = dfx.columns[:0].tolist() + names # interpolate missing data and generate unfiltered raw data matrix in dfRaw dfxRaw = dfx #dfx.interpolate() #calculate low pass butterworth filtered data in df if new_samp_rate