I began this study with two assumptions. Assumption 1: Study quality matters. If the means by which researchers design, carry out, and report on their studies lack in rigor or transparency, theory and practice are likely to be misguided or at least decelerated. Assumption 2 is an implication of Assumption 1: Quality should be measured rather than assumed. Although peer reviews and researcher training in second language acquisition (SLA) are generally considered to be acceptable and thorough, there is very little evidence of the extent of methodological rigor, consistency, and transparency across second language (L2) research.Beyond these assumptions, this study drew from previous research in several different fields. Central to this paper was the research on study quality which has received considerable attention in the context of research synthetic and meta-analytic methods. Because of the shared historical and methodological tradition of SLA and psychology (e.g., Gass, 1993), I consulted as well the American Psychological Association's guidelines for research when developing the instrument used in this study. Much of the empirical motivation for this project also came from within the field of SLA. Previous reviews have raised concerns about methodological practices across a number of subdomains, warranting further and more comprehensive investigation.The first two of four questions posed by the present study asked about the use of study designs and analyses (RQ1) and reporting practices (RQ2) in L2 research. My third question sought to measure the relationship between research practices and outcomes (i.e., effect sizes) found in L2 research (see Plonsky & Gass, 2011). And research question four asked whether and to what extent research practices and outcomes in SLA have changed in recent years. The purpose underlying these questions and the study more generally was not only to better understand conventions in the field but to inform future research practices as well.In order to answer these questions and meet the study's larger purpose, a representative sample of L2 research published in two L2 journals from 1990 to 2010 was collected. Using research synthetic techniques, I surveyed the sample of studies, 606 in total, using a modified version of the instrument used by Plonsky and Gass (2011) in their investigation of study quality in the interactionist tradition of SLA. The coding scheme was designed to extract information related to study identification, design features, analyses, reporting practices, and effect sizes. Descriptive statistics were then calculated for study features to answer the research questions.The overall results of this study point to a number of systematic strengths as well as many flaws across the corpus of L2 research. Of particular concern are incomplete and inconsistent reporting practices (e.g., means without standard deviations) and low statistical power, among other issues. Somewhat surprisingly and in contrast to previous findings (e.g., Plonsky, in press), there was very little evidence of a relationship between study quality and effect sizes, a finding that may reflect the broad substantive scope of the study and field. Finally, comparing research practices over the 1990s and 2000s, I found substantial improvements in almost all categories.The discussion situates the results in terms of reviews from SLA and other fields (e.g., education; Skidmore & Thompson, 2010), shedding light on the methodological and analytical trends and trajectories observed. Based on the findings of the study, I make pointed suggestions for methodological reforms to be enacted by institutions such as the American Association for Applied Linguistics and individuals (e.g., independent researchers, journal editors).
Read
