FAMILY MEMBER AND INTIMATE PARTNER EFFORTS TO REDUCE FIREARM ACCESS WHEN THEIR LOVED ONE IS AT RISK OF HARMING THEMSELVES OR OTHERS By Jennifer Paruk A DISSERTATION Submitted to Michigan State University in partial fulfillment of the requirements for the degree of Criminal Justice – Doctor of Philosophy 2023 ABSTRACT Family members and intimate partners [families] are uniquely positioned to reduce firearm access when their adult loved one is at risk of harming themselves or others, thereby preventing firearm injury. Yet we know little on how or when families are doing so, and if these efforts are safe and/or legal. To identify how and when families were reducing firearm access, a content analysis was conducted using police reports from Extreme Risk Protection Order [ERPO] petitions. These police reports were from the events that inspired police to file for the ERPO [precipitating event or PE] against the at-risk individual [respondent]. This content analysis identified and characterized the ways families tried to reduce the respondent’s firearm access at three time-points: before the PE, during the PE, and after police arrived at the PE. Next, supervised machine learning determined which respondent and PE characteristics were the most influential in predicting whether families reduced firearm access after police arrived at the PE. The content analysis identified that families reduced the respondent’s firearm access in numerous ways. Some families tried to prevent the respondent from accessing a firearm before the PE occurred, with a small number of police reports (2.54%) indicating such efforts. Fifteen percent of reports included information that families tried to keep firearms away from the respondent during the PE: Families hid firearms, sometimes in places where others could access them; asked the respondent for the firearms; and used physical force to try to take firearms. Just over a quarter (26.51%) of reports included information that families reduced firearm access after police officers arrived at the PE. Families gave officers consent and access to search for and remove firearms, asked police for help removing firearms, gave police firearms, and told officers they would continue to prevent firearm access after officers left the scene, including by temporarily storing firearms outside of the home. In the supervised machine learning model, two of the most influential variables predicting whether families reduced firearm access after police arrived at the PE were if the respondent was taken for an involuntary examination (which allows officers to remove firearms) and if officers told someone at the scene about ERPOs. The findings highlight that when families tried to prevent their loved ones from accessing firearms, they sometimes put themselves and others at risk of injury. Additionally, the temporary transfer of firearms to families outside the respondent’s home without a background check— while legal in Florida—may not be legal in other states with universal background check laws. The supervised machine learning models indicate which respondent and incident characteristics should be explored in more detail in future research. Finally, many of the supervised machine learning models had low sensitivity, indicating that indicating that there are likely other influential variables in predicting families reducing firearm access after police arrive that were not included in this study. ACKNOWLEDGEMENTS My sincere gratitude to so many: Dr. April Zeoli, thank you for helping me to figure out how to turn my ERPO coding observations into a research study, continuing to review drafts and supporting me in the dissertation process after moving institutions, and patiently helping me become a more succinct writer. This dissertation is infinitely better because of your guidance. Dr. Cait Cavanagh, thank you for your continued support during this process and helping students give me feedback. Dr. Karen Holt, thank you for your patience in so many email threads to get this dissertation grant funding going. Dr. Scott Wolfe, thank you for your expert feedback and for your support when I was having a tough dissertation day. Dr. Annette Christy, thank you for helping with data access. Your in-depth knowledge of the data and caring spirit helped me during the early days of this dissertation process. Dr. Wenjuan Ma, I never would have thought of this analytic plan without your interest. Thank you for patiently answering all my questions! To all my families (my best buddies Jake and Humboldt, family, extended family, friends, working groups, PhD cohort, walking buddies, lab members, and my broader MSU community), thank you for the advice, encouragement, and sunshine. iv TABLE OF CONTENTS INTRODUCTION .......................................................................................................................... 1 CHAPTER 1: LITERATURE REVIEW ........................................................................................ 5 CHAPTER 2: METHODOLOGY ................................................................................................ 22 CHAPTER 3: CONTENT ANALYSIS FINDINGS .................................................................... 41 CHAPTER 4: SUPERVISED MACHINE LEARNING RESULTS............................................ 47 CHAPTER 5: SUMMARY, IMPLICATIONS, AND FUTURE SCHOLARSHIP ..................... 55 BIBLIOGRAPHY ......................................................................................................................... 70 APPENDIX ................................................................................................................................... 82 v INTRODUCTION Firearms play a prominent role in both homicide and suicide in the United States, so reducing firearm access for those at risk of harming themselves or others is one way to prevent firearm injury. Firearms are more lethal in domestic assaults and suicide attempts than other weapons or methods (Conner et al., 2019; Saltzman et al., 1992), and in 2020, 78.9% of all homicides and just over half of all suicides (52.8%) were committed with firearms (CDC, 2023). Firearm access can be deadly: violent intimate partners’ firearm access is associated with an increased risk of intimate partner homicide (Campbell et al., 2003) and firearm access is associated with increased risk of suicide (Anglemyer et al., 2014a). Therefore, both those at risk of suicide and interpersonal violence share a common intervention point: when someone is at risk of harming either themselves or others, reducing firearm access is one method to prevent injury. Family members and intimate partners [families] are uniquely positioned to prevent someone who is at risk of harming themselves or others from accessing firearms. At-risk individuals often indicate or signal this intention to others beforehand (Bagge et al., 2022; Silver, 2020). Because families spend a substantial amount of time with their loved ones, they might see these suicide warning signs, such as their loved one making a suicide plan (Rudd, 2008). Families might also be victims of the individual’s threats or uses of violence. Additionally, families may be firearm owners themselves or know how and where firearms in the home are stored (Okoro et al., 2005; Sorenson & Wiebe, 2004). This knowledge of firearm storage, in addition to the warning signs, threats, or uses of violence, gives families the opportunity to intervene and prevent injury. 1 To decrease the risk of lethality, families are sometimes advised to prevent the at-risk person from accessing firearms. For example, doctors conducting lethal means safety counseling for suicidal patients and their families advise that family members store their firearms away from the at-risk person (Barber & Miller, 2014). Similarly, intimate partner violence [IPV] victims are sometimes recommended to file a domestic violence restraining order [DVRO] or call law enforcement to prevent their abusive partners from accessing firearms (Battered Women’s Justice Project, n.d.; Lynch & Logan, 2021). Families in some states are also allowed to file Extreme Risk Protection Orders [ERPO], which are civil orders that prevent someone at risk of harming themselves or others from firearm access for a temporary period (e.g., a year). However, research on families’ efforts to reduce firearm access when their loved one is at risk of harming themselves or others is limited to studies of parents of children, civil court interventions, and focus group discussions broadly centered on firearms. For example, researchers have examined parents and caregivers’ safe storage behavior after healthcare interventions (e.g., Miller et al., 2020) and civil court ERPO and DVRO petitions filed by family members and intimate partners (e.g., (Webster et al., 2010; Zeoli et al., 2021). Researchers have also broadly discussed firearms and reducing firearm access in focus groups with IPV victims (Lynch & Logan, 2018); Veterans Affairs’ providers, clients, and their families (Walters et al., 2012); and older adults, their families, and those who work with older adults (Betz et al., 2023). Even though families are uniquely positioned to reduce firearm access and are sometimes advised to do so, we know very little about how they specifically try to reduce firearm access— outside of court intervention—when their adult loved one is at risk of harming themselves or others. By identifying how and when families try to reduce firearm access when their adult loved one is at risk of harming themselves or others, interventions (e.g., educational campaigns) can 2 then be crafted to increase the safety and efficacy of these efforts in preventing firearm injury. Additionally, any legal concerns around families’ efforts to reduce firearm access can be further studied and addressed. To study families’ efforts to reduce firearm access when their loved one is at risk of harming themselves or others, a theoretical framework is needed to suggest when these efforts would occur. The theoretical frameworks of both the Theory of Planned Behavior and police legitimacy suggest that one place to study families’ efforts is when police arrive at the scene of an at-risk individual. The Theory of Planned Behavior (TPB) proposes that someone’s likelihood of engaging in a behavior is partially based on the views of others around them (e.g., friends, supervisors) and their perceived ability to engage in the behavior (Ajzen, 2020). Law enforcement officers are consistently seen as credible messengers around firearm safe storage (Aitken et al., 2020; Crifasi, Doucette, et al., 2018a), are often willing to temporarily store firearms when someone in the home is at-risk of harming themselves (Runyan et al., 2017), and depending on the incident and state law, have the authority to remove firearms from the scene. Therefore, TPB suggests that officers’ ability to remove and temporarily store firearms might give families the perceived social support and ability to reduce their loved one’s firearm access. Studies of police legitimacy also suggest that one place to study families’ efforts reducing firearm access is when police arrive at the scene. Police legitimacy is the belief that officers are legitimate members of authority who can take actions to maintain social order, and civilians therefore assist officers for that reason (Tyler, 2004). It is possible that families trust officers and feel obliged to assist police in removing firearms when someone is at risk of harming themselves or others. Therefore, both theoretical frameworks suggest that one place to study families’ efforts to reduce firearm access is when police arrive at a scene of an at-risk individual. 3 Research Aims To explore families’ efforts to reduce firearm access when their loved one is at risk of harming themselves or others, this study examined police officer reports that were included in ERPO cases. These police reports were from the incident that inspired law enforcement to file for the ERPO [the precipitating event, or PE] for the respondent. The objectives of this dissertation were two-fold: (1) identify and characterize families’ efforts to reduce firearm access at three time points: before the PE, during the PE, and after police arrived at the PE, and (2) identify what respondent and PE characteristics are important in predicting families’ efforts to reduce firearm access after police arrived at the PE. 4 CHAPTER 1: LITERATURE REVIEW This chapter describes the epidemiology of firearm interpersonal violence (including intimate partner violence [IPV] and other interpersonal violence) and firearm suicide. This chapter also describes recommendations given to families to decrease the risk of firearm injury and the limited research on families’ efforts to prevent their loved ones from accessing firearms when there is a risk of interpersonal violence and/or suicide. Finally, this chapter illustrates how more research on families’ efforts to reduce firearm access is needed to identify how and when families are reducing firearm access, and if these efforts are safe, legal, and effective. Firearms and Interpersonal Violence Firearms and IPV Nonfatal Firearm IPV In violent relationships, an abuser’s access to and use of firearms can cause psychological and physical harm. In violent relationships where the abusive partner has a firearm, a majority of victims (67%) report that their violent partner having a firearm makes them feel less safe (Sorenson & Wiebe, 2004). Even the presence of a firearm in the home can make IPV victims feel less safe, knowing that a firearm could be used against them at any time (Lynch & Logan, 2018; Sorenson & Wiebe, 2004). Violent intimate partners use firearms to scare, threaten (e.g., threaten to kill), and harm their partners (Sorenson & Wiebe, 2004). Violent intimate partners also use firearms to control their partners’ behavior (Lynch & Logan, 2018; Sorenson, 2017). IPV victims report that firearms have a unique dangerousness compared to other weapons, and when firearms are used in an incident, intimate partner victims are more likely to be frightened than if another weapon had been used (Sorenson, 2017). Approximately 4.5 million women alive today in the U.S. have been threatened by an intimate partner with a firearm and approximately 5 one million have been shot or shot at by an intimate partner (Sorenson & Schut, 2018), and IPV victims report that this nonfatal firearm abuse has resulted in physical injury that required medical care, feeling fearful and concerned for safety, having nightmares or intrusive thoughts about the gun violence, and missing work or school (Adhia et al., 2021). Non-fatal firearm IPV also affects the victim’s family and acquaintances. Violent intimate partners sometimes threaten their intimate partners by threatening to shoot others (e.g., victim’s children or friends) (Adhia et al., 2021; Logan et al., 2021; Sorenson & Wiebe, 2004). In a study of IPV perpetrators who threatened to shoot others, 34.4% threatened to shoot children, and 41.6% pointed the gun at or directly threatened the victim’s family or friends (Logan et al., 2021). An intimate partner’s access to a firearm is not associated with committing non-fatal IPV, however, in violent relationships, a violent intimate partner’s access to firearms is associated with an increase in the severity of non-fatal IPV (Zeoli et al., 2016). For example, in one study, pregnant women whose violent partners had firearms reported significantly higher levels of violence (McFarlane et al., 1998). However, it is unclear if this relationship between access to firearms and non-fatal abuse severity is due to those who are more likely to commit severe non- fatal violence being more likely to own firearms (Zeoli et al., 2016). Additionally, analyses of police IPV reports show when a firearm was used in an incident, the victim was less likely to have visible injuries, suggesting that abusive partners use firearms to intimidate and coerce their partners (Sorenson, 2017). Firearm Intimate Partner Homicide [IPH] In violent relationships, a perpetrator’s firearm access is associated with an increase in IPH (Zeoli et al., 2016). In relationships with IPV, a violent male intimate partner’s access to 6 firearms is associated with a five times greater risk of killing their female partner (Campbell et al., 2003). The use of firearms in domestic assaults is more likely to result in death than the use of other weapons (Saltzman et al., 1992), and the most common weapon used in IPHs is a firearm (Fridel & Fox, 2019). The use of a firearm in IPH is also associated with additional homicide victims. When a male kills his intimate partner with a firearm, he more likely to kill additional victims than if he had used a different weapon (Smucker et al., 2018). In IPHs, 20% of homicide victims are those who were present at the scene or who tried to intervene (e.g., children and responding police officers), and a majority (70%) of these additional homicide victims are killed with a firearm (Smith et al., 2014). Indeed, twenty percent of all child homicides are related to IPV, and these IPV-related child homicides are more likely to have involved a gun compared to non-IPV-related homicides (Adhia et al., 2019). Firearm IPH and Suicide Intimate partner homicides committed by men, suicide, and firearms are closely linked. In all homicides, the perpetrator dies by suicide in less than 5% of cases (Smucker et al., 2018); however, in intimate partner homicides committed by men, the male perpetrator then dies by suicide in approximately 27-32% of cases (Campbell et al., 2007; Smucker et al., 2018). These male suicides that follow IPH are more likely when the perpetrator uses a firearm to kill his intimate partner: 87% of IPH-suicides are committed by men with firearms (Smucker et al., 2018). Women commit a minority of intimate partner homicides (21.5-23%) (Smucker et al., 2018; Yousuf et al., 2017), and when they do, they rarely die by suicide afterwards (Campbell et al., 2007; Smucker et al., 2018). Of men who commit IPH, previous threats of suicide predict IPH-suicides (Koziol-McLain et al., 2006). 7 Preventing Firearm Access There is evidence that laws that prevent IPV perpetrators from purchasing or possessing firearms are associated with decreased rates of IPH. Federal law orders that those who are under a final DVRO (and have a certain relationship status with the victim) are prohibited from firearm purchase and possession (18 U.S.C. § 922(g)(8)). Many states also extend this DVRO firearm prohibition to additional relationships (e.g., ex-dating partners), temporary DVROs, and include relinquishment provisions that specify how the firearm should be relinquished, and the states that have extended these DVRO firearm prohibitions in these ways are associated with a decrease in IPH rates (Díez et al., 2017; Zeoli et al., 2018). Federal law also prohibits those convicted of a domestic violence misdemeanor from firearm purchase and possession (18 U.S.C. § 922(g)(9)), and this federal law is associated with a decrease in the homicide rates of female intimate partners, male children, and other family members (Raissian, 2016; Zeoli et al., 2018). State laws that include relinquishment provisions for a prohibition for misdemeanor domestic violence convictions are associated with decreased rates of homicide in pregnant and postpartum women (Wallace et al., 2021). Some states also have laws that prohibit those convicted of violent misdemeanors from firearm possession. Although these laws are not designed to specifically target IPV perpetrators, many IPV perpetrators have previous histories of non-IPV violence, and therefore are prohibited from firearm possession under these laws (Zeoli et al., 2018). These state laws that prohibit those convicted of violent misdemeanors from firearm possession are associated with a decreased risk of IPH (Zeoli et al., 2018). 8 Recommendations for IPV Victims around Firearms. The National Hotline for Domestic Violence and the Battered Women’s Justice Project (a national collective of policy and practice organizations) give IPV victims suggestions for trying to prevent their abusive partners from accessing firearms. These suggestions include using a trigger lock to prevent the gun from being fired, petitioning for a DVRO or ERPO, voluntarily turning over the firearms to local law enforcement, requesting law enforcement take firearms for safe keeping, contacting law enforcement to see if they can seize firearms, contacting The Bureau of Alcohol, Tobacco, Firearms and Explosives (ATF) to determine if they can seize firearms, and transferring the firearms to a third-party who will not return the firearms to the abusive partner unless instructed to do so by the victim (Battered Women’s Justice Project, n.d.; National Domestic Violence Hotline, n.d.). However, before victims engage in any of these firearm removal strategies, the Battered Women’s Justice Project advises that victims consult with an attorney, as not all of these options may be legally available to the victim or the victim could be arrested when seeking to have firearms removed (Battered Women’s Justice Project, n.d.). Lynch & Logan (2021) interviewed IPV victim service providers on how they would safety plan with victims when their abuser had access to a firearm. The two most frequent suggestions from service providers were to call law enforcement and to be knowledgeable about the perpetrator’s access to firearms. Specifically, service providers suggested that victims let law enforcement officers know that there is a gun in the home. They also suggested that victims always know the location of the firearm and privately try to remove or disarm the firearm if possible. For example, some service providers suggested that IPV victims try to remove 9 ammunition or try to get the gun out of the house and give it to a trusted person or a law enforcement officer, if safe to do so. How IPV Victims Prevent their Abuser from Accessing Firearms. Research on intimate partners’ efforts to prevent their violent partner from firearm access is limited. Some IPV victims specifically ask a court, as part of their DVRO petition, to have their partners’ firearms removed (D. W. Webster et al., 2010). Indeed, IPV victims are more likely to file for a DVRO if they experience firearm threats than if they do not experience firearm threats (Lynch et al., 2021). Additionally, some IPV victims have disclosed in focus groups that they got rid of the firearms they owned because their abuser could use these firearms against them (Lynch & Logan, 2018). However, we do not know if or when IPV victims are turning over firearms to law enforcement, requesting that law enforcement take firearms for safe keeping, or transferring firearms to third parties to prevent firearm IPV. Firearms and Additional Forms of Interpersonal Violence (Besides IPV) Firearm Access and Increased Rates of Interpersonal Violence (besides IPV) Additional interpersonal violence research suggests that firearm access for those at risk of harming others is associated with increased risk of firearm injury. An analysis of handgun purchasers in California found that those with violent misdemeanor convictions who were allowed to purchase a handgun before new California firearm restriction laws were put in place were more likely to be arrested for new gun and/or violent crimes compared to those who applied to buy handguns after the law was put in place and were denied purchase (Wintemute et al. 2001). In addition, “stand your ground” state laws, which allow individuals to use deadly force (e.g., firearms) as a first, instead of last, resort, are associated with increased homicide and firearm homicide rates (Degli Esposti et al., 2022). Similarly, “right to carry” state laws, which 10 require law enforcement to issue concealed carry permits as long as the requesting individual meets the criteria, are associated with higher state rates of violent crime and homicide (Crifasi, Merrill-Francis, et al., 2018; Donohue et al., 2019). Preventing Firearm Access for Those At-Risk of Interpersonal Violence Laws that prevent those convicted of violent misdemeanors from purchasing and possessing firearms are associated with decreases in firearm homicide rates and hospitalization firearm injury rates (Neufeld et al., 2021; Siegel et al., 2019). Additionally, handgun purchaser licensing laws, which ensure through an extensive process that those who are prohibited from firearms are not allowed a permit to purchase a handgun, are associated with a decrease in county homicide rates (Crifasi, Merrill-Francis, et al., 2018; Rudolph et al., 2015) and fatal mass shootings (Webster et al., 2020). Similarly, repealing a state handgun purchaser licensing law is associated with an increase in firearm homicides (Webster et al., 2014). Firearm Access and Suicide Firearms and Suicide In a suicide attempt, the use of firearms is associated with a higher risk of fatality. Ninety percent of suicide attempts committed with a firearm are fatal, leading to firearms having the highest suicide-case fatality rate (the proportion of suicide attempts that are fatal) (Conner et al., 2019). Comparatively, drowning and hanging, the methods with the second and third highest case fatality rates, are fatal in only 56.4% and 52.7% of attempts, respectively. Drug poisoning, the most common suicide attempt method, is fatal in only 1.9% of attempts. Even though firearms are only used in 4.8% of suicide attempts, 50.6% of suicides are committed with firearms (Conner et al., 2019). Suicide attempts by men are eight times more likely to involve 11 firearms than suicide attempts by women, and even though women are more likely to attempt suicide than men, men are more likely to die by suicide (Conner et al., 2019). Given this lethality, access to firearms is consistently found to be associated with increased risk of suicide (Anglemyer et al., 2014a). This relationship is found across study designs, including individual-level cohort and case control studies, as well as ecological cross- sectional and time-series studies (Azrael & Miller, 2016). For example, at the individual-level, those with firearm access in the home have three times the odds of dying by suicide and sixteen times the odds of dying by firearm suicide compared to those without firearm access in the home (Wiebe, 2003). Similarly, at the ecological level, state-level firearm ownership rates are associated with higher rates of firearm suicide and overall suicide (Miller et al., 2013). Simply having access to a firearm, however, does not cause individuals to become suicidal (Houtsma et al., 2018). Between those who live in a home with a firearm and those who live in a home without a firearm, there is no difference in suicide ideation and planning, anxiety disorders, mood disorders, or substance dependence or abuse (Miller et al., 2009). Likewise, firearm access is not associated with having a suicide plan, however, for those who have a suicide plan, firearm access is associated with a deadlier plan: firearm access is associated with seven-times higher odds of having a suicide plan involving a firearm (Betz et al., 2011). In summary, having access to firearms does not make someone suicidal, however, having access to firearms makes it more likely that a suicide attempt will result in a fatality. Preventing Firearm Access Restricting access to lethal means (e.g., removing firearm access) is one effective method of decreasing the risk of suicide (Azrael & Miller, 2016; Zalsman et al., 2016). According to Azrael and Miller (2016), the effectiveness of lethal means restriction is based on three premises. 12 First, many suicide attempts occur quickly after the plan is made, rather than after an extended amount of time and contemplation. Those who have attempted suicide have reported that the time from making the decision to attempt suicide to the time of attempt is sometimes as little as 10 minutes or less (Deisenhammer, 2009; Li et al., 2002) and a third of those of have seriously considered suicide have reported that the average length of time of these suicidal periods was one hour or less (Drum et al., 2009). Second, those who attempt suicide, including by firearm, are likely to choose a method that is easily available to them (de Moore et al., 1994; Eddleston et al., 2006). Third, out of all of those that survive a suicide attempt, less than 10% will go on to die by suicide (Owens et al., 2002). Therefore, if someone survives a suicide attempt, the attempter will likely not die by a suicide attempt in the future (Azrael & Miller, 2016). Therefore, ensuring there are no firearms in the household could reduce the likelihood that someone is able to quickly get a firearm and attempt suicide by firearm before the suicidal crisis has passed (Azrael & Miller, 2016). Research suggests that restricting firearm access may decrease suicide and firearm suicide rates. One policy intervention prevented Israeli soldiers from bringing their firearms home with them on weekend leave and this policy was associated with a 40% decreased suicide rate. Importantly, suicide rates by other means and suicide rates on weekdays did not change, bolstering the finding that the policy to reduce firearm access was indeed associated with a decrease in suicide (Lubin et al., 2010). In the United States, household firearm ownership fell between 1981 and 2002 and, with this natural experiment, researchers found that suicide and firearm suicide rates also significantly decreased during this time (especially for children), while suicide rates for other means did not change (Miller et al., 2006). Additionally, an analysis of permit-to-purchase laws (2015) suggests that preventing someone from being able to purchase a 13 handgun until they undergo a thorough and time-consuming background check can prevent suicide: Connecticut’s permit-to-purchase law was associated with a decrease in suicide rates, while an appeal of Missouri’s permit-to-purchase law was associated with an increase in suicide rates (Crifasi et al., 2015). Finally, Indiana and Connecticut’s ERPO laws were significantly associated with decreased state rates of firearm suicide (Kivisto & Phalen, 2018). Recommendations to Families Around Firearms Numerous medical associations and suicide prevention agencies suggest that those close to a suicidal person (e.g., family members and intimate partners) can intervene to reduce firearm access (American Academy of Pediatrics, n.d.; American Medical Association, n.d.; Gould et al., 2022). For example, the National Suicide Hotline suggests that those around the at-risk individual (e.g., family members) prevent access to lethal means or have a conversation with the at-risk person about reducing access to lethal means (Gould et al., 2022). Similarly, the American Medical Association encourages physicians to counsel suicidal patients and their families about reducing access to lethal means (American Medical Association, n.d.). Formal healthcare interventions to prevent suicide, including Counseling on Lethal Means (CALM) and Safety Planning Intervention (SPI), also suggest that healthcare providers engage family and friends in helping the patient make a plan to restrict access (e.g., having a family member store the firearm for a short time), or advising parents restrict firearm access from suicidal youth (Johnson et al., 2011; B. Stanley & Brown, 2012). A national study of adults who live in a home with a firearm found that 7.5% of participants had been spoken to by a healthcare provider about firearm safety. Sixteen percent of these patients were recommended to remove their firearms from the home if they or someone in their family was going through a hard time (Conner et al., 2021). 14 Families’ Efforts to Prevent Firearm Access There is limited research on families’ efforts to reduce firearm access to prevent suicide. A small body of research in pediatric and family practice settings has examined parent or caregiver behaviors to increase the safe storage of firearms when a child was at risk of suicide. This literature suggests that for children who are at risk of suicidal behavior, parents who receive lethal means counseling or firearm lockboxes and cable locks are more likely to prevent their child’s firearm access (Kruesi et al., 1999; Miller et al., 2020; Runyan et al., 2016). For example, when children were in the emergency room because of a suicide attempt, suicide ideation, or another behavioral health concern, or who screened positive for suicide ideation, clinicians in a multi-site hospital intervention encouraged their caregivers to engage in safe storage practices and remove firearms from the home or store them safely. The hospitals also provided these caregivers with firearm lockboxes and cable locks. This intervention was associated with caregivers being twice as likely to report safer storage of firearms two weeks after the hospital visit (Miller et al., 2020). However, this body of research does not explore families reducing firearm access when an adult was at risk of harming themselves. Adults are legally allowed to own firearms, making reducing firearm access of adults uniquely different than reducing firearm access of children. Researchers have also conducted focus groups with Veterans’ Affairs [VA] providers, clients, and their families (Walters et al., 2012) and older adults, their families, and those that work with older adults (Betz et al., 2011) about the risks of firearm access during times of heightened risk (for suicide, interpersonal violence, and accidental injury) and reducing access to firearms. These focus groups highlighted that families sometimes did not know how to broach the topic of firearm removal with the at-risk person or were worried about discussing it for fear 15 of harming the relationship or causing tension with the at-risk person (Betz et al., 2023). Focus group participants also discussed reducing the at-risk person’s firearm access. In the VA focus groups, some family members voiced that they supported the veteran keeping their firearms locked and family members outside the home responsible for the keys during times of heightened risk, however, both family members and clinicians were worried about families’ safety if the veteran wanted the keys back (Walters et al., 2012). Several family members also reported that they prevented their loved ones from accessing firearms (Betz et al., 2023; Walters et al., 2012), however, these incidents were not discussed in detail. Participants in both VA and older adult focus group studies also reported several legal concerns around temporarily reducing firearm access. First, focus group participants reported that they were sometimes unsure if firearms could legally be temporarily stored outside the firearm owners’ home (Walters et al., 2012). Many states have universal background check laws, which require the firearm recipient to first pass a background check. Background check policies were designed to prevent firearms from falling into the wrong hands, but they might have unintended consequences if they prevent or delay transferring firearms away from someone at- risk of suicide (McCourt et al., 2017). Additionally, focus group families worried if they could legally intervene when their family member had a constitutional right to a firearm and if they faced legal liability for not removing their family member’s firearms and someone was injured (Betz et al., 2023). Extreme Risk Protection Orders (ERPOs) In some states, family members and intimate partners can also use ERPOs to prevent their loved one from access if their loved one is at risk of harming themselves or others. Nineteen states and the District of Columbia (D.C.) have implemented ERPO laws that allow law 16 enforcement to petition for an ERPO; thirteen of these states and D.C. also allow family members and intimate partners to petition. If the ERPO is granted, the respondent is prohibited from firearm access for a temporary period (e.g., one year). A descriptive analysis of Oregon ERPO petitions found that the 33 petitions filed by families and household members contained descriptions of risk of harm to self (79%) and others (82%). A majority (79%) of the respondent in these petitions filed by family and household members were male. However, while ERPO research identifies risk factors described in ERPO petitions, whether these families engage in additional efforts before filing for an ERPO to prevent firearm access is still uninvestigated. Need for More Research on Families’ Efforts to Reduce Firearm Access More research is needed to explore families’ efforts to reduce firearm access when their loved one is at risk of harming themselves or others. Research suggests that families are worried about legal concerns and retaliation when reducing firearm access. However, research has not identified specifically how families are intervening when their adult loved one is at risk of harming themselves and others, and if these efforts are safe, legal, and effective. Additionally, there is no research on which families choose to reduce firearm access. Although there is research on who commits firearm IPH (Fridel & Fox, 2019), dies by firearm suicide (Conner et al., 2019), stores firearms unsafely (Crifasi, Doucette, et al., 2018a), and the respondents of ERPOs filed by families (Zeoli et al., 2021), we do not yet know which families are likely to intervene and reduce firearm access when their loved one is at risk of harming themselves and others. Research has also yet to comprehensively examine how incident factors (e.g., suicide threat, use of violence) may influence whether families intervene to reduce firearm access. For example, an IPV victim may be more likely to remove their partner’s firearms because some IPV 17 victims feel safer when firearms are removed (Vittes et al., 2013). On the other hand, IPV victims might also be less likely to prevent firearm access if they are afraid of retaliation (Lynch & Logan, 2018). By exploring how and when families reduce firearm access, researchers can then develop targeted interventions to support families’ safety. Theoretical Frameworks to Study Families’ Efforts to Reduce Firearm Access A theoretical framework is first needed to suggest when families might reduce firearm access so researchers can explore specifically how and when these efforts occur. Both the Theory of Planned Behavior and research on police legitimacy suggest that one place for researchers to study is after police arrive at a scene when someone is at risk of harming themselves or someone else. Theory of Planned Behavior The Theory of Planned Behavior [TPB] argues that stronger intentions are more likely to influence behavior (Ajzen, 1991) and has been used to identify predictors of third-party interventions to prevent suicide (Kuhlman et al., 2017) and as a conceptual framework for studying the firearm safe storage practices of women with children (Johnson et al., 2008). TPB has also been used to explain IPV victims’ intentions and behaviors to leave an abusive relationship (Edwards et al., 2015). TPB proposes that someone’s intention toward the behavior comprises their attitudes toward the behavior, their perception of subjective norms about the behavior, and their perceived behavioral control (Ajzen, 1991). Applying TPB to reducing firearm access, the theory suggests that the strength of a family’s intention to reduce their loved one’s access affects their likelihood they will do so. Families’ intentions to reduce firearm access consists of their attitudes toward reducing firearm access, their belief that others will approve and support reducing firearm access, and their belief that they can reduce firearm access. 18 TPB theory and research further suggest that someone’s ability to perform the intended behavior moderates the effect of intention on behavior (Ajzen, 2020). In other words, although someone might intend to engage in a behavior, they need the ability to do so. TPB suggests that the more actual behavioral control that someone has, the more likely they are to follow through on their intention (Ajzen, 2020). For example, an analysis of suicide gatekeeper behaviors found that participants were less likely to refer a suicidal person for help if they reported greater situational barriers (e.g., no quiet space to talk) (Kuhlman et al., 2017). Thus, according to TPB, families who intend to prevent firearm access will be more likely to act if they have the actual ability to do so. TPB theory suggests that law enforcement might increase families’ intentions to reduce firearm access after an incident for two reasons. First, according to TPB, for intentions to increase, families must believe that others will approve of prevention efforts. Law enforcement officers are consistently seen as one of the most credible messengers around firearm safety (Aitken et al., 2020; Anestis et al., 2021; Crifasi, Doucette, et al., 2018a), although this credibility varies by race and firearm storage practices when officers deliver messages on firearm safety for suicide prevention (Anestis et al., 2021). Black firearm owners score law enforcement officers as significantly less credible than White firearm owners, those who store their firearms locked view officers as significantly less credible than those who store their firearms unlocked, and those who store their firearms in a gun safe view law enforcement officers as significantly less credible messengers than those who do not store their firearms in a gun safe. However, Black firearm owners, firearm owners who store firearms locked, and firearm owners who store their firearms in a gun safe still rank law enforcement officers as the most credible messengers (Anestis et al., 2021). Therefore, law enforcement officers asking families about firearms and 19 removing firearms from the scene may signal to families that they approve of families reducing firearm access. Second, law enforcement officers may give families the perceived and actual ability to reduce firearm access. A survey of law enforcement agencies in eight U.S. states found that 75% of agencies are willing to provide temporary firearm storage, and an 84% are willing to provide temporary storage when a gun owner is worried about the mental stability of an adult family member and wants guns stored outside the home. Fifty-three percent of law enforcement agencies are also willing to temporarily store firearms that belong to another family member because of concerns about the stability of the gun owner (Runyan et al., 2017). In addition to law enforcement officers voluntarily storing firearms, depending on the incident factors and the state, law enforcement can also seize or remove firearms from the scene of an incident. For example, in Florida, when law enforcement officers take someone into protective custody for an involuntary examination and they have made a credible threat of violence to someone else, officers are allowed to take a firearm that the person possesses and ask for the voluntary surrender of firearms kept in the residence (Fla. Stat. § 394.463). Law enforcement’s ability to remove firearms and temporarily store them may give families the perceived and actual ability they need to reduce firearm access. Police Legitimacy Research on police legitimacy also suggests that families may cooperate with and assist law enforcement officers removing firearms from a scene. Police legitimacy is the belief that officers are legitimate members of authority, and that acting in this authority, officers can take actions to maintain social order and decrease crime (Tyler, 2004). In believing that officers can exercise this authority, civilians feel responsible for following and cooperative with police 20 directives (Tyler, 2004). Police legitimacy influences whether civilians cooperate with and assist the police (Sunshine & Tyler, 2003). The feelings of police legitimacy may be based on trust and that officers’ motives are acting in the best interests of the public (Tyler & Huo, 2002). Therefore, families may feel like they should help law enforcement officers remove firearms from the scene when law enforcement is allowed to do so. The Current Study The purpose of this dissertation study was to explore families’ efforts to reduce firearm access when their loved one was at risk of harming themselves or others using police reports from ERPO cases. These police reports were from the events that inspired police to file for the ERPO [precipitating event or PE] against the at-risk individual [respondent]. In Phase One, a content analysis identified and characterized families’ efforts to reduce the respondent’s firearm access at three time-points: before the PE, during the PE, and after police arrived at the PE. Next, in Phase Two, supervised machine learning identified which respondent and PE characteristics were important in predicting families’ efforts to reduce firearm access after police arrived. Content Analysis (Phase One) RQ#1: How do families reduce the respondent’s firearm access before the PE? RQ#2: How do families reduce the respondent’s firearm access during the PE? RQ#3: How do families reduce the respondent’s firearm access after police arrive at the PE? Multivariate Analysis (Phase Two) RQ#4: What respondent and PE characteristics are important for predicting families’ efforts to reduce firearm access after police have arrived at the PE? 21 CHAPTER 2: METHODOLOGY Overview The aims of this dissertation were to identify and characterize family and partners’ (hereby called families) efforts to reduce firearm access when their loved one was at risk of harming themselves or others and to determine which respondent and PE characteristics were influential in predicting these efforts. In Phase 1 of this dissertation, a content analysis was conducted using police reports from Florida Extreme Risk Protection Order (ERPO) cases, which had been collected in a multi-state ERPO study. These police reports were from events that inspired officers to petition for an ERPO [precipitating event or PE]. The content analysis identified and characterized families’ efforts to reduce firearm access at three time points: before the PE, during the PE, and after police arrived at the PE. In Phase 2, supervised machine learning identified which respondent and PE characteristics were important in predicting families’ reducing firearm access to reduce firearm access after police arrived at the PE. Figure 1 summarizes the data and study designed used. 22 Figure 1 Data and Study Design Overview Multi-State ERPO Study: Collected all Florida ERPOs from March 2018 - June 2020 (N=4,695) Coded random 50% of cases for case characteristics, including respondent demographics and PE characteristics (N=2,406) Dissertation Study: Applied inclusion criteria (N=473) Phase 1: - Content analysis, coding: - Families’ efforts to reduce firearm access at three time-points: Before PE During PE After police arrived at PE - Additional PE characteristics - Descriptive Statistics Phase 2: Supervised machine learning, predicting families’ efforts after police arrived at the PE Florida ERPO Cases Data The ERPO cases used in this research were collected by Dr. Annette Christy and Dr. April Zeoli as part of a study of a multi-state study of ERPOs.1 Drs. Christy and Zeoli collected all Florida ERPO cases that were filed from March 2018 (when the ERPO law was implemented) through June 2020, gathering a total of 4,695 cases. All of these ERPO cases were petitioned for by law enforcement because only law enforcement is allowed to petition for an ERPO in Florida (Fla. Stat. §790.401). 1 “A multi-state evaluation of Extreme Risk Protection Orders: Implementation, Outcomes, and Jurisdictional Variations” funded in 2020 by the National Collaborative for Gun Violence Research 23 Existing Data As part of this multi-state study, research assistants coded a random 50% (n=2,406) of these ERPO cases for case characteristics, including respondent demographics and PE characteristics. Descriptions of example PEs can be found in Appendix Table 1. To ensure reliability, research staff double-coded each case until interrater reliability scores of 0.80 were reached. Inclusion Criteria Inclusion criteria were applied to these 2,406 cases from the Multi-State ERPO Study for this dissertation. A case had to meet the following four inclusion criteria: (1) The ERPO case included a police report (e.g., incident report, offense report, arrest report) of the PE. When Florida law enforcement petition for an ERPO, they are required to include an affidavit explaining “the specific statements, actions, or facts that give rise to a reasonable fear of significant dangerous acts by the respondent” (Fla. Stat. §790.401). In addition to an affidavit, or in lieu of describing the respondent’s actions in the affidavit, law enforcement officers sometimes included their report from PE. (2) An adult family member or an intimate partner was at the scene of the PE and/or was available to talk to police after they arrived (including by phone). Adult family members were defined as immediate and extended family members who were (or were assumed to be) over the age of 18. Adult intimate partners were defined as current or former intimate partners who were (or were assumed to be) aged 18 or older. (3) The respondent had possession of or access to a firearm. (4) The respondent was aged 18 or older. 24 Twenty percent (n=473) of the cases coded in the multi-state ERPO study met the above dissertation inclusion criteria. Cases that met the inclusion criteria were significantly more likely to contain descriptions of suicide, interpersonal violence, alcohol use, arrests, and information about firearms than cases that did not meet the inclusion criteria (Appendix Table 2). Advantages of Using Police Reports Using police reports has several advantages for characterizing and predicting families’ efforts to reduce firearm access. First, police reports contain descriptions of what happened during the incident, such as families hiding guns from their loved one and other event characteristics, such as the at-risk person threatening suicide. These reports also contain descriptions of how families interacted with officers, such as families giving officers consent to search for firearms. Additionally, police reports include the statute or crime under which someone was arrested or taken into protective custody, which determines if officers can remove firearms from the scene. In Florida, police are required to seize firearms “found upon the person arrested” if the person is being arrested for displaying, using, or attempting to use a firearm while committing or attempting to commit a felony (Fla. Stat. § 790.08). Additionally, when officers are taking someone into custody for an involuntary mental health examination (hereby called involuntary examination), they “may seize and hold a firearm or any ammunition the person possesses at the time of taking him or her into custody if the person poses a potential danger to himself or herself or others and has made a credible threat of violence against another person.” If police take custody of the person at the person’s residence, then the law enforcement officer also “may seek the voluntary surrender of firearms or ammunition kept in the residence which have not already been seized under subparagraph” (Fla. Stat. § 394.463). Officers seizing firearms or seeking the voluntary surrender of firearms may influence families’ efforts to reduce firearm 25 access. In summary, police reports contain descriptions of families’ behaviors and PE characteristics, which allows for the characterization and prediction of families’ efforts to reduce firearm access. Content Analysis (Phase One) Overview Analytic Plan To address the first aim of the dissertation (identifying and characterizing families’ efforts to reduce firearm access), a content analysis was conducted. In analyzing a body of text, content analyses can be used to categorize the actions that people take, as well as the actors/participants involved in these actions (Krippendorff, 2019). Using police reports from ERPO petitions, families’ efforts to reduce firearm access, as well as additional PE characteristics needed to address my second aim (predicting these efforts) were coded. Units In content analyses, units are the set of text, documents, or other media that are independent elements to be examined (Krippendorff, 2019). In this dissertation, units were police reports. If there was more than one police report included in an ERPO petition for the same PE (e.g., two police officers responded to the scene and they each wrote a report), all reports from that single PE were considered one unit. Coding and Inter-Rater Reliability Creating a codebook is one way to create reproducible, analyzable records in a content analysis (Krippendorff, 2019). Using 30 police reports where the intimate partner was a victim of the respondent’s violence, a codebook of families’ efforts to reduce firearm access (Table 1) and additional PE characteristics (Table 2) was created. 26 A second way of creating reproducible, analyzable records in a content analysis is to use a second coder and ensure inter-rater reliability. In content analyses, the data collected should remain consistent, regardless of which analyst is coding the text. Discrepancies between coders indicate that there are inter-coder differences in the interpretation and application of the codebook (Krippendorff, 2019). To measure the reliability of the coding, the dissertation author (JP) and another coder (SO) independently coded 50 cases (10.6%) using an instrument that JP developed in Qualtrics to record coding. SO was a coder on Dr. Zeoli’s multi-state ERPO study and was already familiar with Florida ERPO casefiles. An interrater reliability test was conducted to measure the coding reliability. The Cohen’s kappa statistic measures the proportion of agreement between two coders accounting for chance (Sim & Wright, 2005), and a kappa statistic of 0.80 is usually the standard in content analyses (Krippendorff, 2019). The kappa statistic was 0.89 (indicating strong agreement), so JP coded the remaining 423 police reports independently and then returned to the initial 50 double-coded cases and cleaned discrepancies. During coding, three additional codes that seemed relevant to families reducing firearm access were identified between JP and SO. These additional codes were 1) officers telling someone on scene about ERPOs or that they would seek an ERPO, 2) police officers including in their report that they would seek an ERPO but did not report telling someone on scene, and 3) intimate partners or family members lived with the respondent. A codebook for these three additional codes was created with definitions and examples (Table 3) with input from SO, and then JP independently coded the 473 cases for these three additional codes. 27 Table 1 Codebook of Families’ Efforts to Reduce Firearm Access, developed with 30 cases Code Before PE Examples “IP took the gun from him” [a few days before PE] % (n) 6.7% (2) Definition Narrative includes that before the PE, the family tried to prevent the respondent’s firearm access/possession/use During PE Narrative includes that during the PE, the family member/IP took efforts to prevent the respondent from firearm access/possession/use After Police Arrived at PE Gave police Narrative includes that the family access/consent to search for/remove firearms member/IP gave police access to firearms, or gave consent to search for firearms and/or remove them “IP explained that she has given [respondent] all of his personal belongings other than a few minor items. The only items IP refuses to turn over [to respondent] is [respondent]'s firearms.” “The IP advised she then grabbed the slide to the 3% (1) pistol and pushed his hand away throwing it on the floorboard.” Note: Keeping a respondent from a location is not included in this category, unless the narrative included that this was specifically to prevent firearm access. For example, an IP locking the R out of the house would not be coded here; an IP locking the R out of the house specifically so he could not access his guns would be coded here. The IP “provided the code for the safe and provided consent for the police department to take the firearms” The IP “signed a consent to search form along with giving verbal consent to search their vehicle” The IP “stated that he could not find the firearm in the house, but that it was more than likely in their vehicle that [respondent] uses for primary transportation. IP opened the vehicle at which point I located a handgun in the trunk of the vehicle.” 28 13.3% (4) Table 1 (cont’d) Asked police help with firearms Gave police firearms Narrative includes that the family member/IP told police that they wanted the firearms removed or asked police to take firearms Narrative includes that the family member/IP relinquished or gave firearms to police “IP asked that I take the firearm for safe keeping” “IP advised she wanted the firearms out of the hotel 3% (2) room” “IP then freely relinquished all of the guns in the 10% (3) residence to LE for safety reasons” “IP provided deputies on scene with all firearms from the residence” “IP returned to the front door with a black Taurus revolver in hand. IP… collected the revolver from her own residence, without the direction of LE” Will prevent access Narrative includes that the family IP “stated she would be giving [respondent’s] 6.7% (2) member/IP told police that they will take efforts to prevent the R from firearm access/possession/use firearms to her brother.” “IP was willing to turn over remaining firearms to stepdaughter in order to get them out of the apartment to prevent [respondent] from having access.” IP would buy a safe to keep firearms in and only she would have the combination to prevent husband from getting access 29 Table 2 Codebook of Additional PE Characteristics, developed with 30 cases Term Refused police Definition Narrative includes that the family access/consent to search for/remove firearms member/IP refused to give police access to firearms, or refused consent to search for firearms and/or remove them Examples “Officers requested IP hand over the weapons to [PD] for safekeeping, however, she refused to allow officers into her home or access to the weapons” “IP was uncomfortable with officers looking through her belongings and asked that she have a legal representative and her daughters present before officers complete any further searches.” Present during the PE Adult family member or intimate Spouse at home while respondent attempts suicide partner is at the scene where the precipitating incident took place (precipitating incident) Present with police Adult family member or intimate Respondent’s spouse arrives at home after precipitating Intimate partner threatened by respondent (precipitating incident) partner is at the scene when law enforcement officers arrive or are able to talk to the officer in-person or on the phone before the officer write up their report incident while police are already at home Intimate partner leaves residence after being threatened by respondent during the precipitating incident and goes to neighbor’s house; police interview intimate partner while she is still at the neighbor’s house While responding to the precipitating incident, police talk to the adult respondent’s parent who lives with the respondent on the phone Crime or statute under which someone arrested or taken for an emergency involuntary mental health examination Crime or statute listed on police Baker Act report as offense committed or reason for report 394.463(2)(A)(2) 30 Table 3 Codebook of Three Additional PE Characteristics, Developed After First Round of Coding Code Told someone Definition Police included in report narrative that they told someone at the PE about the ERPO process or that they would be filing for an ERPO about ERPOs Initiated ERPO process without telling someone at PE Police included in the report narrative that they/the department will be following up with an ERPO but did not report telling someone at the PE Lived with Family member or intimate partner lives with respondent the respondent or owns where the respondent lives Examples “I explained the RPO process to R and he voluntarily agreed to surrender his firearms to the Sheriff’s office for safekeeping.” “[Officer] advised [Respondent’s wife] he would be initiating a RPO and wished to retrieve all of the firearms within the residence. [Respondent’s wife] voluntarily handed over all of the firearms known to be within a residence. [Respondent’s wife] signed a property sheet on scene acknowledging all firearms would be at the [PD] for safekeeping.” “I completed an affidavit in support of a temporary ex-parte RPO and forwarded it to the [sheriff’s office] legal section.” “An RPO attachment/request has been attached to this report as [respondent] has access to at least one weapon(s). Legal was notified via email.” “I will be pursuing a RPO.” Note: The category above (police mentioning RPOs to someone on scene) and this category are exclusive. If the above category is checked, this category should not be checked. Family member or intimate partner address the same address as R’s address. Report notes that IP is the respondent’s “live-in girlfriend.” 31 Descriptive Analyses Frequencies were calculated to describe families’ efforts to reduce firearm access. Because these efforts have never been reported, this basic descriptive research is needed. Multivariate Analyses (Phase Two) Overview The second aim of this dissertation was to determine the relative importance of respondent and PE characteristics in predicting families’ efforts to reduce firearm access. To complete this aim, five supervised machine learning models were run and then ensembled together to identify the most important respondent and PE factors. Advantages of Supervised Machine Learning Supervised machine learning is a statistical method that can address the methodological challenges of model complexity and multicollinearity that can arise in traditional logistic regression models. Without previous research on families’ efforts to reduce firearm access, there are many respondent and PE characteristics to consider, and a traditional logistic regression model could be overfit. As variables are added and increase the complexity of the model, the variance will increase, and the model will be less likely to generalize to new data. Additionally, multiple factors likely predict families’ efforts, and some of these factors are similar and highly correlated with each other. For example, the respondent threatening or using firearm violence and the police being able to remove firearm(s) because the respondent committed a felony with a firearm are highly correlated (0.54, p<0.001). However, it is important to tease out the effects of these variables on families’ efforts to reduce firearm access, therefore an analytic technique that can handle these challenges is needed. While traditional models (e.g., non-regularized logistic regression models) are useful when testing hypotheses about the specific relationship between predictor variables and the outcome and when there are limited variables in the model, machine 32 learning is useful when there is little or no a priori knowledge on the topic and when predictors have not been previously defined in the literature (Rajula et al., 2020). Machine learning allows researchers allows researchers to find predictive patterns with many predictor variables that are highly related (Bzdok et al., 2018). Supervised machine learning has two additional benefits. First, supervised machine learning allows multiple algorithms to be combined (“ensembled”) together into a meta- algorithm. As each supervised machine learning algorithm has its limitations, ensembling the algorithms together into a meta-algorithm might yield a prediction potentially greater than just using one machine learning algorithm. Second, supervised machine learning splits the dataset into a training dataset, which is used to develop the model(s), and a testing dataset, which is used to test the performance of the model(s) and increases replicability(Boehmke & Greenwell, 2020; Flach, 2012). The model’s performance is commonly assessed using sensitivity, specificity, and accuracy. Sensitivity measures the proportion of true positives predicted correctly (how good the model is at predicting efforts when those efforts truly occur), specificity measures the proportion of true negatives predicted correctly (how good the model is at predicting no efforts when there truly are no efforts), and accuracy is the proportion of all true results. For example, an accuracy of 99% indicates that 99% of the time the prediction is accurate (Wang et al., 2010). Outcome Variable Based on the sample size and the positive proportion of the outcome value, there were two possible binary outcome variables to use in the supervised machine learning models: all efforts to reduce firearm access (n=473, 35.94% yes), or efforts to reduce firearm access after police arrive (n=464, 26.51% yes). A random forest model, which typically provides good predictive performance with little tuning (Boehmke & Greenwell, 2020), identified that the 33 outcome variable efforts after police arrived yielded greater model accuracy and sensitivity (Appendix table 3). Therefore, efforts to reduce firearm access after police arrived was selected as the outcome variable for the supervised machine learning models. Predictor Variables In determining which predictors to include in the machine learning models, all of the respondent and PE characteristics previously coded in the Multi-State ERPO Study and the PE characteristics from the content analysis in Phase 1 were considered. Families’ demographics were not considered because these data could not be consistently collected from police reports, as many victims’ data was blacked out. Any of these potential predictor variables that had near- zero variance (identified with the R package caret) were either combined as theoretically appropriate with other variables (e.g., opioid use with illicit drug use) or were dropped from the analysis. Predictor variables that have little variance can cause problems during the resampling process in machine learning and typically give very little information to a model (Boehmke & Greenwell, 2020). As there were numerous ways to combine targets of threats and uses of violence so that they passed the near-zero variance requirement, chi square tests were conducted, and predictor group combinations which were significant at p<0.10 were used. The final list of predictor variables used in the supervised machine learning models included both respondent and PE characteristics. A definition of each predictor variable and a description of how it was constructed can be found in Appendix Table 4. Respondent demographics included in the models were age, race/ethnicity, gender, and whether the respondent had a mental health issue or was a current or former member of the armed services. Respondent characteristics of whether they lived with an intimate partner or 34 family member, and if the respondent possessed or had access to a handgun, long gun, or more than one gun were also included. PE characteristics included in the models were the respondent’s suicide risk (exhibited ideations, threats, or plans; aborted a suicide attempt; attempted suicide; referenced or used a firearm in the suicide risks), interpersonal violence against families (used violence, used firearm violence, threatened violence, threatened firearm violence), interpersonal violence against others (used violence, used firearm violence, threatened violence, threatened firearm violence), threatened a mass shooting/multiple victim shooting, used alcohol, used drugs, exhibited irrational or erratic behaviors, was arrested, was taken for an involuntary mental health examination (which allows officers to remove firearms), and committed a felony with a firearm during the PE (which allows officers to remove firearms). Officers telling someone at the scene about ERPOs and officers writing in their reports that they would seek an ERPO but did not report telling someone at the scene were also both included in the models. Finally, as officers’ ERPO training and protocol development may not occur immediately after an ERPO law is passed (Pear et al., 2021; Swanson et al., 2019), the number of months since the Marjory Stoneman Douglas High School Public Safety Act (MSDHSPSA) was also included as a predictor. This act created Florida’s ERPO law and allows officers to remove firearms when someone was subject to an involuntary mental health examination and was passed less than a month after the mass shooting at the high school (Marjory Stoneman Douglas High School Public Safety Act, 2018). Bivariate Hypothesis Tests Before the supervised machine learning models were run, bivariate hypothesis tests (i.e., Chi-square, Fisher’s Exact, and Wilcoxon sum-rank tests) determined differences in predictors 35 between families who reduced firearm access after police arrived at the PE and families who did not reduce firearm access after police arrived at the PE. Training and Testing Datasets Data were separated into a training dataset (80%, n=370) and testing dataset (20%, n=94). To address the low sensitivity, the positive class (families did reduce firearm access access after police arrived) was upsampled. This upsampling process randomly samples the positive class with replacement (Kuhn & Johnson, 2016), so that the number of cases in which families did reduce firearm access after police arrived were the same number of cases in which families did not reduce firearm access after police arrived. Hyperparameter Tuning Hyperparameters are extra model parameters whose values are not estimated from the data and must be set manually to increase the model’s predictive performance. Hyperparameters find the balance between overfitting and underfitting the model. To find the optimal hyperparameter values for each model, k-fold cross validation and a grid search was used. In this process, the training data was divided into ten (a common value of k) equal subsets of data. Next, for each set of hyperparameters, the model was trained on k-1 folds (“training fold”) and then tested for model performance on the remaining fold (“holdout fold”), until each fold was used as the holdout fold once. The model’s performance across the nine folds was then averaged into a single performance measure. The set of hyperparameters with the best average performance measure was selected (Flach, 2012). The Area Under the Curve (AUC) of the receiver operating characteristic (ROC) curve was used to measure model performance. The ROC curve plots the sensitivity vs the specificity at different classification thresholds, and the AUC is a summary measure of the model’s performance (Kuhn & Johnson, 2016). 36 Models Five commonly used supervised machine learning models (random forest, stochastic gradient boosting, elastic net regularization, support vector machine, and naïve Bayes) were run using the caret package in R (Kuhn, 2008). Random Forests. Decision trees classify the data, and in the process, form tree-like structures. Starting with the root node, the decision tree algorithm finds the best feature to split (classify) the data, so that the overall error between the actual classification and predicted classification is minimized. The tree then splits into two branches. In each path, when the algorithm finds the next best feature to split the data (the node), the node then splits into additional branches. This process continues until the tree becomes too complex, the researcher decides to stop the process with stopping criterion, or the data is unable to be split into different classifications. This ending node is called a “leaf.” In random forests, each decision tree is created using a different sample of bootstrapped data. As many of the trees could be highly correlated, random forests build a forest of de- correlated trees to improve the model’s performance. Random forests does this during the splitting process: when at a decision node, the algorithm is limited to a random subset of variables. Random forests then ensemble the predictions of these de-correlated decision trees together to create an average prediction, leading to a more predictive model than by just using a single decision tree (Boehmke & Greenwell, 2020; Flach, 2012; Trappenberg, 2020). Using the ranger method in caret, following hyperparameters were tuned: the number of predictors to select from at each split; the minimum number of observations in a node, which creates stopping criterion for each tree; and the splitting rule (Kuhn, 2008; Kuhn & Johnson, 2016). 37 Stochastic Gradient Boosting. Like random forests, stochastic gradient boosting machines ensemble decision trees together. However, random forests use deep (long decision paths from root to leaf) de-correlated trees, ensemble all the trees at the end of the process, and each tree contributes equally to the model. However, gradient descent boosting machines ensemble shallow trees in sequence and each new tree’s predictive ability “boosts” the predictive ability of the previous tree. The algorithm continues until it identifies the minimal loss function (the smallest difference between the observed outcome and the predicted outcome). The trees, therefore, do not contribute equally to the final model. Stochastic gradient boosting uses a random subsample of data for each decision tree. This random sampling adds some randomness into the algorithm’s descent towards the lowest loss function. Although randomness does not guarantee the machine learning algorithm will find the global minimum loss function, the randomness helps the algorithm avoid local minimum loss functions (Boehmke & Greenwell, 2020; Flach, 2012). Using the gb method in caret, the following hyperparameters were tuned: the number of trees, the learning rate, the depth of each tree, and the minimum number of observations in a node (Kuhn, 2008; Kuhn & Johnson, 2016). Support Vector Machine (SVM). In SVM, all data points are plotted in an n- dimensional space, where n is based on the number of features (e.g., variables). As data patterns are often not linearly separable, the goal of SVM is to create a decision boundary through this higher-dimensional space that classifies the data into two categories (efforts to reduce firearm access vs. no efforts). A kernel function enlarges the feature space so that the SVM algorithm can find this decision boundary. The data points closest to the decision boundary are called “support vectors” and the linear combination of these support vectors creates the decision boundary (“hyperplane”). The SVM model classifies all points on one side of the hyperplane as 38 one class (e.g., efforts to reduce access) and those on the other side as the other class (no efforts) (Boehmke & Greenwell, 2020; Flach, 2012; Trappenberg, 2020). Using the svmRadial method in caret, the radial basis function kernel was used the following hyperparameters were tuned: “c” (a penalty for each misclassified data point) and sigma, which controls the smoothness of the decision boundaries (Kuhn, 2008). Regularized Logistic Regression – Elastic Net. Regularized ridge logistic regression adds in a penalty term to loss function to shrink, or regularize, the estimated coefficients. By shrinking the coefficients, regularized regression reduces model complexity and decreases the variance, increasing the likelihood that the model will generalize to future data (Boehmke & Greenwell, 2020). Elastic Net is a combination of regularized ridge regression and regularized lasso regression. In ridge regression, the penalty term is the sum of the squared coefficients and this regression shrinks the coefficients towards zero, and in lasso regression, the penalty term is the sum of the absolute values of the coefficients, and this regression shrinks some coefficients to zero. Elastic net regression combines these two types of penalties to give the researcher the benefit of both types of regulariztion (Kuhn & Johnson, 2016). Using the glmnet method in caret, the following hyperparameters were tuned: alpha, which is the mixing parameter between ridge and lasso regression, and lambda, which is the regularization rate (Kuhn, 2008; Kuhn & Johnson, 2016). Naïve Bayes. The naïve Bayes algorithm is based on Bayes Theorem, which says that probability that A occurs given that B occurs P(A|B), the probability that B occurs given that A occurs P(B|A), and the probability of both A and B occurring without any given conditions. In order to allow the computation to scale with the large number of predictor variables, the naïve Bayes classifier assumes that all the predictor variables are conditionally independent. However, 39 data is often correlated and violates this assumption. For example, a naïve Bayes algorithm can be used to predict whether an email is spam using the words of the email as predictors, and certain words are often correlated. However, even when the data violate the assumption of independence, the naïve Bayes classification algorithm still performs well when making classifications. This is because the naïve Bayes classifier is not seeking the exact probability of the classified outcome (engaging in firearm access prevention efforts); the classifier is simply looking for if the likelihood of engaging in firearm prevention efforts is higher than the likelihood of not engaging in these efforts. In other words, the naïve Bayes classification algorithm is a poor estimator, but performs well with classification, even when the assumption of independent predictors is violated. Ensembled model Using the caretEnsemble package in R, each of the five models described above was run so that each model used the same cross-validation folds. A generalized linear model then created a linear blend of the five models. Variable Importance Metric Depending on the purpose of the algorithm, the goal may be to increase sensitivity, specificity, or another performance measure (Kuhn & Johnson, 2016). Officers helping families reduce firearm access may help prevent firearm injury, and falsely predicting families reducing firearm access with police might not cause harm, so sensitivity was prioritized in selecting which model to compute the variable importance metric. In this dissertation, the final ensembled model provided a weighted estimate of the importance of each predictor variable. Institutional Review Board Approval This research was designated Not Human Research by MSU’s IRB (STUDY00007645 40 CHAPTER 3: CONTENT ANALYSIS FINDINGS The first aim of the study was to identify and characterize family and intimate partners’ (hereby called families) efforts to reduce firearm access. Using 473 police reports from ERPO cases, the content analysis identified and characterized how families tried to reduce firearm access at three timepoints: before the precipitating event (PE), during the PE, and after police arrived at the PE. Across all timepoints, 35.94% of the police reports included information that families tried to reduce the respondent’s access to firearms. This chapter describes families’ efforts in detail, and Table 4 shows a breakdown of each effort. Table 4 Families’ Efforts to Reduce Firearm Access (N=473) At any time Before the PE During the PE (n=368) a After police arrived at the PE (n=464)b Gave consent/access Asked for help Gave firearms Would prevent access after officers left the PE a Some families were not present during the PE. b Some families were not present with police officers. Reports that included information that families tried to reduce access 170 (35.94%) 12 (2.54%) 54 (14.67%) 123 (26.51%) 63 (51.22%) 13 (10.57%) 23 (18.70%) 36 (29.27%) Efforts to Reduce Firearm Access Before the PE Twelve police reports (2.54%) included information that families tried to prevent the respondent from accessing firearms before the PE. These efforts included hiding firearms, taking firearms from the respondent’s home, and putting locks on firearms or moving them to secure places (e.g., a safe). These actions did not always result in a complete reduction of firearm access. For example, a woman took her boyfriend’s firearm after he held it to his head and threatened suicide, but returned the firearm later in the week after he told her that he went to counseling. 41 Efforts to Reduce Firearm Access During the PE Fifteen percent of police reports included information that families intervened during the PE to reduce firearm access. Many of these interventions included hiding the firearm or moving it so the respondent could not get it. Families hid firearms in a myriad of places inside the residence, including in other bedrooms, under the bed, in closets or dressers, in the bathroom, and in more secure places, such as safes. Families also hid guns outside in vehicles and car trunks, a flowerpot, a recycling bin, and other places in the yard. In one case, one woman unloaded the firearm, threw the bullets into a laundry basket, and hid the unloaded firearm in their neighbor’s backyard. To reduce access during the PE, some families used physical force. For example, a woman slapped her son to get a shotgun away, and a man got his ex-girlfriend into a chokehold and told her to drop the gun. In another case, a woman fought with her husband over the bag where firearms were stored, and jumped on top of the bag so he could not get a gun out. Some police narratives reported that families tried to talk the respondent down or asked the respondent to give them the firearm. For example, one woman hugged her granddaughter to calm her down and let her cry on her shoulder, while the respondent’s mother took the gun away from the respondent and hid it. In another case, a woman said that she asked her husband for the gun and after a few times of asking, he unloaded the gun and gave it to her. Not all these efforts to reduce access during the PE were successful or would have kept the respondent completely from accessing firearms. For example, one woman said that she made several attempts to calm her husband down and stop him from loading his shotgun, but after he loaded the final round, he advised her to “Get the hell out of here,” and so she ran out of the 42 residence. Another woman removed the guns and ammunition from the home and put them in the car but told police that her boyfriend still had the keys to the car. Finally, some efforts to reduce firearm access resulted in the respondent retaliating and fighting back. In one case, the respondent grabbed a butcher knife and then grabbed his girlfriend by the throat after she refused to take the lock off the firearm. In another case, the respondent hit his son in the back and in the face with the stock of the gun after his son tried to grab the gun from him. Efforts to Reduce Firearm Access After Police Arrived at the PE Twenty-seven percent of police reports (n=123) included information that families reduced firearm access after police arrived. Families gave officers consent and/or access to search for and remove firearms, asked for officers’ help keeping the respondent away from firearms, gave officers firearms, and told officers that they would prevent the respondent from firearm access after officers left. Gave Police Consent and/or Access to Remove Firearms Of the cases in which families reduced access after officers arrived, just over half (51.22%) included information that families gave officers consent or access to search for and remove firearms. Officers reported that some families gave them specific descriptions of firearms and where they could be found in the residence and gave consent for police to remove these firearms. Other police narratives explained that families gave police written or verbal consent to search the house or property for firearms. For example, one respondent had put the firearm in a safe during the PE, and when police arrived, his fiancé completed a waiver of search for the safe. Officers then seized the firearm, placed it into evidence, and gave her a property 43 receipt. In another case, a woman gave officers consent to search a vehicle and property for firearms after her husband refused to talk to police. Asked Police for Help Reducing Firearm Access Of cases in which families reduced firearm access after officers arrived, 10.57% included information that families asked officers to take firearms or told officers that they did not want access to firearms. For example, a woman asked police to take custody of firearms until her husband could get mental health assistance, and officers reported that they took the firearms for safekeeping. In another case, before a respondent was transported for an involuntary examination, he gave officers permission to give his firearms to his sister. The respondent’s sister told police that she did not feel comfortable taking the firearms because she and the brother lived together, and her brother would still be able to access the firearms after being released from the involuntary examination. Officers reported that they took the guns and placed them into property for safekeeping. Gave Police Firearms Nineteen percent of the cases in which families reduced firearm access after police arrived included information that families directly gave police officers firearms. In some cases, officers specifically described that families did so freely. For example, police reported one woman “freely relinquished” all the guns in the residence to the police for safety reasons. In other cases, officers simply described that families gave them the firearm, so it is unclear if family or partners were doing this because police are allowed to seize firearms in some situations under the law, or if they were doing so freely. For example, officers simply described that the respondent’s mother opened a safe and gave officers the firearm her son had used to threaten his girlfriend and other family members, and that they placed the firearm into evidence. 44 Would Prevent Firearm Access After Officers Left the PE Of cases in which families reduced access after officers arrived, 29.27% included information that families would try to keep the respondent from accessing firearms after officers left the scene. Families did this in several ways. First, if officers were removing firearms from the residence and the families could not find all of the firearms, some families said that they would continue looking for the firearm after officers left and call officers to remove the firearm once they found the remaining firearms. Second, some of the guns at the respondent’s residence belonged to families who did not live with the respondent, and families sometimes told officers they would take their firearms back to their own residences. Additionally, some families who did not live with the respondents agreed to temporarily store the respondent’s firearms to keep them away from the respondent. Sometimes it was clear that these firearms would be safely stored at the new location. For example, in one case, before the respondent was transported for an involuntary mental health evaluation, all firearms in the home were turned over to the respondent’s brother-in-law, who told officers that he had a gun safe at his home and would store the firearms there. However, in other cases, the police report did not specify how families would store these firearms in the new location, so it was unclear if the firearms would be stored safely (e.g., in a gun safe). Finally, some families who had their own firearms in the home they shared with the respondent told police that they would prevent the respondent from having access to these firearms. For example, one mother said that she would secure the pistol that the respondent used in the incident so that the respondent would not be able to get to the pistol anymore, and another father agreed to secure the firearms in the home. Some of these discussions with police happened during conversations about the ERPO process. For example, after the respondent was taken for an involuntary examination, the 45 respondent’s wife told officers that she did not want to turn over her three firearms for safekeeping. The police told her that they were going to file an ERPO, and the wife signed an affidavit agreeing that she would store the firearms in a way that the respondent would not have access to them. In a different case, a police officer filled out ERPO paperwork with the respondent’s wife and son present, and the wife told police that she wanted to do a third-party transfer so her son could take the firearms, and her son took the six firearms to his residence. Summary of Content Analysis Findings In police reports with families present that were included in ERPO cases, 35.94% of reports included that families tried to reduce the respondent’s firearm access. A small proportion of reports (2.54%) included information that families reduced access before the PE, and families in these cases sometimes did not completely prevent the respondent’s firearm access. Fifteen percent of reports included information that families tried to stop the respondent from accessing firearms during the PE. Families did so by hiding or moving firearms to new locations inside and outside the residence, trying to calm the respondent down, and using force to take the gun or keep a gun away from a respondent, but sometimes were unsuccessful or faced retaliation. Just over a quarter of police reports (26.51%) included information that families reduced firearm access after officers arrived at the PE. Families gave officers consent and/or access to search residences and other property for firearms and remove them, asked officers for help in removing firearms, and gave officers firearms. Families also told officers that they would prevent the respondent from accessing firearms after officers left PE, including by temporarily storing firearms for the respondent outside the respondent’s residence. Some of families’ efforts to reduce firearm access after police arrived occurred during conversations with officers about ERPOs. 46 CHAPTER 4: SUPERVISED MACHINE LEARNING RESULTS Supervised learning was used to address the second aim of this study (to determine which respondent and PE characteristics are the most influential in predicting families’ efforts to reduce firearm access after police arrived). This chapter describes the predictor variables used in the supervised machine learning models and how they were associated with families’ efforts to reduce firearm access after police arrived. This chapter also describes the supervised machine learning models’ performance measures and which respondent and PE characteristics were the most influential in the ensembled model’s prediction. Predictor Variables and Bivariate Hypothesis Tests Table 5 shows respondent and PE characteristics that were used as predictor variables in the machine learning models, by whether families reduced firearm access after police arrived. Families were significantly more likely to reduce firearm access after police arrived when the respondent exhibited suicide ideation, threats, or plans (p=0.030) and if families lived with the respondent (p=0.000). Families’ efforts to reduce firearm access after police arrived were also more likely if the respondent had access to or possession of a long gun (p=0.000) or if the respondent had more than one firearm (p=0.000). Police actions during the PE were also associated with families’ efforts to reduce firearm access after police arrived. Police had the authority to remove firearms if the respondent committed a felony with a firearm during the PE and when the respondent was subject to an involuntary examination during the PE, and both items were associated with families reducing firearm access after police arrived, but in different directions. When the respondent committed a felony with a firearm, families were less likely to reduce firearm access when police arrived (p=0.047), and when the respondent was subject to an involuntary examination during the PE, 47 families were more likely to reduce firearm access after police arrived (p=0.000). Families were also more likely to reduce firearm access after police arrived when officers told someone at the scene about the ERPO process or that they were going to petition for an ERPO (p=0.000). Finally, the time since the Marjory Stoneman Douglas High School Public Safety Act (MSDHSPSA) was enacted was also positively associated with families’ efforts to remove firearms after police arrived (p=0.017). Correlations between all the predictor variables to each other can be found in Appendix Figure 1. 48 Table 5 Respondent and PE Characteristics Used as Predictors in the Machine Learning Models, by Whether Families Reduced Firearm Access After Police Arrived at the PE (N=464) Characteristics All cases (N=464) Families reduced firearm access after police arrived at PE (n=123) Families did not reduce firearm access after police arrived at PE (n=341) Test statistica p-value Respondent characteristics Female Age: Mean (SD) Race/ethnicity White (reference category) Black Other Race/Ethnicity Mental health issue Current or former armed services Lived with an intimate partner or family member Access to or possession of a handgun Access to or possession of a long gun Access to or possession of more than one firearm Precipitating Event (PE) characteristics Suicide ideation, threats, or plans Suicide aborted attempt Suicide attempt Firearm involvement in suicide risk Use of violence against IP/family Firearm use of violence against IP/family Threat of violence against IP/family Firearm threat of violence against IP/family Use of violence against others 11.85% (55) 41.04 (15.41) 8.13% (10) 42.36 (16.18) 13.205 (45) 40.57 (15.11) 2.22 -0.90 3.66 0.136 0.370 0.160 78.05% (96) 10.57% (13) 11.38% (14) 43.90% (54) 10.57% (13) 84.55% (104) 88.62% (109) 64.23% (79) 72.36% (89) 66.67% (82) 21.14% (26) 8.94% (11) 50.41% (62) 28.46% (35) 8.13% (10) 28.46% (35) 21.95% (27) 7.32% (9) 77.42% (264) 15.54% (53) 7.04% (24) 36.07% (123) 10.26% (35) 63.05% (215) 82.70% (282) 43.11% (147) 46.04% (157) 55.43% (189) 15.54% (53) 6.45% (22) 46.92% (160) 31.38% (107) 12.61% (43) 37.24% (127) 30.21% (103) 12.90% (44) 2.35 0.01 19.45 2.39 16.14 25.13 4.70 2.00 0.85 0.44 0.36 1.79 3.08 3.06 2.79 0.125 0.924 0.000 0.122 0.000 0.000 0.030 0.157 0.357 0.507 0.546 0.181 0.080 0.081 0.095 77.59% (360) 14.22% (66) 8.19% (38) 38.15% (177) 10.34% (48) 68.75% (319) 84.27% (391) 48.71% (226) 53.02% (246) 58.41% (271) 17.03% (79) 7.11% (33) 47.84% (222) 30.60% (142) 11.42% (53) 34.91% (162) 28.02% (130) 11.42% (53) 49 Table 5 (cont’d) Firearm use of violence against others Threat of violence against others Firearm threat of violence against others Mass shooting/multiple-victim target threat Alcohol use Substance use Irrational or erratic behaviors Involuntary mental health examination under authority of the Baker Act 5.81% (27) 28.02% (130) 22.84% (106) 0.86% (40) 26.94% (125) 6.90% (32) 13.36% (62) 58.84% (273) 3.25% (4) 24.39% (30) 18.70% (23) 11.38% (14) 29.27% (36) 7.32% (9) 13.01% (16) 74.80% (92) Committed a felony with a firearm Arrested Officers told someone at the PE about ERPOs Officers wrote they would seek an ERPO but did 27.16% (126) 37.72 (175) 12.07% (56) 25.65% (119) 20.33% (25) 30.89% (38) 22.76% (28) 30.08% (37) not report telling someone at the PE 6.74% (23) 29.33% (100) 24.34% (83) 7.62% (26) 26.10% (89) 6.74% (23) 13.49% (46) 53.08% (181) 29.62% (101) 40.18% (137) 8.21% (28) 24.05% (82) 0.09 1.63 1.62 0.46 0.05 0.02 17.60 3.95 3.32 18.04 1.73 0.183 0.296 0.201 0.203 0.497 0.830 0.893 0.000 0.047 0.069 0.000 0.189 Number of months since MSDHSPSA: Mean (SD) 17.11 (7.59) 0.017 a Chi-square tests were conducted for all predictors except for the following: a Wilcoxon rank-sum test was used for age and the number of months since MSDHSPSA was enacted since these variables were not normally distributed, and a fisher’s exact test was used for firearm use against others, as the cell size <5. 16.61 (7.59) 18.51 (7.44) -2.40 50 Supervised Machine Learning Models Table 6 shows the predictive performance measures for each of the supervised machine learning models. The models ranged in accuracy from 0.49-0.75 when testing their prediction on the testing dataset. In other words, the models accurately predicted if families would reduce access after police arrived in 49%-75% of the cases in the testing dataset. The sensitivity of the models was frequently low, and ranged from 0.12-0.80, meaning that the models correctly identified 12%-80% of cases in which families reduced the respondent’s access to firearms after police arrived. The Naïve Bayes model resulted in the highest sensitivity (0.80). The specificity of the models was higher than the sensitivity for every model except the Naïve Bayes model and ranged from 0.38-0.97. In other words, the models correctly identified 38%-97% of cases in which the family did not reduce access to firearms after police arrived. The Support Vector Machine model resulted in the highest specificity (0.97). Table 6 Performance Metrics of Supervised Machine Learning Prediction Models Accuracy Sensitivity Specificity Accuracy 95% Confidence Interval 0.64 Elastic Net Random Forest 0.70 Stochastic Gradient Boosting 0.66 0.75 Support Vector Machine 0.49 Naïve Bayes 0.66 Ensemble (0.54, 0.74) (0.60, 0.79) (0.55, 0.75) (0.64, 0.83) (0.39, 0.59) (0.55, 0.75) 0.56 0.20 0.24 0.12 0.80 0.32 0.67 0.88 0.83 0.97 0.38 0.78 Ensembled Model The predictive accuracy of the ensembled model was 0.66 (95% CI:0.55-0.75), meaning that the ensembled model accurately predicted whether families reduced access to firearms after police arrived in 66% of cases in the testing dataset. Like many of the individual models, the sensitivity of the ensembled model was low, and when families did reduce access to firearms 51 after police arrived, the ensembled model correctly predicted this 32% of the time. The specificity of the ensembled model was higher, and when family and partners did not reduce firearm access after police arrived, the model correctly predicted this 78% of the time. The ensembled model was used to calculate the variable importance metrics. The ensembled model had higher sensitivity measures than the Random Forest, Stochastic Gradient Boosting, and Support Vector Machine models, and although the Naïve Bayes and Elastic Net model had higher sensitivity measures than the ensembled model, these models had lower overall accuracies. The importance of each variable was calculated for each model and then averaged by the weight of the overall ensembled model to create an overall importance metric (Deane-Mayer & Knowles, 2023). Variable Importance Metrics The variables with the highest importance metrics were the most powerful predictors in the ensembled model. The six predictor variables in the ensembled model with the highest overall importance scores were, in order (Figure 2): the respondent’s age, the time since the MSDPHSA was implemented, if the respondent had more than one firearm, if a family member or partner lived with the respondent, if the respondent was subject to an involuntary examination (and therefore police were allowed to remove firearms), and if police mentioned ERPOs to someone at the PE. These variable importance scores do not convey information about the nature of the relationship between the predictors and families reducing firearm access after police arrived (e.g., positive or negative association); they simply measure the contribution of each variable to the model’s predictive performance (Kuhn & Johnson, 2016). 52 Figure 2 Variable Importance Metrics for Ensembled Model Summary of Findings In bivariate hypothesis tests, several respondent and PE characteristics (including actions by police) were significantly associated with families reducing firearm access after police arrived. Families were more likely to reduce firearm access after police arrived if the respondent had a long gun or more than one gun, lived with families, and exhibited suicide ideation, threats, or plans during the PE. Police actions and their authority to remove firearms were also significant: Families were more likely to reduce firearm access if officers told someone about the ERPO process during the PE and if the respondent was subject to an involuntary examination during the PE (which allows officers to remove firearms) and less likely to reduce firearm access if the respondent committed a felony with a firearm during the PE (which also allows officers to remove firearms). Families were also more likely to reduce firearm access after police arrived as the number of months increased since the MSDHSPSA was implemented. 53 The variables most important in the ensembled model’s prediction of whether families reduced firearm access after police arrived in the testing dataset were identified. The most powerful predictors were the respondent’s age, the number of months since the MSDHSPSA was implemented, if the respondent had more than one firearm, if the respondent was subject to an involuntary examination during the PE (which allows officers to remove firearms), and if officers told someone at the PE about ERPOs. These variable importance metrics simply measure the relative importance in making the model’s prediction and do not convey specific information about the nature of the relationship between the predictor and families reducing firearm access after police arrived. In predicting whether families reduced firearm access after police arrived in the testing dataset, the machine learning models (including the ensembled model) often had low sensitivity, indicating that there are likely important predictors that were not included in this study. 54 CHAPTER 5: SUMMARY, IMPLICATIONS, AND FUTURE SCHOLARSHIP Introduction Family members and intimate partners (hereby called families) are in a unique position to prevent firearm injury when someone is at risk of harming themselves or others. First, families may notice suicide warning signs because they have close and frequent contact with the person who is at risk (Rudd, 2008) or may be victims of the individual’s threats or violence. Second, families may be firearm owners or know where firearms are stored in the home (Okoro et al., 2005; Sorenson, 2017). These warning signs, close contact, and knowledge of firearm storage may provide families the opportunity to reduce firearm access, thereby preventing firearm injury. However, we do not know how or when families try to reduce firearm access for adults at risk for harming themselves or others, or if these efforts are safe, legal, and effective. This dissertation begins to answer these important questions. Using police reports from events that precipitated police petitioning for an ERPO (precipitating event or PE), a content analysis identified and characterized families’ efforts to reduce firearm access at three time- points: before the PE, during the PE, and after police arrived at the PE. Next supervised machine learning identified the most important respondent and PE characteristics in predicting whether families reduced firearm access after police arrived at the PE. This section summarizes key findings, implications for practice and theory, study limitations, and suggestions for future research. Key Findings Families reduced their loved one’s access to firearms in numerous ways. Some families tried to prevent the respondent from accessing a firearm before the PE occurred, with a small number of police reports (2.54%) indicating such efforts. Fifteen percent of reports included 55 descriptions of families’ attempts to keep firearms away from the respondent during the PE. Families hid or moved the firearms (sometimes in places where others could access them), asked the respondent for the firearms, and used physical force to keep them away from the respondent, but were not always successful and sometimes faced retaliation for trying to keep the respondent from firearms. Twenty-seven percent of reports detailed families’ efforts to reduce firearm access after police officers arrived at the PE. Families gave officers consent and access to homes and property to search for and remove firearms. Families also asked officers for help keeping firearms away from the respondent and gave officers firearms. Finally, families told officers that they would continue to make firearms inaccessible after officers left, including by temporarily storing guns outside of the respondent’s home for safekeeping. The ensembled supervised machine learning model identified the most influential respondent and PE characteristics in predicting whether families reduced access after officers arrived at the PE. Two of the most influential variables were if the respondent was subjected to an involuntary examination during the PE (which allows police to remove firearms) and if officers told someone at the PE about the ERPO process or that they were going to file for an ERPO. Other influential variables included the respondent’s age, if the respondent lived with families or partners, if the respondent had access to or possession of more than one firearm, and the number of months since Florida allowed officers to file ERPO petitions and remove firearms when taking someone for an involuntary examination. The sensitivity of the ensembled model and other supervised machine learning models was low, indicating that there are likely other influential variables in predicting families reducing firearm access after police arrive that were not included in this study. 56 Implications for Practice Firearm Safety Messaging Needed for Families This dissertation found evidence that families sometimes allowed firearms to get into new hands. Families sometimes stored their own firearms in ways that allowed the respondent access during the PE (e.g., unlocked in cars). Families also hid firearms from the respondent in locations that could be easily accessed by other household members, including children. Non- household family members sometimes temporarily took firearms to their own homes, and it was unclear who would have access at the new location. Putting guns into new hands or storing them so that others have access can increase the risk of firearm injury. Access to firearms is associated with increased risk of suicide, including for adolescents and those who do not own firearms but simply live in a house with firearms (Anglemyer et al., 2014b; Miller et al., 2022). Unsafe storage practices are also associated with unintentional firearm injuries for child and adolescents (Grossman et al., 2005). Therefore, putting firearms in unsecured places could put someone new at risk of firearm injury. To prevent the person at-risk and for others around them from firearm injury, families need firearm safety messaging. Firearm safety messaging is designed to educate families about safe storage methods and encourage someone who is an acute suicidal or homicidal crisis not to have access to firearms (McGee, 2003; I. H. Stanley et al., 2017). This messaging might persuade firearm-owning family members to keep their guns away from the at-risk person, so firearms are unavailable during a crisis moment. If the firearm owner themselves is the at-risk person, firearm safety messaging can help families work with that person to temporarily remove firearms from the home before a crisis moment. For families who temporarily store the at-risk 57 person’s firearms, safe storage education may help them store new firearms safely and not unintentionally give others access. However, current firearm safety messaging for families has several limitations. First, families may not receive firearm safety messaging, including during times of increased risk, and may not trust those delivering the messaging. Healthcare providers have the potential to deliver firearm safety messages: they see patients with homicidal or suicidal ideation, serious mental illness, or who are at increased risk for firearm-related injury (Wintemute et al., 2016) and may discuss other safety messages with patients (e.g., seatbelt use and smoking cessation) (Betz et al., 2016). Although health organizations recommend healthcare providers screen patients for firearm access and offer assistance as appropriate (Weinberger et al., 2015), healthcare providers rarely do (Roszko et al., 2016), even when a patient has screened positive for suicide risk (Dobscha et al., 2014). Even if healthcare professionals attempt to counsel on firearm safety, they are consistently rated as poor messengers (Aitken et al., 2020; Anestis et al., 2021; Crifasi, Doucette, et al., 2018b). Healthcare providers are sometimes viewed as knowing little about firearms and gun ownership culture (Knoepke et al., 2017), and owners may not want physicians to judge them on the number of guns they have (Aitken et al., 2020). Almost half of firearm owners and a third of non-firearm owners who live with firearm owners believe it is never acceptable for healthcare providers to bring up firearms (Betz et al., 2016). This indicates that some families may not receive or trust firearm safety messaging from healthcare providers, and that families need additional messengers who have more credibility and who can promote safety during times of heightened risk of firearm injury. The second limitation of current firearm safety messaging is the lack of specific, systematic guidance on how to intervene when families learn that their loved one’s access to 58 firearms can be dangerous. Veterans Affairs providers noted that guidance for clinicians is not systematic, resulting in clinicians’ ad hoc efforts to work with clients and their families to reduce firearm access (e.g., identifying who in the family can take firearms) (Walters et al., 2012). The 988 Suicide and Crisis Lifeline recommends that loved ones try to reduce access to lethal means, but it does not provide specific guidance on how to have conversations with loved ones about the risk of firearm access, how to remove firearms, and what to do with the firearms after they have been removed (988 Suicide and Crisis Lifeline, n.d.). Families report that the uncertainty of when and how to take action around firearms, including how to broach the topic, makes them reluctant to intervene (Betz et al., 2023). Third, families may not know what their legal options are for keeping firearms away from their loved ones. VA providers, veterans, and veterans’ families report that they sometimes were unsure if firearms could legally be stored somewhere outside of the veteran’s home (Walters et al., 2012). Many states have universal background check laws which require the person receiving the gun to undergo a background check first (McCourt et al., 2017). A few states with background check laws exempt temporary transfers, but the exemption provisions are often vague (McCourt et al., 2017). This dissertation found evidence that respondents and their families gave firearms to others without a background check to reduce the risk of suicide or other firearm injury. These actions were legal because Florida does not require a background check when someone (other than a licensed dealer) transfers a gun to another person; however, the same actions could be illegal in other states depending on the states’ background check laws. Adding to their confusion, families are also unclear about the legal consequences they could face if they do not remove guns from older parents or those with a diagnosis such as Alzheimer’s (Betz et al., 2023). 59 Finally, suggestions to reduce firearm access when someone is at risk of harming themselves or others may not be safe for all families. Some IPV victims believe that removing abusers’ guns would result in retaliation against them (Lynch & Logan, 2018). In this dissertation, respondents sometimes retaliated against families for taking—or trying to take— their gun during the PE. In VA focus groups, clinicians and veterans’ families also had safety concerns for families who temporarily stored guns for veterans that the veterans later wanted back (Walters et al., 2012). Families reducing firearm access when someone is at risk of harming themselves or others may not be safe, and some firearm safety messaging may be more appliable to situations of suicide risk rather than situations of violence. In summary, families in my dissertation sometimes put the respondent, themselves, and others around them at risk for firearm injury, indicating that families may benefit from firearm safety messaging. However, much of the current firearm safety messaging lacks specific guidance, does not address safety or legal concerns, and comes from untrusted messengers. While families are uniquely positioned to intervene, they need additional safety messaging and support from trusted messengers. Law Enforcement Engagement Law enforcement agencies and officers could play a large role in encouraging safe storage and helping families ensure that their loved ones are not able to access firearms when they are at heightened risk for firearm injury. First, law enforcement agencies can provide temporary firearm storage when there is a risk of firearm injury. In this dissertation, some people willingly gave firearms to police or asked them to take firearms for safekeeping. In a survey of agency leaders (Runyan et al., 2017), approximately 65% reported that their agency was asked to provide temporary firearm storage within the past year, and three-quarters reported that they 60 currently offer this service. Most agency leaders reported that they were likely to offer temporary storage if a gun owner was concerned about the mental stability of an adult family member (or themselves) and wanted guns stored outside the home. Even when someone who is not the gun owner wants police to store a family member’s guns, just over half of agency leaders said that they would be likely to provide temporary storage (Runyan et al., 2017). Second, police officers can provide firearm safety recommendations when they respond to a scene, and these may be influential. Law enforcement officers are consistently ranked as preferred messengers in safer storage (Aitken et al., 2020; Anestis et al., 2018; Crifasi, Doucette, et al., 2018a), and ninety-five percent of agency leaders think that officers talking about safe gun storage is acceptable to most people in their communities (Brooks-Russell et al., 2019). In this dissertation, police talking about ERPOs at the scene of the PE was one of the most influential predictors in families reducing firearm access after police arrived. Separately from officers discussing ERPOs at the scene, officers occasionally gave families recommendations for how they could keep themselves or their loved ones safe from firearm injury. For example, one officer told the respondent’s wife to put the gun in a safe place where the respondent would not know where it was or to give it to a family member. Another officer advised a respondent’s husband, who owned the firearm, to change the lock combination or remove the firearm from the residence. Officers can provide these informative and possibly influential firearm safety messages—in addition to explaining ERPOs—when responding to an incident and talking with families at the scene. Third, law enforcement officers can seize firearms at the scene of the incident (depending on state law and what occurred during the incident), and families can assist during this seizure. In this study, families gave police consent or access to remove firearms, sometimes giving specific 61 information on how many firearms were in the home, what type of firearms they were, and where they were located. Families also provided keys or combinations to safes or other storage locations which allowed police to remove firearms. Even though law enforcement can play a large role in supporting families who are trying to prevent firearm injury (and did so in this dissertation), a variety of concerns currently inhibit police involvement and deserve future study. Approximately a quarter of agencies do not offer temporary storage for firearm owners (Brooks-Russell et al., 2019), and these agencies report numerous barriers, including a lack of personnel time, not having enough physical space to store firearms, and potential legal liability of storing and returning firearms (Brooks-Russell et al., 2019). Future research should identify how agencies that offer temporary storage were able to overcome these barriers. Additionally, research is needed to understand if officers’ firearm storage beliefs and personal practices are associated with their firearm safety recommendations or offers of temporary storage. A survey of one agency’s officers found that 59% of officers stored their weapons at home unlocked, 68% stored them loaded, and 44% reported storing them unlocked and loaded (Coyne-Beasley et al., 2001). Additionally, a majority of agency leaders believe that a suicidal individual will find a way to end their life, no matter what anyone tells them (Brooks-Russell et al., 2019). These personal storage habits and beliefs may prevent some officers from fully working with families who also store firearms unsafely or in incidents where suicide is a concern. Future research should explore how individual police officer views on firearm injury and safe storage affect their firearm safety recommendations or offers of temporary storage. Finally, although large majorities of civilians report that police are preferred messengers on firearm safety, not all express that view (Anestis et al., 2018; Crifasi, Doucette, et al., 2018a). In a national study, 23% of those surveyed did not believe police to be excellent or 62 good messengers (Crifasi, Doucette, et al., 2018a). Moreover, firearm owners who are Black (compared to white), store their firearms in a gun safe (compared to owners who do not store firearms in a gun safe), and owners who store their firearms locked (compared to owners who do not store their firearms locked) are significantly less likely to view police as credible messengers for suicide prevention. However, Black firearm owners and owners who store their firearms in a gun safe and/or locked still rank officers as the most credible messengers on firearm safety to prevent suicide (Anestis et al., 2021). Future research should explore if civilians want firearm safety messages at the scene of an incident, and if this varies across demographics, firearm storage, and type of incident (e.g., suicide vs interpersonal violence). In summary, police officers are commonly seen as trusted firearm safety messengers and can provide temporary firearm storage, give firearm safety recommendations at the scene of an incident, and seize firearms depending on state law and incident characteristics. This dissertation suggests that police officers can support families in preventing firearm injury when their loved ones are at heightened risk for injuring themselves or others. Just over a quarter of police reports included information that families reduced firearm access after police arrived. Additionally, the respondent being subject to an involuntary examination at the PE (which allows officers to remove firearms) and officers talking about ERPOs at the PE were influential in predicting whether families would reduce firearm access after police arrived. However, the potential for police involvement in providing firearm safety messaging and support may be limited, as agencies and officers report structural and attitudinal barriers, and the perception of police as good messengers is not universal. More research is needed to fully understand law enforcement’s role in promoting firearm safety. 63 Implications for Theory The supervised machine learning models had low sensitivity, meaning that when families reduced their loved one’s access to firearms after police arrived, the models only correctly predicted this a small amount of the time. This low sensitivity suggests that there are other important variables that predict families reducing firearm access after police arrive, but they were not included in this study. Variable selection was limited by what officers included in their reports, however, by conducting future research and using theory to explore families’ behaviors, researchers can identify variables that they can systematically be collected and tested in future models. The Theory of Planned Behavior is one theory that could guide future research with families who are firearm owners or who live in a house with firearms. The Theory of Planned Behavior [TPB] suggests that families’ intentions to prevent their loved ones from accessing a gun drives families taking action to do so, and that this intention is driven by families’ three beliefs: that preventing their loved one from accessing firearms will lead to the desired outcome (e.g., the loved one’s safety), that important others (e.g., extended family members) approve of reducing firearm access, and that they have the ability to reduce firearm access (Ajzen, 1991). The stronger this intention, the more likely families will take action to reduce firearm access (Ajzen, 1991). This theory also suggests that the actual ability (compared to belief in ability) moderates the effect of intention (Ajzen, 1991). Therefore, it is possible that families who have stronger intentions to prevent their loved ones from firearm access are more likely give officers firearms, and that officers’ legal authority to remove firearms or their capability to provide temporary firearm storage moderates families’ giving officers firearms. However, the police reports used in this dissertation study did not include descriptions of families’ attitudes and 64 beliefs. Future studies should explore if officers’ firearm safety advice increases families’ beliefs that others support them in reducing firearm access. Additionally, researchers should explore if police officers offer temporary firearm storage at the scene of an incident increases families’ perceived ability to reduce their loved one’s access to firearms. Researchers should also explore the role that police legitimacy has in families reducing firearm access after police arrive. Police legitimacy is the belief that officers are legitimate members of authority, and that acting in this authority, officers can take actions to maintain social order and decrease crime (Tyler, 2004). Civilians feel responsible for following this authority and therefore cooperate with police directives (Tyler, 2004). Indeed, police legitimacy influences whether civilians cooperate with and assist the police (Sunshine & Tyler, 2003). In this dissertation, one of the most important predictors of families reducing access was the police taking the respondent for an involuntary examination, which allows police to remove firearms. It is possible that families trusted police officers and felt obliged to cooperate with and assist them, and therefore gave officers firearms or consent to search and remove firearms. Additionally, one predictor of police legitimacy is procedural justice, the view that police acted in an unbiased, transparent, and respectful way (Tyler, 2004). Before the police arrived at the scene, the respondent had threatened or used violence against themselves or others in a large majority of cases, and it is possible that families felt the police acted appropriately by taking the respondent in for an involuntary examination, and this increased their view of police legitimacy and the likelihood of reducing firearm access. Future research should explore if families’ trust in police, their obligation to obey police, and their view of how police acted during the incident impacts their efforts to reduce firearm access with police. 65 Limitations and Future Scholarship Document Context In content analyses, the texts are contexts of their original use (Krippendorff, 2019), and officers’ reports in this study included succinct descriptions of how they responded to the scene, what they observed at the scene, and what witnesses, victims, and defendants told them. These police reports did not contain a detailed description of everything that occurred before the PE, during the PE, and after officers arrived at the PE, and it is likely that the frequencies of families’ efforts to reduce firearm access and PE characteristics in this dissertation were an underestimation of their true frequencies. For example, some families may have reduced firearm access before the PE but did not tell officers or told officers, but officers did not include this in their reports. Additionally, police reports did not contain descriptions of families’ attitudes and beliefs—including those on police legitimacy and if others support reducing firearm access— limiting the theoretical interpretation of why families’ reduced firearm access. Future research should conduct in-depth interviews with families after police officers have left a scene where someone was at risk of harming themselves or others. In these interviews, researchers should explore the context around families reducing firearm access and how families made sense of their and officers’ actions around firearms. These interviews identify constructs (e.g., trust in police officers) to quantitatively model in future analyses. Differences across Law Enforcement Agencies and Counties The police reports came from ERPO cases that were filed by law enforcement agencies across Florida. Some agencies submitted only one or a few ERPOs during the study period, which created issues of near-zero variance in the supervised machine learning models and prevented agency and county from being included as predictors. However, it is possible that 66 agency and county are important predictors. Not all agencies offer temporary firearm storage (Runyan et al., 2017), therefore, it is possible that families were less likely to give firearms to police for safekeeping if the agency did not offer temporary firearm storage. County may also be an influential predictor because county leadership can spur ERPO training and development of ERPO protocols for law enforcement and other criminal justice officials (Pear et al., 2021). As officers talking about ERPOs at the scene was an important predictor in the supervised machine learning models, it is possible that families were more likely to reduce firearm access after police arrived in counties where police had received education on the risks of firearm injury when someone is at risk of harming themselves or others. Future research predicting families’ efforts to reduce firearm access after police arrive should use a sample size large enough to test for agency and county differences. Construct validity There are two potential threats to construct validity. First, it is possible that I did not identify all efforts to reduce firearm access in the reports. Second, because of the sample size, I had to collapse all efforts after police arrived into a single binary construct, and it is possible that this construct was too general of an explication and more meaningful inferences might have been drawn if each effort after police arrived was modeled separately. For example, families giving police consent to search for firearms might be theoretically different from families who gave police firearms, and different predictor variables might have been important for these different family efforts. Future research should examine family efforts in larger samples to allow for different outcome analyses. 67 External validity The frequency of families’ efforts to reduce firearm access and the most important predictors may not hold when studying other incidents when someone is at risk of harming themselves or others. This dissertation study used police reports from ERPO casefiles, and there are likely differences between these incidents and incidents in which someone is at risk of harming themselves or others and police are not called. There are numerous reasons why families do not call the police for assistance, including fear of discrimination by police and retaliation by the at-risk person (Ammar et al., 2005; Langton et al., 2012; Potter, 2008; Reaves, 2017), which could also affect whether families try to reduce firearm access. For example, IPV victims report that reducing their partners’ firearm access could result in retaliation (Lynch & Logan, 2018), so families who do not call the police for fear of retaliation by the at-risk person may also not reduce firearm access. There are also likely differences between the police reports submitted for ERPOs that were used in this study and police reports that were not submitted for ERPOs. It is possible that officers were more likely to petition for an ERPO after responding to an incident where families were trying to reduce firearm access, however, it is also possible that a lack of family efforts to reduce firearm access inspired officers to file for an ERPO to prevent firearm injury. Additionally, some ERPO casefiles in my dissertation included multiple police reports (one from each of the officers who responded to the PE) while some casefiles included a single report, and it is unknown if all possible reports were included in the ERPO casefile. It is possible that families reduced firearm access with an officer but that officer’s report was not included in the ERPO casefile. Future research that uses police reports to characterize and predict families’ efforts to reduce firearm access should collect police reports directly from police agencies. 68 Finally, the multi-state ERPO study coded the PE characteristics (e.g., suicide ideation) that were used in this dissertation study, and research assistants in the multi-state ERPO study used additional documents in the ERPO casefile (e.g., police officer affidavit) if they were available, in addition to the police report to code PE characteristics. Therefore, it is possible that the frequency of PE characteristics in this dissertation study are higher than if the study had just used police reports to code PE characteristics. Conclusion Despite these limitations, this dissertation study identified and characterized how families reduced firearm access from an adult who was at risk of harming themselves or others, which has not been studied in the literature. This study demonstrated the need for firearm safety education and support for families who are trying to prevent firearm injury, as well as future research to understand police officers’ potential in promoting firearm safety, especially during times of heightened risk of firearm injury. This dissertation study also identified the most important respondent and PE characteristics used by the ensembled model to predict whether families reduced firearm access after police arrived. Future research should examine these influential variables in greater detail to understand how they specifically impact families’ efforts to reduce firearm access. 69 BIBLIOGRAPHY 18 U.S.C. § 922(g)(8). 18 U.S.C. § 922(g)(9). 988 Suicide and Crisis Lifeline. (n.d.). Retrieved May 26, 2023, from https://www.bethe1to.com/bethe1to-steps-evidence/ Adhia, A., Austin, S. B., Fitzmaurice, G. M., & Hemenway, D. (2019). The role of intimate partner violence in homicides of children aged 2–14 years. American Journal of Preventive Medicine, 56(1), 38–46. https://doi.org/10.1016/j.amepre.2018.08.028 Adhia, A., Lyons, V. H., Moe, C. A., Rowhani-Rahbar, A., & Rivara, F. P. (2021). Nonfatal use of firearms in intimate partner violence: Results of a national survey. Preventive Medicine, 106500. https://doi.org/10.1016/j.ypmed.2021.106500 Aitken, M. E., Minster, S. D., Mullins, S. H., Hirsch, H. M., Unni, P., Monroe, K., & Miller, B. K. (2020). Parents’ perspectives on safe storage of firearms. Journal of Community Health, 45(3), 469–477. https://doi.org/10.1007/s10900-019-00762-2 Ajzen, I. (1991). The theory of planned behavior. Organizational Behavior and Human Decision Processes, 50(2), 179–211. https://doi.org/10.1016/0749-5978(91)90020-T Ajzen, I. (2020). The theory of planned behavior: Frequently asked questions. Human Behavior and Emerging Technologies, 2(4), 314–324. https://doi.org/10.1002/hbe2.195 American Academy of Pediatrics. (n.d.). 12 Things Parents Can Do to Help Prevent Suicide. HealthyChildren.Org. Retrieved August 12, 2022, from https://www.healthychildren.org/English/health-issues/conditions/emotional- problems/Pages/Ten-Things-Parents-Can-Do-to-Prevent- Suicide.aspx?_ga=2.86532888.1821171995.1660304200- 1934161048.1660304200&_gl=1*en4soh*_ga*MTkzNDE2MTA0OC4xNjYwMzA0Mj Aw*_ga_FD9D3XZVQQ*MTY2MDMwNDIwMC4xLjAuMTY2MDMwNDIwMC4w American Medical Association. (n.d.). AMA adopts policy to address increases in youth suicide and save lives. Retrieved June 1, 2022, from https://www.ama-assn.org/press- center/press-releases/ama-adopts-policy-address-increases-youth-suicide-and-save- lives#:~:text=AMA%20policy%20also%20supports%20lethal,people%20from%20dying %20by%20suicide. Ammar, N. H., Orloff, L. E., Dutton, M. A., & Aguilar-Hass, G. (2005). Calls to Police and Police Response: A Case Study of Latina Immigrant Women in the USA. International Journal of Police Science & Management, 7(4), 230–244. https://doi.org/10.1350/ijps.2005.7.4.230 70 Anestis, M. D., Bond, A. E., Bryan, A. O., & Bryan, C. J. (2021). An examination of preferred messengers on firearm safety for suicide prevention. Preventive Medicine, 145, 106452. https://doi.org/10.1016/j.ypmed.2021.106452 Anestis, M. D., Butterworth, S. E., & Houtsma, C. (2018). Perceptions of firearms and suicide: The role of misinformation in storage practices and openness to means safety measures. Journal of Affective Disorders, 227, 530–535. https://doi.org/10.1016/j.jad.2017.11.057 Anglemyer, A., Horvath, T., & Rutherford, G. (2014a). The accessibility of firearms and risk for suicide and homicide victimization among household members: A systematic review and meta-analysis. Annals of Internal Medicine, 160(2), 101–110. https://doi.org/10.7326/M13-1301 Anglemyer, A., Horvath, T., & Rutherford, G. (2014b). The accessibility of firearms and risk for suicide and homicide victimization among household members: A systematic review and meta-analysis. Annals of Internal Medicine, 160(2), 101–110. https://doi.org/10.7326/M13-1301 Azrael, D., & Miller, M. J. (2016). Reducing Suicide Without Affecting Underlying Mental Health. In R. C. O’Connor & J. Pirkis (Eds.), The International Handbook of Suicide Prevention (pp. 637–662). John Wiley & Sons, Ltd. https://doi.org/10.1002/9781118903223.ch36 Bagge, C. L., Littlefield, A. K., Wiegand, T. J., Hawkins, E., Trim, R. S., Schumacher, J. A., Simons, K., & Conner, K. R. (2022). A controlled examination of acute warning signs for suicide attempts among hospitalized patients. Psychological Medicine, 1–9. https://doi.org/10.1017/S0033291721004712 Barber, C. W., & Miller, M. J. (2014). Reducing a suicidal person’s access to lethal means of suicide: A research agenda. American Journal of Preventive Medicine, 47(3, Supplement 2), S264–S272. https://doi.org/10.1016/j.amepre.2014.05.028 Battered Women’s Justice Project. (n.d.). S.A.F.E. Tool. Retrieved August 21, 2022, from https://www.preventdvgunviolence.org/assets/documents/downloadable- resources/ncpoffc-safe-tool-2016.pdf Betz, M. E., Azrael, D., Barber, C., & Miller, M. (2016). Public opinion regarding whether speaking with patients about firearms is appropriate: Results of a national survey. Annals of Internal Medicine, 165(8), 543. https://doi.org/10.7326/M16-0739 Betz, M. E., Barber, C., & Miller, M. (2011). Suicidal behavior and firearm access: Results from the second injury control and risk survey. Suicide and Life-Threatening Behavior, 41(4), 384–391. https://doi.org/10.1111/j.1943-278X.2011.00036.x Betz, M. E., Polzer, E. R., Knoepke, C. E., Prater, L. C., Simonetti, J. A., Lee, T. M., Meador, L. E., & Nearing, K. A. (2023). Cars, guns, aging, and “giving up the keys.” The Gerontologist, 63(4), 717–730. https://doi.org/10.1093/geront/gnac142 71 Boehmke, B., & Greenwell, B. M. (2020). Hands-On Machine Learning with R. Taylor & Francis Group. https://www-taylorfrancis-com.proxy1.cl.msu.edu/pdfviewer/ Brooks-Russell, A., Runyan, C., Betz, M. E., Tung, G., Brandspigel, S., & Novins, D. K. (2019). Law enforcement agencies’ perceptions of the benefits of and barriers to temporary firearm storage to prevent suicide. American Journal of Public Health, 109(2), 285–288. https://doi.org/10.2105/AJPH.2018.304802 Bzdok, D., Altman, N., & Krzywinski, M. (2018). Statistics versus machine learning. Nature Methods, 15(4), 233–234. https://doi.org/10.1038/nmeth.4642 Campbell, J. C., Glass, N., Sharps, P. W., Laughon, K., & Bloom, T. (2007). Intimate partner homicide: Review and implications of research and policy. Trauma, Violence, & Abuse, 8(3), 246–269. https://doi.org/10.1177/1524838007303505 Campbell, J. C., Webster, D., Koziol-McLain, J., Block, C., Campbell, D., Curry, M. A., Gary, F., Glass, N., McFarlane, J., Sachs, C., Sharps, P., Ulrich, Y., Wilt, S. A., Manganello, J., Xu, X., Schollenberger, J., Frye, V., & Laughon, K. (2003). Risk factors for femicide in abusive relationships: Results from a multisite case control study. American Journal of Public Health, 93(7), 1089–1097. https://doi.org/10.2105/AJPH.93.7.1089 CDC. (2023, May 19). Web-based Injury Statistics Query and Reporting System (WISQARS). https://www.cdc.gov/injury/ wisqars Conner, A., Azrael, D., & Miller, M. (2019). Suicide case-fatality rates in the United States, 2007 to 2014. Annals of Internal Medicine, 171(12), 885–895. https://doi.org/10.7326/M19-1324 Conner, A., Azrael, D., & Miller, M. (2021). Firearm safety discussions between clinicians and u.s. adults living in households with firearms: Results from a 2019 national survey. Annals of Internal Medicine, 174(5), 725–728. https://doi.org/10.7326/M20-6314 Coyne-Beasley, T., Johnson, R. M., Charles, L. E., & Schoenbach, V. J. (2001). Firearm storage practices of officers in a law enforcement agency in the south. American Journal of Preventive Medicine, 21(2), 118–123. https://doi.org/10.1016/S0749-3797(01)00327-0 Crifasi, C. K., Doucette, M. L., McGinty, E. E., Webster, D. W., & Barry, C. L. (2018a). Storage Practices of US Gun Owners in 2016. American Journal of Public Health, 108(4), 532– 537. https://doi.org/10.2105/AJPH.2017.304262 Crifasi, C. K., Doucette, M. L., McGinty, E. E., Webster, D. W., & Barry, C. L. (2018b). Storage Practices of US Gun Owners in 2016. American Journal of Public Health, 108(4), 532– 537. https://doi.org/10.2105/AJPH.2017.304262 Crifasi, C. K., Merrill-Francis, M., McCourt, A., Vernick, J. S., Wintemute, G. J., & Webster, D. W. (2018). Correction to: Association between firearm laws and homicide in urban 72 counties. Journal of Urban Health, 95(5), 773–776. https://doi.org/10.1007/s11524-018- 0306-y Crifasi, C. K., Meyers, J. S., Vernick, J. S., & Webster, D. W. (2015). Effects of changes in permit-to-purchase handgun laws in Connecticut and Missouri on suicide rates. Preventive Medicine, 79, 43–49. https://doi.org/10.1016/j.ypmed.2015.07.013 de Moore, G. M., Plew, J. D., Bray, K. M., & Snars, J. N. (1994). Survivors of self-inflicted firearm injury. Medical Journal of Australia, 160(7), 421–425. https://doi.org/10.5694/j.1326-5377.1994.tb138267.x Deane-Mayer, Z. A., & Knowles, J. E. (2023). Package “caretEnsemble.” https://cran.r- project.org/web/packages/caretEnsemble/caretEnsemble.pdf Degli Esposti, M., Wiebe, D. J., Gasparrini, A., & Humphreys, D. K. (2022). Analysis of “stand your ground” self-defense laws and statewide rates of homicides and firearm homicides. JAMA Network Open, 5(2), e220077. https://doi.org/10.1001/jamanetworkopen.2022.0077 Deisenhammer, E. A., Ing, C.-M., Strauss, R., Kemmler, G., Hinterhuber, H., & Weiss, E. M. (2008). The duration of the suicidal process: How much time is left for intervention between consideration and accomplishment of a suicide attempt? The Journal of Clinical Psychiatry, 69(1), 5230. https://doi.org/10.4088/JCP.07m03904 Díez, C., Kurland, R. P., Rothman, E. F., Bair-Merritt, M., Fleegler, E., Xuan, Z., Galea, S., Ross, C. S., Kalesan, B., Goss, K. A., & Siegel, M. (2017). State intimate partner violence–related firearm laws and intimate partner homicide rates in the United States, 1991 to 2015. Annals of Internal Medicine, 167(8), 536. https://doi.org/10.7326/M16- 2849 Dobscha, S. K., Denneson, L. M., Kovas, A. E., Corson, K., Helmer, D. A., & Bair, M. J. (2014). Primary care clinician responses to positive suicidal ideation risk assessments in veterans of Iraq and Afghanistan. General Hospital Psychiatry, 36(3), 310–317. https://doi.org/10.1016/j.genhosppsych.2013.11.007 Donohue, J. J., Aneja, A., & Weber, K. D. (2019). Right-to-carry laws and violent crime: A comprehensive assessment using panel data and a state-level synthetic control analysis. Journal of Empirical Legal Studies, 16(2), 198–247. https://doi.org/10.1111/jels.12219 Drum, D. J., Brownson, C., Burton Denmark, A., & Smith, S. E. (2009). New data on the nature of suicidal crises in college students: Shifting the paradigm. Professional Psychology: Research and Practice, 40(3), 213–222. https://doi.org/10.1037/a0014465 Eddleston, M., Karunaratne, A., Weerakoon, M., Kumarasinghe, S., Rajapakshe, M., Rezvi Sheriff, M. H., Buckley, N. A., & Gunnell, D. (2006). Choice of poison for intentional self-poisoning in rural sri lanka. Clinical Toxicology, 44(3), 283–286. https://doi.org/10.1080/15563650600584444 73 Edwards, K. M., Gidycz, C. A., & Murphy, M. J. (2015). Leaving an abusive dating relationship: A prospective analysis of the investment model and theory of planned behavior. Journal of Interpersonal Violence, 30(16), 2908–2927. https://doi.org/10.1177/0886260514554285 Fla. Stat. §790.401. Flach, P. (2012). Machine Learning: The Art and Science of Algorithms That Make Sense of Data. Cambridge University Press. Fridel, E. E., & Fox, J. A. (2019). Gender differences in patterns and trends in U.S. homicide, 1976–2017. Violence and Gender, 6(1), 27–36. https://doi.org/10.1089/vio.2019.0005 Gould, M. S., Lake, A. M., Kleinman, M., Galfalvy, H., & McKeon, R. (2022). Third-party callers to the national suicide prevention lifeline: Seeking assistance on behalf of people at imminent risk of suicide. Suicide and Life-Threatening Behavior, 52(1), 37–48. https://doi.org/10.1111/sltb.12769 Grossman, D. C., Mueller, B. A., Riedy, C., Dowd, M. D., Villaveces, A., Prodzinski, J., Nakagawara, J., Howard, J., Thiersch, N., & Harruff, R. (2005). Gun storage practices and risk of youth suicide and unintentional firearm injuries. JAMA, 293(6), 707–714. https://doi.org/10.1001/jama.293.6.707 Houtsma, C., Butterworth, S. E., & Anestis, M. D. (2018). Firearm suicide: Pathways to risk and methods of prevention. Current Opinion in Psychology, 22, 7–11. https://doi.org/10.1016/j.copsyc.2017.07.002 Johnson, R. M., Frank, E. M., Ciocca, M., & Barber, C. W. (2011). Training mental healthcare providers to reduce at-risk patients’ access to lethal means of suicide: Evaluation of the calm project. Archives of Suicide Research, 15(3), 259–264. https://doi.org/10.1080/13811118.2011.589727 Johnson, R. M., Runyan, C. W., Coyne-Beasley, T., Lewis, M. A., & Bowling, J. M. (2008). Storage of household firearms: An examination of the attitudes and beliefs of married women with children. Health Education Research, 23(4), 592–602. https://doi.org/10.1093/her/cym049 Kivisto, A. J., & Phalen, P. L. (2018). Effects of risk-based firearm seizure laws in Connecticut and Indiana on suicide rates, 1981–2015. Psychiatric Services, 69(8), 855–862. https://doi.org/10.1176/appi.ps.201700250 Knoepke, C. E., Allen, A., Ranney, M. L., Wintemute, G. J., Matlock, D. D., & Betz, M. E. (2017). Loaded Questions: Internet Commenters’ Opinions on Physician-Patient Firearm Safety Conversations. Western Journal of Emergency Medicine, 18(5), 903–912. https://doi.org/10.5811/westjem.2017.6.34849 74 Koziol-McLain, J., Webster, D., McFarlane, J., Block, C. R., & al, et. (2006). Risk factors for femicide-suicide in abusive relationships: Results from a multisite case control study. Violence and Victims, 21(1), 3–21. https://doi.org/10.1891/088667006780927358 Krippendorff, K. (2019). Content Analysis: An Introduction to Its Methodology (4th ed.). SAGE Publications, Inc. Kruesi, M. J. P., Grossman, J., Pennington, J. M., Woodward, P. J., Duda, D., & Hirsch, J. G. (1999). Suicide and violence prevention: Parent education in the emergency department. Journal of the American Academy of Child & Adolescent Psychiatry, 38(3), 250–255. https://doi.org/10.1097/00004583-199903000-00010 Kuhlman, S. T. W., Walch, S. E., Bauer, K. N., & Glenn, A. D. (2017). Intention to enact and enactment of gatekeeper behaviors for suicide prevention: An application of the theory of planned behavior. Prevention Science, 18(6), 704–715. https://doi.org/10.1007/s11121- 017-0786-0 Kuhn, M. (2008). Building predictive models in r using the caret package. Journal of Statistical Software, 28, 1–26. https://doi.org/10.18637/jss.v028.i05 Kuhn, M., & Johnson, K. (2016). Applied Predictice Modeling. Springer. Langton, L., Statistician, B., Berzofsky, M., Krebs, C., & Smiley-McDonald, H. (2012). Victimizations Not Reported to the Police, 2006-2010. Li, X., Xu, Y., Wang, Y., Yang, R., Zhang, C., Ji, H., Bian, Q., Ma, Z., He, F., & Fei, L. (2002). Characteristics of serious suicide attempts treated in general hospitals. Chinese Mental Health Journal, 16(10), 681–684. Logan, T. K., Link to external site, this link will open in a new window, & Lynch, K. (2021). Exploring abuser firearm-related attitudes, behaviors, and threats among women with (ex)partners who threatened to shoot others. Journal of Threat Assessment and Management. Lubin, G., Werbeloff, N., Halperin, D., Shmushkevitch, M., Weiser, M., & Knobler, H. Y. (2010). Decrease in suicide rates after a change of policy reducing access to firearms in adolescents: A naturalistic epidemiological study. Suicide and Life-Threatening Behavior, 40(5), 421–424. https://doi.org/10.1521/suli.2010.40.5.421 Lynch, K. R., Boots, D. P., Jackson, D. B., & Renzetti, C. M. (2021). Firearm-related abuse and protective order requests among intimate partner violence victims. Journal of Interpersonal Violence, 08862605211001474. https://doi.org/10.1177/08862605211001474 Lynch, K. R., & Logan, T. (2018). “You better say your prayers and get ready”: Guns within the context of partner abuse. Journal of Interpersonal Violence, 33(4), 686–711. https://doi.org/10.1177/0886260515613344 75 Lynch, K. R., & Logan, T. (2021). “Always know where the gun is”: Service providers perceptions of firearm access, violence, and safety planning during the covid-19 pandemic. Journal of Interpersonal Violence, 08862605211046270. https://doi.org/10.1177/08862605211046270 Marjory Stoneman Douglas High School Public Safety Act, Fl. ALS 3, 2018 (2018). McCourt, A. D., Vernick, J. S., Betz, M. E., Brandspigel, S., & Runyan, C. W. (2017). Temporary transfer of firearms from the home to prevent suicide: Legal obstacles and recommendations. JAMA Internal Medicine, 177(1), 96–101. https://doi.org/10.1001/jamainternmed.2016.5704 McFarlane, J., Soeken, K., Campbell, J., Parker, B., Reel, S., & Silva, C. (1998). Severity of abuse to pregnant women and associated gun access of the perpetrator. Public Health Nursing, 15(3), 201–206. https://doi.org/10.1111/j.1525-1446.1998.tb00340.x McGee, K. S. (2003). Review of evaluations of educational approaches to promote safe storage of firearms. Injury Prevention, 9(2), 108–111. https://doi.org/10.1136/ip.9.2.108 Miller, M., Azrael, D., Hepburn, L., Hemenway, D., & Lippmann, S. J. (2006). The association between changes in household firearm ownership and rates of suicide in the United States, 1981-2002. Injury Prevention, 12(3), 178–182. https://doi.org/10.1136/ip.2005.010850 Miller, M., Barber, C., Azrael, D., Hemenway, D., & Molnar, B. E. (2009). Recent psychopathology, suicidal thoughts and suicide attempts in households with and without firearms: Findings from the national comorbidity study replication. Injury Prevention, 15(3), 183–187. https://doi.org/10.1136/ip.2008.021352 Miller, M., Barber, C., White, R. A., & Azrael, D. (2013). Firearms and suicide in the united states: Is risk independent of underlying suicidal behavior? American Journal of Epidemiology, 178(6), 946–955. https://doi.org/10.1093/aje/kwt197 Miller, M., Salhi, C., Barber, C., Azrael, D., Beatriz, E., Berrigan, J., Brandspigel, S., Betz, M. E., & Runyan, C. (2020). Changes in firearm and medication storage practices in homes of youths at risk for suicide: Results of the safety study, a clustered, emergency department–based, multisite, stepped-wedge trial. Annals of Emergency Medicine, 76(2), 194–205. https://doi.org/10.1016/j.annemergmed.2020.02.007 Miller, M., Zhang, Y., Prince, L., Swanson, S. A., Wintemute, G. J., Holsinger, E. E., & Studdert, D. M. (2022). Suicide deaths among women in california living with handgun owners vs those living with other adults in handgun-free homes, 2004-2016. JAMA Psychiatry, 79(6), 582–588. https://doi.org/10.1001/jamapsychiatry.2022.0793 76 National Domestic Violence Hotline. (n.d.). Safety Planning Around Guns and Firearms. Safety Planning Around Guns and Firearms. Retrieved May 26, 2023, from https://www.thehotline.org/resources/safety-planning-around-guns-and-firearms/ Neufeld, M. Y., Poulson, M., Sanchez, S. E., & Siegel, M. B. (2021). State firearm laws and nonfatal firearm injury-related inpatient hospitalizations: A nationwide panel study. Journal of Trauma and Acute Care Surgery, Publish Ahead of Print. https://doi.org/10.1097/TA.0000000000003445 Okoro, C. A., Nelson, D. E., Mercy, J. A., Balluz, L. S., Crosby, A. E., & Mokdad, A. H. (2005). Prevalence of household firearms and firearm-storage practices in the 50 states and the district of columbia: Findings from the behavioral risk factor surveillance system, 2002. Pediatrics, 116(3), e370–e376. https://doi.org/10.1542/peds.2005-0300 Owens, D., Horrocks, J., & House, A. (2002). Fatal and non-fatal repetition of self-harm: Systematic review. British Journal of Psychiatry, 181(3), 193–199. https://doi.org/10.1192/bjp.181.3.193 Pear, V. A., Schleimer, J. P., Tomsich, E., Pallin, R., Charbonneau, A., Wintemute, G. J., & Knoepke, C. E. (2021). Implementation and perceived effectiveness of gun violence restraining orders in California: A qualitative evaluation. PLOS ONE, 16(10), e0258547. https://doi.org/10.1371/journal.pone.0258547 Potter, H. (2008). Battle Cries: Black women and intimate partner abuse. New York University Press. Raissian, K. M. (2016). Hold your fire: Did the 1996 federal gun control act expansion reduce domestic homicides? Journal of Policy Analysis and Management, 35(1), 67–93. https://doi.org/10.1002/pam.21857 Rajula, H. S. R., Verlato, G., Manchia, M., Antonucci, N., & Fanos, V. (2020). Comparison of conventional statistical methods with machine learning in medicine: Diagnosis, drug development, and treatment. Medicina, 56(9), 455. https://doi.org/10.3390/medicina56090455 Reaves, B. (2017). Police response to domestic violence, 2006-2015. U.S. Department of Justice, Bureau of Justice Statistics. Rudd, M. D. (2008). Suicide warning signs in clinical practice. Current Psychiatry Reports, 10(1), 87–90. https://doi.org/10.1007/s11920-008-0015-4 Rudolph, K. E., Stuart, E. A., Vernick, J. S., & Webster, D. W. (2015). Association between connecticut’s permit-to-purchase handgun law and homicides. American Journal of Public Health, 105(8), e49–e54. https://doi.org/10.2105/AJPH.2015.302703 Runyan, C. W., Becker, A., Brandspigel, S., Barber, C., Trudeau, A., & Novins, D. (2016). Lethal means counseling for parents of youth seeking emergency care for suicidality. 77 Western Journal of Emergency Medicine, 17(1), 8–14. https://doi.org/10.5811/westjem.2015.11.28590 Runyan, C. W., Brooks-Russell, A., Brandspigel, S., Betz, M., Tung, G., Novins, D., & Agans, R. (2017). Law enforcement and gun retailers as partners for safely storing guns to prevent suicide: A study in 8 mountain west states. American Journal of Public Health, 107(11), 1789–1794. https://doi.org/10.2105/AJPH.2017.304013 Saltzman, L. E., Mercy, J. A., O’Carroll, P. W., Rosenberg, M. L., & Rhodes, P. H. (1992). Weapon involvement and injury outcomes in family and intimate assaults. JAMA, 267(22), 3043–3047. https://doi.org/10.1001/jama.1992.03480220061028 Siegel, M., Pahn, M., Xuan, Z., Fleegler, E., & Hemenway, D. (2019). The impact of state firearm laws on homicide and suicide deaths in the usa, 1991–2016: A panel study. Journal of General Internal Medicine, 34(10), 2021–2028. https://doi.org/10.1007/s11606-019-04922-x Silver, J. (2020). Space between concern and crime. Criminology & Public Policy, 19(1), 253– 270. https://doi.org/10.1111/1745-9133.12474 Sim, J., & Wright, C. C. (2005). The kappa statistic in reliability studies: Use, interpretation, and sample size requirements. Physical Therapy, 85(3), 257–268. https://doi.org/10.1093/ptj/85.3.257 Smith, S. G., Fowler, K. A., & Niolon, P. H. (2014). Intimate partner homicide and corollary victims in 16 states: National violent death reporting system, 2003–2009. American Journal of Public Health, 104(3), 461–466. https://doi.org/10.2105/AJPH.2013.301582 Smucker, S., Kerber, R. E., & Cook, P. J. (2018). Suicide and additional homicides associated with intimate partner homicide: North Carolina 2004-2013. Journal of Urban Health: Bulletin of the New York Academy of Medicine, 95(3), 337–343. https://doi.org/10.1007/s11524-018-0252-8 Sorenson, S. B. (2017). Guns in intimate partner violence: Comparing incidents by type of weapon. Journal of Womens Health, 26(3), 249–258. https://doi.org/10.1089/jwh.2016.5832 Sorenson, S. B., & Schut, R. A. (2018). Nonfatal Gun Use in Intimate Partner Violence: A Systematic Review of the Literature. Trauma, Violence & Abuse, 19(4), 431–442. https://doi.org/10.1177/1524838016668589 Sorenson, S. B., & Wiebe, D. J. (2004). Weapons in the lives of battered women. American Journal of Public Health, 94(8), 1412–1417. https://doi.org/10.2105/AJPH.94.8.1412 Stanley, B., & Brown, G. K. (2012). Safety planning intervention: A brief intervention to mitigate suicide risk. Cognitive and Behavioral Practice, 19(2), 256–264. https://doi.org/10.1016/j.cbpra.2011.01.001 78 Stanley, I. H., Hom, M. A., Rogers, M. L., Anestis, M. D., & Joiner, T. E. (2017). Discussing firearm ownership and access as part of suicide risk assessment and prevention: “Means safety” versus “means restriction.” Archives of Suicide Research, 21(2), 237–253. https://doi.org/10.1080/13811118.2016.1175395 Sunshine, J., & Tyler, T. R. (2003). The role of procedural justice and legitimacy in shaping public support for policing. Law & Society Review, 37(3), 513–548. https://doi.org/10.1111/1540-5893.3703002 Swanson, J. W., Easter, M. M., Alanis-Hirsch, K., Belden, C. M., Norko, M. A., Robertson, A. G., Frisman, L. K., Lin, H.-J., Swartz, M. S., & Parker, G. F. (2019). Criminal justice and suicide outcomes with Indiana’s risk-cased gun seizure law. Journal of the American Academy of Psychiatry and the Law Online, 47(2), 188–197. https://doi.org/10.29158/JAAPL.003835-19 Trappenberg, T. P. (2020). Fundamentals of Machine Learning. Oxford University Press. Tyler, T. R. (2004). Enhancing police legitimacy. The ANNALS of the American Academy of Political and Social Science, 593(1), 84–99. https://doi.org/10.1177/0002716203262627 Tyler, T. R., & Huo, Y. J. (2002). Trust in the Law: Encouraging Public Cooperation with the Police and Courts. Russell Sage Foundation. Vittes, K. A., Webster, D. W., Frattaroli, S., Claire, B. E., & Wintemute, G. J. (2013). Removing guns from batterers: Findings from a pilot survey of domestic violence restraining order recipients in California. Violence Against Women, 19(5), 602–616. https://doi.org/10.1177/1077801213490561 Wallace, M. E., Vilda, D., Theall, K. P., & Stoecker, C. (2021). Firearm relinquishment laws associated with substantial reduction in homicide of pregnant and postpartum women: Study examines the association of firearm relinquishment laws with homicide of pregnant and postpartum women. Health Affairs, 40(10), 1654–1662. https://doi.org/10.1377/hlthaff.2021.01129 Walters, H., Kulkarni, M., Forman, J., Roeder, K., Travis, J., & Valenstein, M. (2012). Feasibility and acceptability of interventions to delay gun access in va mental health settings. General Hospital Psychiatry, 34(6), 692–698. https://doi.org/10.1016/j.genhosppsych.2012.07.012 Wang, N., Zeng, N. N., & Zhu, W. (2010). Sensitivity, Specificity, Accuracy, Associated Confidence Interval And ROC Analysis With Practical SAS Implementations. NESUG Proceedings: Health Care and Life Sciences, Baltimore, Maryland. http://www.nesug.org/Proceedings/nesug10/hl/hl07.pdf 79 Webster, D., Crifasi, C. K., & Vernick, J. S. (2014). Effects of the repeal of Missouri’s handgun purchaser licensing law on homicides. Journal of Urban Health, 91(2), 293–302. https://doi.org/10.1007/s11524-014-9865-8 Webster, D. W., Frattaroli, S., Vernick, J. S., O’Sullivan, C., Roehl, J., & Campbell, J. C. (2010). Women with protective orders report failure to remove firearms from their abusive partners: Results from an exploratory study. Journal of Women’s Health, 19(1), 93–98. https://doi.org/10.1089/jwh.2007.0530 Webster, D. W., McCourt, A. D., Crifasi, C. K., Booty, M. D., & Stuart, E. A. (2020). Evidence concerning the regulation of firearms design, sale, and carrying on fatal mass shootings in the United States. Criminology & Public Policy, 19(1), 171–212. https://doi.org/10.1111/1745-9133.12487 Weinberger, S. E., Hoyt, D. B., Lawrence, H. C., Levin, S., Henley, D. E., Alden, E. R., Wilkerson, D., Benjamin, G. C., & Hubbard, W. C. (2015). Firearm-related injury and death in the united states: A call to action from 8 health professional organizations and the american bar association. Annals of Internal Medicine, 162(7), 513–516. https://doi.org/10.7326/M15-0337 Wiebe, D. J. (2003). Homicide and suicide risks associated with firearms in the home: A national case-control study. Annals of Emergency Medicine, 41(6), 771–782. https://doi.org/10.1067/mem.2003.187 Wintemute, G. J., Betz, M. E., & Ranney, M. L. (2016). Yes, you can: Physicians, patients, and firearms. Annals of Internal Medicine, 165(3), 205–213. https://doi.org/10.7326/M15- 2905 Yousuf, S., McLone, S., Mason, M., Snow, L., Gall, C., & Sheehan, K. (2017). Factors associated with intimate partner homicide in Illinois, 2005–2010: Findings from the Illinois Violent Death Reporting System. Journal of Trauma and Acute Care Surgery, 83(5S), S217. https://doi.org/10.1097/TA.0000000000001578 Zalsman, G., Hawton, K., Wasserman, D., van Heeringen, K., Arensman, E., Sarchiapone, M., Carli, V., Höschl, C., Barzilay, R., Balazs, J., Purebl, G., Kahn, J. P., Sáiz, P. A., Lipsicas, C. B., Bobes, J., Cozman, D., Hegerl, U., & Zohar, J. (2016). Suicide prevention strategies revisited: 10-year systematic review. The Lancet Psychiatry, 3(7), 646–659. https://doi.org/10.1016/S2215-0366(16)30030-X Zeoli, A. M., Malinski, R., & Turchan, B. (2016). Risks and targeted interventions: Firearms in intimate partner violence. Epidemiologic Reviews, 38(1), 125–139. https://doi.org/10.1093/epirev/mxv007 Zeoli, A. M., McCourt, A., Buggs, S., Frattaroli, S., Lilley, D., & Webster, D. W. (2018). Analysis of the strength of legal firearms restrictions for perpetrators of domestic violence and their associations with intimate partner homicide. American Journal of Epidemiology, 187(11), 2365–2371. https://doi.org/10.1093/aje/kwy174 80 Zeoli, A. M., Paruk, J., Branas, C. C., Carter, P. M., Cunningham, R., Heinze, J., & Webster, D. W. (2021). Use of extreme risk protection orders to reduce gun violence in Oregon. Criminology & Public Policy, 20(2), 243–261. https://doi.org/10.1111/1745-9133.12544 81 APPENDIX Table 1A Summarized Examples of Precipitating Events (PE) PE Characteristic Suicide risk Use and/or threat of violence against family Examples The respondent threatened to kill himself and told his ex-wife to call his mother and tell her that he loves her. When officers arrived, the respondent put the gun to his head. The respondent poured rubbing alcohol under his mother’s door and threatened to light it on fire. When his mother exited the bedroom, the respondent had a rifle in his hand, cornered her in the hallway, knocked the phone out of her hand, and racked the firearm. Use and/or threat of While on the phone with his university about an outstanding violence against others Suicide and use/threat of violence Mass shooting/multiple- victim firearm threat Irrational or erratic behaviors financial balance, the respondent threatened to blow up the university building. The respondent then threatened to slit an officer’s throat as he was being taken for an involuntary examination. The respondent showed up at her ex-partner’s home, got into a verbal argument with him, and then shot six rounds into the air. She also texted her mother that she wanted to kill everyone, her four-year-old son, and herself. The respondent put a gun to his wife’s head and told her that if she left him, he would kill her and her three children. When the respondent went into the bathroom, the respondent’s wife left the residence with her children. The respondent’s wife reported that the respondent had an episode of hallucinations and he stabbed himself several times with a butcher knife. When being treated by the fire department, the respondent yelled for the demons to go away. 82 Table 2A Differences Between Predictor Variables from Multi-Site ERPO Study that Did and Did Not Meet the Study Inclusion Criteria p-value Predictor variable All Multi-Site ERPO cases (N=2,406) Cases that did not meet study inclusion criteria (n=1,933) Cases that met study inclusion criteria (n=473) Test statistic 12.55% (302) 39.76 (17.18) 74.76% (1,786) 17.83% (426) 7.41% (177) 33.63% (809) 8.35% (201) Respondent demographics Female Average age (SD) Race White Black Other Mental health issue Armed services Suicidality in PE Ideation, threats, or plans Aborted attempt Attempt Firearm Interpersonal violence in PE 377% (15.67%) Use of violence against family 5.40% (130) Use of firearm violence against family 12.18% (293) Use of violence against others 6.48% (156) Use of firearm violence against others Threat of violence against family 22.78% (548) Threat of firearm violence against family 16.67% (491) Threat of violence against others Threat of firearm violence against others Multiple victim/mass shooting threat Alcohol and substance use Alcohol use Substance use 44.51% (1,071) 9.35% (225) 5.24% (126) 33.58% (808) 43.93% (1,057) 35.62% (857) 12.64% (304) 18.41% (443) 6.23% (150) 12.62% (244) 39.40 (17.56) 12.26% (58) 41.20 (15.46) 0.05 -2.71 0.832 0.007 73.41% (1,419) 18.52% (358) 8.07% (156) 32.59% (630) 7.92% (153) 41.28% (798) 7.45% (144) 4.81% (93) 30.21% (584) 11.95% (231) 3.83% (74) 12.36% (239) 6.62% (128) 19.81% (383) 13.86% (268) 47.70% (922) 38.64% (747) 13.61% (263) 77.59% (367) 14.38% (68) 8.03% (38) 37.84% (179) 10.15% (48) 57.72% (273) 17.12% (81) 6.98% (33) 47.36% (224) 30.87% (146) 11.84% (56) 11.42% (54) 5.92% (28) 34.88% (165) 28.12% (133) 28.54% (135) 23.26% (110) 8.67% (41) 4.59 0.101 4.70 2.47 41.55 41.96 3.59 50.08 102.91 47.72 0.32 0.31 49.10 55.59 56.62 39.24 8.39 0.030 0.116 0.000 0.000 0.058 0.000 0.000 0.000 0.572 0.578 0.000 0.000 0.000 0.000 0.004 16.40% (317) 6.00% (116) 26.64% (126) 7.19% (34) 26.52 0.92 0.000 0.338 83 Table 2A (cont’d) Additional PE characteristics Arrest Average month (SD) Irrational or erratic behaviors Firearm information Handgun Long gun More than one firearm 31.09% (748) 16.18 (7.53) 15.42% (371) 29.44% (569) 15.95 (7.50) 15.88% (307) 37.84% (179) 17.18 (7.57) 13.53% (64) 12.54 -3.27 1.61 0.000 0.001 0.204 61.97% (915) 35.29% (849) 65.21% (1,569) 56.49% (1,092) 32.13% (621) 68.29% (1,320) 84.36% (399) 48.20% (228) 52.64% (249) 125.18 43.01 41.00 0.000 0.000 0.000 84 95% Confidence Interval (0.53, 0.74) Sensitivity Specificity 0.12 0.20 0.93 0.88 Table 3A Summarized Examples of Precipitating Events (PE) Outcome Variable Accuracy Any effort to reduce firearm 0.64 access Effort to reduce firearm access 0.70 (0.60, 0.79) after police arrived 85 Table 4A Descriptions of Predictor Variables Predictor variable Respondent demographics Age Gender Race/ethnicity Member of armed services Mental health issue Suicidality in PE Ideation, threats, or plans Aborted attempt Attempt Firearm Interpersonal violence in PE Use of violence Description Respondent’s age (in years). Age was missing for one observation, and as age was not normally distributed, I used the median age (38) to replace that single missing observation. Respondent’s gender. (female=1, male=0) Respondent’s race and/or ethnicity. Race/ethnicity was collapsed into three categories: white, black, and other race/ethnicity. White was used as the reference category in the model. Respondent was a current or former member of the armed services. (1=yes, 0=no evidence) Respondent had a mental health diagnosis; was described as having psychiatric medication; had a psychiatric hospitalization; was described as autistic; or having attention deficit disorder, a developmental delay, cognitive decline related to age (e.g., Alzheimer’s), or cognitive impairment from medical illness or unstated causes. (1=yes, 0=no evidence) Respondent indicated that they were having thoughts about killing themselves, explicitly stated an intention to kill themselves, or planned for a suicide attempt (e.g., texted goodbye to a loved one). (1=yes, 0=no evidence) Respondent took steps to engage in suicide but did not go through with it (e.g., put gun to head but then put the gun down). (1=yes, 0=no evidence) Respondent took potentially lethal action (e.g., shot themselves with the intent to die), but did not die. (1=yes, 0=no evidence) Respondent used or referenced a gun in suicide ideation, threats, plan, aborted attempt, or attempt. (1=yes, 0=no evidence) Respondent behaved in a way that could have inflicted harm, even if the harm did not occur (e.g., punched someone or punched a wall near someone). Coded if the respondent used violence against a family member/intimate partner or anyone else. (1=yes, 0=no evidence). (1=yes, 0=no evidence) 86 Table 4A (cont’d) Use of firearm violence Threat of violence Threat of firearm violence Multiple victim or mass shooting firearm use/threat Substance use in PE Alcohol use Substance use Police could remove firearms Baker Act Felony with a firearm Firearm information Handgun Long gun Respondent used a firearm in a way that could have inflicted harm, even if the harm did not occur (e.g., pistol whipped someone or shot at someone and missed). Coded if the respondent used firearm violence against a family member/intimate partner or anyone else. (1=yes, 0=no evidence). Respondent communicated a desire or intent to harm someone else (e.g., told their partner they would kill them). Coded if the respondent threatened violence against a family member/intimate partner or anyone else. (1=yes, 0=no evidence) Respondent used or referenced a firearm in their communication of a desire or intent to harm someone else (e.g., pointed a gun a someone). Coded if the respondent threatened firearm violence against a family member/intimate partner or anyone else. (1=yes, 0=no evidence) Respondent used or threatened firearm violence against a group of three or more people or a location that would conceivably have three or more people (e.g., a school). (1=yes, 0=no evidence) Respondent used alcohol. (1=yes, 0=no evidence) Respondent used illicit drugs, opioids (including prescription opioids), marijuana, other substances, or was vaguely described as using drugs (“took pills”). (1=yes, 0=no evidence) Respondent was taken into protective custody under the authority of the Baker Act for an emergency involuntary mental health examination. (1=yes, 0=no evidence) Respondent was arrested or police report submitted with requested charges for committing a felony with a firearm. (1=yes, 0=no evidence) The respondent had access to or possession of a handgun (pistol or revolver, gun with a short barrel that can be held with one hand). (1=yes, 0=no evidence) The respondent had access to or possession of a long gun (shot gun or rifle, a gun with a long barrel that is generally held with two hands and braced against the shoulder) 87 Table 4A (cont’d) More than one firearm Officers mentioned ERPO in report Officers told someone about ERPOs on scene Officers did not tell someone on scene about ERPOs Other PE characteristics Arrest Month Irrational or erratic behaviors Family lived with respondent The number of firearms was missing from 5% of observations. As this variable was not normally distributed, I used the median number of firearms (2) to replace the missing values. Next, as count variables with extreme values can create problems for machine learning models (Zheng & Casari, 2018) and the number of firearms ranged from 1-131, I created a binary variable (1=access to or possession of more than 1 firearm, 0=access or possession to one firearm) Officer told someone on scene about the ERPO process or that they were going to submit an ERPO petition. (1=yes, 0=no evidence) Officers mentioned that they were going to pursue an ERPO, but did not include in their report that they told someone on scene. (1=yes, 0=no evidence) Respondent was arrested. (1=yes, 0=no evidence) Number of months since the Parkland Marjory Stoneman Douglas High School Public Safety Act (MSDHSPSA) was enacted. Reports from petitions from March 2018 were coded as 1 and petitions from June 2020 were coded as 28. Respondent engaged in behaviors that did not seem to make sense or were out of character (e.g., talking to a hallucination), or were described as irrational or erratic. (1=yes, 0=no evidence) Family members or an intimate partner lived with the respondent or owned where the respondent lived. (1=yes, 0=no evidence) 88 Figure 1A Correlation Plot of Predictors 89