HARNESSING THE COLLECTIVE INTELLIGENCE OF STAKEHOLDERS TO UNDER STAND SOCIAL - ECOLOGICAL SYSTEMS By Payam Aminpour Mohammadabadi A DISSERTATION Submitted to Michigan State University in partial fulfillment of the requirements for the degree of Community Sustainability Doctor of Philosophy 2020 ABSTRACT HARNESSING THE COLLECTIVE INTELLIGENCE OF STAKEHOLDERS TO UNDERSTAND SOCIAL - ECOLOGICAL SYSTEMS By Payam Aminpour Mohammadabadi Collective Intelligence (CI) is an amplified, meta - intelligence that emerges when a distributed collective of individuals aggregate their inputs in order to solve a problem, often with the help of communication or knowledge pooling . Important ly, CI outcomes (e.g., solutions, decisions, judgments, wisdom and knowledge) are generally more problem - adequate and societies can therefore solve key and pressing problems that no individual can resolve alone. Importantly, with recent advances in digital technologies, we now have more potential to harness the full power of human collectives to better address fast - evolving, complex problems facing human societies, m any of which are complex issues that are resulted from the interactions between humans and natural ecosystems. Problems like anthropogenic environmental changes, biodiversity loss, and over - consumption of natural resources, which often take place in so c alled social - ecological systems (SES s), require adequate knowledge and complete understandings about complex relationships between intertwined social and environmental dimensions. Such understanding s are difficult to achieve in many contexts due to data sc arcity and scientific knowledge limitations. This dissertation explores the potentials of using CI approaches to leverage the local knowledge of environmental and natural resources stakeholders to better understand SESs, develop adequate knowledge of compl ex human - environment interactions, and inform sustainability decisions. First, this dissertation synthetizes key insights from biological, cognitive, behavioral, and management sciences literature to develop a framework that guides the design and generatio n of CI in human groups. This framework organizes fundamental design elements of CI and thus can help researchers, communities, and policymakers, especially in data - poor situations, design crowd - based approaches to aggregating knowledge of local people and stakeholders in order to achieve accurate and reliable understandings of complex human - environment interactions. Additionally, this dissertation empirically test s CI approaches using three real - world fisheries case studies . The first empirical study uses an example of inland freshwater pike fisheries to explore how CI of local stakeholders can be harnessed through aggregation of their mental models about human - environment interactions. This study shows that the aggregated model can provide scientifically sound insights about how the ecosystem and humans are coupled, and how their interactions are influenced by various management strategies. The second empirical study uses an example of striped bass fisheries in Massachusetts , to explore the impact of knowl edge diversity on the CI of local stakeholders while pooling their local knowledge about the complex human - environment interactions . The final study uses an example of U.S. Atlantic coasts to scale up the se CI approaches by crowdsourcing inputs from a very large population of change impacts on ocean fisheries across a large social and ecological gradient. This study demonstrates perfect match among stakeholder - d predictions of behavioral changes, and empirical patterns of climate change disturbances. knowledge for understanding the complexity of SESs have considerable implications for dealing with scientific and management uncertainties, while many untapped potentials still remain. iv To my beautiful wife. F or her love and support, for all the late nights and early mornings . I love you. v A CKNOWLEDGMENTS I would like to express my sincere gratitude to my advisor and great friend Dr. Steven Gray for his guidance, encouragement and motivation, patience, and immense knowledge throughout my Ph.D. research. I have learned a lot from him about how to be a scholar. I could not have imagined having a better advisor for my Ph.D. study and a mentor for my academic and personal life. Thanks for showing me a fantastic example of a scientist that I ever want to be. I would like to extend my thank s to all my research committee members, Dr. Rebecca Jordan, Dr. Maria Claudia Lopez, and Dr. Joshua Introne for their insightful comments and guidance. I would also like to sincerely thank my external mentors, collaborators, co - authors and research adviser s for their support and friendship over the past four years: Dr. Antonie Jetter from Portland State University, Dr. Philippe Giabbanelli from Miami University, Dr. Steven Scyphers and Dr. Jonathan Grabowski from Northeastern University, Dr. Robert Arlingha us from Leibniz - Institute of Freshwater Ecology and Inland fisheries, Dr. Michael O'Rourke from MSU Center for Transdisciplinarity, Dr. Lex Paulson from UM6P School of Collective Intelligence, and Dr. Carina Antonia Hallin from Copenhagen Business School - the Collective Intelligence Unit. Without their support, advice, and mentorship this research would not be finished. I would also like to thank my many friends in the Department of Community Sustainability for being my second family at MSU. Last but not l east , I owe my deepest gratitude to my wife Niyaz, my parents Foruzan and for their unconditional love, patience, encouragement and support. vi TABLE OF CONTENTS L IST OF TABLES ................................ ................................ ................................ ..................... viii LIST OF FIGURES ................................ ................................ ................................ ..................... ix INTRODUCTION ................................ ................................ ................................ ......................... 1 OVERVIEW OF THE PROBLEM ................................ ................................ ............................ 1 OVERVIEW O F THE METHODOLOGY ................................ ................................ ................ 3 DISSERTATION OUTLINE ................................ ................................ ................................ ..... 6 REFERENCES ................................ ................................ ................................ ........................... 8 CHAPTER 1 ................................ ................................ ................................ ................................ 12 1 BECOMING INTELLIGENT ABOUT COLLECTIVE INT ELLIGENCE AND PUBLIC POLICY ................................ ................................ ................................ ....................... 12 ABSTRACT ................................ ................................ ................................ .............................. 12 1.1 THE POWER OF THE COLLECTIVE ................................ ................................ ............ 13 1. 2 COLLECTIVE INTELLIGENCE FRAMEWORK ................................ .......................... 14 1.2.1 The problem ................................ ................................ ................................ ............... 14 1.2.2 The collective ................................ ................................ ................................ ............. 15 1.2.3 The aggregation mechanism ................................ ................................ ...................... 16 1.3 POLICY IMPLICATIONS ................................ ................................ ............................... 19 APPENDIX ................................ ................................ ................................ ............................... 22 REFERENCES ................................ ................................ ................................ ......................... 26 CHAPTER 2 ................................ ................................ ................................ ................................ 29 2 WISDOM OF STAKEHOLDE R CROWDS IN COMPLEX SOCIAL - ECOLOGICAL SYSTEMS ................................ ................................ ................................ ....... 29 ABSTRACT ................................ ................................ ................................ .............................. 29 2.1 INTRODUCTION ................................ ................................ ................................ ............ 30 2.2 EXPERIMENTAL DESIGN ................................ ................................ ............................ 34 2.3 RESULTS ................................ ................................ ................................ ......................... 36 2.4 DISCUSSION ................................ ................................ ................................ ................... 42 2.5 METHODS ................................ ................................ ................................ ....................... 46 2.5.1 Description of study system and context ................................ ................................ ... 46 2.5.2 Mental models ................................ ................................ ................................ ............ 48 2.5.3 Mental model aggregation ................................ ................................ ......................... 50 2. 5.4 FCM analyses ................................ ................................ ................................ ............. 51 APPENDIX ................................ ................................ ................................ ............................... 58 REFERENCES ................................ ................................ ................................ ......................... 73 CHAPTER 3 ................................ ................................ ................................ ................................ 78 vi i 3 THE DIVERSITY BONUS IN POOLING LOCAL KNO WLEDGE ABOUT COMPLEX PROBLEMS ................................ ................................ ................................ ........... 78 ABSTRACT ................................ ................................ ................................ .............................. 78 3.1 INTRODUCTION ................................ ................................ ................................ ............ 79 3.2 RESULTS ................................ ................................ ................................ ......................... 83 3.3 DISCUSSION ................................ ................................ ................................ ................... 90 3.4 METHODS ................................ ................................ ................................ ....................... 93 3.4.1 Mental models and fuzzy cognitive maps ................................ ................................ .. 93 3.4.2 Online crowdsourcing implementation ................................ ................................ ...... 94 3.4.3 Collective intelligence and knowledge pooling ................................ ......................... 95 3.4.4 ................................ ................................ 96 3.4.5 Network analysis of stakeholder - driven models ................................ ........................ 97 APPENDIX ................................ ................................ ................................ ............................... 99 REFERENCES ................................ ................................ ................................ ....................... 116 CHAPTER 4 ................................ ................................ ................................ .............................. 122 4 CROWDSOURCING MENTAL MODELS FOR PREDICTIN G BEHAVIORAL RESPONSES TO CLIMATE CHANGE ................................ ................... 122 ABSTRACT ................................ ................................ ................................ ............................ 122 4.1 INTRODUCTION ................................ ................................ ................................ .......... 123 4.2 RESULTS ................................ ................................ ................................ ....................... 125 4.2.1 Overall Climate Concern ................................ ................................ ......................... 125 4.2.2 Fishing Characteristics ................................ ................................ ............................. 128 4.2.3 Simulating Climate Change in Mental Models ................................ ........................ 128 4.3 DISCUSSION ................................ ................................ ................................ ................. 132 4.4 METHODS ................................ ................................ ................................ ..................... 135 4.4.1 Ethics Statement ................................ ................................ ................................ ....... 135 4.4.2 Survey Instrument ................................ ................................ ................................ .... 135 4.4.3 Survey Data Collection ................................ ................................ ............................ 136 4.4.4 Empirical Data ................................ ................................ ................................ ......... 137 4.4.5 Analyses ................................ ................................ ................................ ................... 138 APPENDIX ................................ ................................ ................................ ............................. 142 REFERENCES ................................ ................................ ................................ ....................... 151 viii LIST OF TABLES Table 2.S1. The list of all concepts used to build fuzzy cognitive maps. The list of factors were derived from independent focus groups with anglers and mental model pre - tests with both anglers and experts, to identify key con cepts relevant to the pike fishery. Participants were given the freedom to add additional concepts and the final list of all identified concepts was 19 concepts coded from node 0 to node 18 in all fuzzy cognitive maps. ................................ .......... 70 Table 2.S2. The performance of the crowd model generated by different aggregation methods. The last column shows the overall performance of the crowd models generated by different aggregation methods. The overall performance is calculated by subtracting the normalize d total error from one. The normalized total error itself is the mean of normalized dynamic and structure errors. The normalized dynamic and structure errors are respectively the standardized euclidian dynamic and structure distances between the crowd and expert models as described in methods section. Therefore, the normalized performance serves as an interpretive criterion to rank the accuracy of aggregated models in approximating the structure and dynamic behavior of the scientific expert model. ................................ ................................ ................................ ................. 71 Table 2.S3. Fisheries knowledge and education metrics assessed using questionnaires after the 0.84. Statistical test on mean differences is based on ANOVA , with post - hoc t est T ukey b for homogenous variances, and D unnett - T - 3 for heterogeneous variances. ................................ ...... 72 Table 3.S1. The number of participants from each stakeholder type and the number of nodes and connections used in their mental models. The mean and standard deviation of number of concepts (i.e., nodes) and connections (i.e., edges) are shown by stakeholder types. ................ 115 Table 4.S 1. Survey sample demographics and fishing characteristics of respondents within each state. ................................ ................................ ................................ ................................ ............ 149 ix LIST OF FIGURES Figure 1.1. Collective intelligence framework for policy - making. This framework provides insights into new hypothetical pathways to aggregate information (i.e., units of knowledge) from a group of individuals and thereby offer solutions that are more optimal than how any single individual could have addressed a particular problem (examples are provided in the appendix, F igure 1. S 1). ................................ ................................ ................................ ................................ .. 18 Figure 1.S1. Examples of collective intelligence ( CI ). Each example demonstrates a unique approach to harness the ci of a collective by aggregating individual inputs to solve a particular problem. See F igure 1.1 for more information about sub - categories (i.e., PU , CX, DV, EX, GS, EN, TM, GF, SI , and AG ). ................................ ................................ ................................ ............. 24 Figure 1.S2. The sankey diagram showing the flow between sub - components of ci framework. Considering examples from F igure 1. S 1, the diagram shows where a ci ca n come from and where it can end up, with possible intermediate steps, where the width of the connections between two nodes visualizes the quantity of examples used these pairs of nodes (i.e., sub - components). ................................ ................................ ................................ ................................ . 25 Figure 2.1. Centrality profiles of different groups (in color) and the expert reference model (in black/grey). Axes in the radar charts show the centrality of s ystem elements that are important for fishery management decisions. Katz index is used to measure the centrality (see M ethods). 37 Figure 2.2. Agreement on strong causal patterns in the fcm of stakeholder - specific groups, the crowd and the experts. The crowd map has the highest degree of matched patterns (~70% matched) with experts; the stakeholder - specific groups perform substantially better (a mong 53% to 63% of correct matches) than the null - unwise model (only ~30% correct matches). Weak relationships with an edge weight less than 0.33 (the first tertile in zero to one continuum, corresponding to the weak interval) were removed from the maps to get the strong causal patterns (see A ppendix , F igure 2. S 2). Error bars display standard errors. ................................ .... 38 Figure 2.3. Eigenvalue similarity index. Within each group (x axis), each point represents one individual and is placed according to the eigenvalue similarity index (y axis). The similarity index represents the structural mismatch with the experts mental model. The s warm plots reflect the density of points around any distance value, while black squares represent the aggregated mental models for each group. The crowd model has the smallest distance from (highest similarity with) the experts model. Interestingly, for al l stakeholder groups, aggregated model is located below the densest area of the plot, illustrating the woc effect (the average model outperforms most individuals). Yet, this effect is notably higher in the crowd. ........................... 39 Figure 2.4. The dynamic distance between the experts model and the stakeholder - derived models based on 10,000 randomly generated scenarios (experiments). Each experiment randomly selects a set of concepts (nodes in the fcms) and changes their values to produce outputs (see methods). (a) each cell in the color - bar graph represents a random scenario with colors denoting the dynamic distance. (b) boxplots illu strate the distribution of these dynamic distances x for each group in 10,000 experiments. The mean of dynamic distances from the reference model is the smallest in the crowd. ................................ ................................ ................................ .......... 40 Figure 2.5. The sampling and averaging effect on performance error in crowds built by drawing and aggregating mental models using two aggregation methods. (a) single level aggregation. (b) multi - level aggregation. Samples were formed by randomly drawing individuals f rom all 218 participants. Data are shown for 100 repeats per sample size. (test of > 100 random crowd assignment show no significant difference). ................................ ................................ ................. 41 Figure 2.S1. An example of pike ecology and management fuzzy cognitive map generated by one participant in the workshops: the individually collected mental models graphically display the perceived cause - and - effect relationship s of ecological and social concepts affecting each other (e.g., how habitat quality affects juvenile pike that later grow into harvestable size, or how fish - eating birds, stocking, or angling pressure affect the pike population). Note that the mes are in german. ................................ ................................ ................................ .... 66 Figure 2.S2. The aggregated fuzzy cognitive maps of pike ecology and management in different models, (d) map generated by aggregating all 218 individual models using multi - level reference model. Red arrows represent negative relationships, and blue a rrows represent positive relationships between concepts. Weak relationships with a weight less than 0.33 were removed from the maps for a more clear illustration. ................................ ................................ .................. 67 Figure 2.S3. Distribution of group emphasis on different concepts and its variation with group model. Given each specific group size , we randomly sampled 100 groups by drawi ng individual mental models. Each chart has 19 box plots (one for each concept in the model), each shows the distribution of degree centrality of a concept in 100 random samples. Degree centrality represents the perceived importance of the concept, base d on its connections to other concepts (i.e., the number of inward and outward facing arrows). The x - axis shows the 19 concepts coded from 0 to 18 (see A ppendix , T able 2. S 1 for the name of the concepts). The y - axis indicates the degree centrality of each concept. ................................ ................................ ................................ . 68 Figure 3.1. The representation of cause - and - effect relationships between 15 overlapping concepts shared by all stakehold er groups and the diverse crowd. Ecological components are green and social components are purple. The aggregated graphs of (a) commercial fishers, (b) recreational fishers, (c) fisheries managers, and (d) the diverse crowd were evaluated by scientific exp erts to assess their accuracy in terms of causal relationships and feedback loops. Evaluations were conducted in 5 steps as shown in (d). ................................ ............................... 85 Figure 3.2. Fishery response (i.e., relative normalized changes in value of concepts) to six different scenarios simulating (a) increased inclement weather for fishing, (b) increased water temperature, (c) decreased water quality, (d) increased price of fish, (e) increased demand, and (f) increased poaching and illegal activities. ................................ ................................ ................. 86 xi Figure 3.3. Expert evaluation of aggregated models. The box plots represent the distribution of performance was measured using a 7 point likert scale for each item of interview sheets (see A ppendix ). The assigned accuracies of structural items (i.e., five sub - structures illustrated in F igure 3. 1) were averaged and normalized to a scale between 0 and 1. Similarly, the assigned accuracies of dynamic items (six scenarios illustrated in fig ure 2) were averaged and normalized to a scale between 0 and 1. The 2d scatter plot in (c) shows the overall score given to four models by each expert, where x - - axis is ucture. ................................ ................................ .................... 88 Figure 3.4. Deviation of the prevalence of complex causal motifs in aggregated models relative to uniform random graphs for (a) bi - directionality, (b) indirect effect, (c) multiple effects, and (d) feedback loops. Black dots represent 10,000 random graphs and the blue line shows the expected value of motif counts. Red dashes represent the deviation of each model from the expected value. ................................ ................................ ................................ ................................ ....................... 90 Figure 3.S1. An example of a fuzzy cognitive map ( FCM ) representing a mental model about striped bass fishery. The fcm was created using mental modeler online platform at www.mentalmodeler.org. Boxes demonstrate system concepts defined by the individual modeler and arrows indicate causal relationships between concepts. ................................ ...................... 106 Figure 3.S2. All in dividual fuzzy cognitive maps (FCM ) representing the mental models of 32 participants about striped bass fishery in massachusetts. ................................ ........................... 107 Figure 3.S3. Multi - level aggregation method. At the first level, individual maps are aggregated the second level, the resulting group means are aggregated using the median of their edge weights to produce the crowd model. ................................ ................................ ......................... 108 Figure 3.S4. Aggregated mental model of recreational fishers. Circles demonstrate unique system concepts mentioned by the individuals of type recreational fisher. Ecological - dimension concepts are green and human - dimension components are purple. Weighted blue/red arro ws indicate positive/negative causal relationships between concepts. The arrows thickness represents the strength of the causal relationships ranged from - 1 to +1. The weight of the arrows are computed using equation 3. S 1. ................................ ................................ ................................ ... 109 Figure 3.S5. Aggregated mental model of commercial fishers. Circles demonstrate unique system concepts mentioned by the individuals of type commercial fisher. Ecological - dimension concepts are green and human - dimension components are purple. Weighted blue/red arrows i ndicate positive/negative causal relationships between concepts. The arrows thickness represents the strength of the causal relationships ranged from - 1 to +1. The weight of the arrows are computed using equation 3. S 1. ................................ ................................ ................................ ... 110 Figure 3.S6. Aggregated mental model of fisheries managers. Circles demonstrate unique system concepts mentioned by the individuals of type manager. Ecological - dimension concepts are green and human - dimension components are purple. Weighted blue/red arrows indicate po sitive/negative causal relationships between concepts. The arrows thickness represents the xii strength of the causal relationships ranged from - 1 to +1. The weight of the arrows are computed using equation 3. S 1. ................................ ................................ ................................ .................... 111 Figure 3.S7. Aggregated mental model of the diverse crowd. Circles demonstrate a parsimonious list of system concepts mentioned by all individuals of all stakeholder types. This parsimonious list of system concepts is obtained by a multi - level aggregation method. Ecol ogical - dimension concepts are green and human - dimension components are purple. Weighted blue/red arrows indicate positive/negative causal relationships between concepts. The arrows thickness represents the strength of the causal relationships ranged from - 1 to +1. The weight of the arrows are computed using equation 3. S 1. ................................ ........................... 112 Figure 3.S8. The frequency and the relative percentage of each category of system concepts across three stakeholder groups. The numbers on bar - graphs indicate the frequency of concepts under each specific category. X - axis shows the relative percentage. ................................ ......... 113 Figure 3.S9. those appeared in more than one model, but not all models. The opinion table in (a) shows (white) or there is no consensus among experts (half - black, half - white). The percent of false errors ................................ ....................... 113 Figure 3.S10. Step - by - step written instructions for participants to direct them how to use online mental modeling tool and create fuzzy cognitive maps repressing their perception of striped bass fisheries in ma and social - ec ological relationships driving this system. ................................ .... 114 Figure 4.1. Community maps of three study regions representing F lorida ( FL ), N orth C arolina ( NC ), and M assachusetts ( MA ) built by aggregating individual fcms from each region. The inset shows the nc community model with details (see A ppendix, F igure 4. S 1 for details about other states). Blue/red arrows indicate positive/negative causal relationships between concepts. Edge weights represent perceived strength of the causal links. ................................ ........................... 126 Figure 4.2. arming oceans (a), storminess (b), and fish decline (c) alongside patterns of empirical data on water temperature (d), storminess (e), and fisheries stock status (f) trends. Levels of significance are illustrated in (a - c) by asterisks ( p - value< 0.05, p - value< 0.01, and p - value< 0.001 are shown by one, two, and three asterisks respectively). (note: stock status trends are missing 1997 - 98 data points due to the unavailability of information for the number of overfishing stocks. In addition, stock status d ata are classified based on NOAA fisheries regions: MA is included in N ew E ngland; NC is partially included in the M id and S outh A tlantic; and FL is partially included in the S outh A tlantic and G ulf). ................................ ...... 127 Figure 4.3. Diversity of target species across regions. (a) species accumulation curves shows the cumulative percentage of total primary target species reached by a given number of unique species. (b) circular chart for each region shows the target species and their percentage. Only target species with more than 10% are labeled for each region: S triped bass ( SB ), red drum ( RD ), summer flounder ( SF ), spotted seatrout ( SS ), snook ( SK ), and red snapper ( RS ). The horizontal bar charts show the calculated S hannon diversity index ( H ). ................................ ..................... 129 xiii Figure 4.4. target species abundance. For each state results are shown for various combinations of water temperature and storminess jointly (a - c); water temperature individually (d - f); and water storminess individually (g - from high decrease ( - 1) to high increase (+1). H eat map shows the perceived changes in target species abundance from high decrease (dark red) to high increase (dark blue). ........................ 130 Figure 4.5. regarding the intended fishing days. For each state results are shown for various combinations of water temperature and storminess jointly (a - c); water temperature in dividually (d - f); and water storminess individually (g - from high decrease ( - 1) to high increase (+1). Heat map shows the intended number of days fished from high decrease (dark red) to hig h increase (dark blue). ................................ ............ 131 Figure 4.S1. Community maps of study regions representing (a) F lorida ( FL ) and (b) M assachusetts ( MA ) built by aggregating individual fcms from each region. ........................... 144 Figure 4.S2. (a) M ean sea surface temperature for 2017. (b) T rends in monthly mean sea surface temperature from 1997 - 2017. ................................ ................................ ................................ ..... 145 Figure 4.S3. Screenshot of survey question used to determine survey respondents target species. The answer to this survey question was then populated into the subsequent mental model survey questions. ................................ ................................ ................................ ................................ .... 146 Figure 4.S4. Screenshot of survey question used to ascribe edge weight relationships among concepts. ................................ ................................ ................................ ................................ ...... 147 Figure 4.S5. Adjacency matrix showing the corresponding relationships among model concepts derived from the online survey. ................................ ................................ ................................ .. 148 1 INTRODUCTION OVERVIEW OF THE PROB LEM unprecedented biodiversity loss, widespread overexploitation of natural resources, and extensive environmental degradation highlight the substantial scale of human influence on the e arth. These far - reaching environmental consequences of anthropogenic disturbances often take place in so called social - ecological systems (SESs) 1,2 , where in humans and the nature interact reciprocally 3 . These couplings between human and natural components typically lead to the emergence of complexity in different contexts and different scales 3,4 . Managing such complexi ty , however, requires adequate understanding about multiple social and environmental components, their two - sided interrelationships, and their resulting dynamics 4 . This understanding can therefore help us better predict the impacts of environmental and so cial perturbations on coupled systems, and how these systems respond to various management decisions and environmental changes . Immediate consequences of such predictions would be an improvement in the sustainability of ecosystems and human societies 5 . Notwithstanding , in many cases, adequate understanding about complex SESs is difficult to achieve due to widespread limitations and shortages in scientific data, knowledge , tools, and methods to model these complex systems 4,6,7 . To fill this gap, study of SES has faced an increased interest in the use of local knowledge of stake holders 6,8,9 e nvironmental and natural resource users who hold valuable knowledge about social - ecological dynamics , s ample the natural environment from their routine interactions w ith SES s through activities like fishing or hunting 10 , and may share information about environmental, policy or social changes across their social networks and generations 11 . These human - nature interactions allow stakeholders to 2 accumulate and refine kno wledge and observations across years and locations (e.g. , anglers moving among lakes) 12 . This local knowledge (LK), also known as local ecological or traditional knowledge, is considered a rich source of information 13,14 , especially in data - poor and data - scare situations where data - driven scientific assessments are very limited , and therefore researchers often attempt to incorporate LK into environmental models and resource management 15,16 . I ncorporating relevant stakeholder input into SES modeling , however, remains fundamentally challenging due to methodological insufficiencies 17,18 . One key challenges associated with this process is the inability to quantify and address uncertainty in L K that leads scientific community to question the quality and v alidity of information provided by non - scientist stakeholders 14 . Although stakeholders represent a free and widespread source of information , almost always, knowledge held by stakeholders represents different levels of expertise and reflects diverse persp ectives 19 . Yet, the unknown accuracy and the wide range of considerably rise concerns about the validity and reliability of using this source of information in scientific processes 20 . To overcome these challenges, it is necessary to advance the formal use of LK in the study of SES through development of innovative approaches that incorporate inputs and, at the same time , enhance the reliability and accuracy of these stakeholder - driven inputs. Therefore, it i s of utmost relevance to foster methodological developments that allow researchers to harness the LK of nonscientist stakeholders and achieve robust understandings about complex human - environment interdependences while meeting a scientifically acceptable accuracy and reliability. 3 OVERVIEW OF THE METH ODOLOGY This dissertation explores the potential for harnessing the c ollective intelligence (CI) of resource stakeholders to advance the formal use of LK in developing better understandings about complex hum an - environment interdependencies and the ir resulting dynamics. CI is a term used to describe a group phenomenon that emerges from the interactions of various individuals such that the group ends up being more intelligent , i.e., more capable of solving prob lems, making decisions, or answering questions than any individual within the group. CI methods rely on the problem - solving efforts o f groups, often based on a proper aggregation of their individual opinions, judgments and knowledge , which can potentially lead to a superior intelligence (aka. a collectively intelligent system) . This property of the collective may enable the group to solve complex problems in a way no individual can accomplish 21 23 . According to this definition, CI can first thought to be a natural phenomenon, common to many species like ants, honeybees , birds, and fish . For examples, groups of army ants foraging for food can collectively form complex organizations (i.e., assemblages ) , such as bridge s out of their b odies to reach disconnected areas 24 ; and s chool s of fish can collectively form gigantic mass es of fish, while escaping from predators, increase the chance s of survival 25 . CI is also a common phenomenon a mong human societies . At the simplest level, h ighly synchronized human groups can achieve physical capabilities above and beyond what individual humans can do (e.g., a group can simply lift heavier objects than what in dividuals can do). In a more sophistic ated manner, h uman societies practice democracy and incorporate public opinions into important decisions to thr ive culturally and economically 26 and o rganizations practice collaborative problem - solving to integrate diverse knowledge and expertise 27 . Impo rtantly, however, in modern days, online interactions among millions of people 4 contribute to shape the public , and yet smart, discourse on health, social , environmental, and political issues: m illions of o nline web users contribute their customized , anecdo tal knowledge to (i.e., Wikipedia) 28 ; and globally distributed c itizen s cientists work together collectively to expand the scale of data collection and contribute to better environmental conservation 29 . By looking at these examples from the nature and human societies , one important question that needs to be addressed u surpasses individual intelligence or problem - solving capabilities ? This has long been a fund amental question for researchers from a range of disciplines to study collectives and has led many theoreticians to explain th e underlying factors that make collectives smarter than individuals: For example, t (1785) 30 explains the power of collective decision making and has been a fundamental theoretical assumption for epistemic democracy and other democratic theories of decision - making characterized by majority voting . In the context of estimation , Fran of 800 people accurately estimating 31 . About a century later, to spotlight in his 2004 book 23 with a series of examples whe re the average response from a large crowd of independent individuals accurately estimate d various quantities while outperforming the majority of individuals . Scott Page offers a theoretical explanation for thi s phenomenon in his 2007 book 32 . He explains that there is noise associated with each individual judgment, and taking the average over a large number of responses filters out the noise of over - and under - estimates, and therefore moving the aggregated response closer to the truth. Based on this theore tical explanation, the 5 crowd error is equal to the mean of individual error s, minus the ir variance . Consequently , as diversity of judgments increases the variance of individual errors increases, and thus the crowd collective error decreases. For that reaso n, Page calls it a Diversity Theorem A nother form of CI frequently observable in socially interacting animal species is known 33 Swarm i ntelligence emerges from the ability of a network of individuals to work together synchronou sly to accomplish complex tasks. This is therefore a common source of CI among social s pecies like ants and honeybees. However, in 2017, Louis Rosenberg proposed that, once connected into real - time systems with synchronous social interactions among members , humans can also amplify their group intelligence by forming human swarms, which can outperform the vast majority of individuals when solvin g problems and making decisions 34 . Even though h umans did not evolve the natural ability to form a s warm intelli gence, w ith the aim of networking technologies, humans can also connect with each other to form We just need the right technology to turn those connections into real - time systems 34 In the 21 st century, b y leveraging the p ower of emerging online technologies , we should CI to address our complex problems we face today . Internet - based t echnologies like o nline surveys, artificial s warming platforms, cyber - enabled micro markets such as Amazon Mechanical Turk, and p rediction m arket tools can help us more conveniently, and at an unprecedented scale , aggregate the knowledge, wisdom, and insights of diverse groups of people distributed all around the world into a single intelligent soluti on to our complex problems. As a result, CI has been shown as a powerful tool for wide - spread application in a range of areas such as innovation management, democratizin g policies, medical diagnostics . Despite promising findings scattered in various fields , there lacks an overarching framework that 6 can reconcile these findings and guide the generation of new forms of CI. C onsiderably less attention has been paid to CI applications in natural resource management and understanding coupled social and environme ntal change s beyond citizen science. As a result, the degree to which a group of local stakeholders can collectively arrive at an amplified intelligence that provides adequate and reliable understanding of complex human - environment relationships remains a largely unexplored (and potentially underutilized) area. This dissertation aims to reconcile theoretical and empirical findings scattered in various fields , develop a general CI framework, and eventually design and implement new forms of CI to fill these g aps . DISSERTATION OUTLINE Firstly, i n chapter 1, the past and current state s of CI theoretical and empirical research from social science s , biological science s , and managerial and political science s are synthesized to develop a n overarching, state - of - the - art framework that guides the generation of new collectively intelligent systems. Based on this framework, new approaches to harness the CI of local stakeholders were designed with the aim of developing robust understandings about complex huma n - environment interactions in SESs . To empirically test these approaches, three real - world case studies were implemented with fisheries examples. In chapter 2, the potential for harnessing the CI of local stakeholders in recreational pike fisheries in Ger many is explored. This study empirically demonstrates how the knowledge of a crowd of local stakeholders, once aggregated through cognitive mapping techniques, can adequately model social - ecological relationships and predict how the inland freshwater lake ecosystems may respond to different management strategies. This study offers methodological guidance for aggregating the input of crowds of resource users to generate high - quality system models . 7 In chapter 3, using a case o f striped b ass fisheries in Massa chusetts, the benefits of pooling local knowledge from a diverse group of stakeholders are explored. Using a novel online mental - w isdom of 31 and more d ive rsity b onus , 35 this study tests the ideas about how pooling informal knowledge from local people, who interact with the natural resources and may not necessarily hold formal scientific knowledge about their environment, may produce accurate and reliable scientific understandings that can inform sustainability decisions. Results demonstrate that the crowdsourced knowledge , once aggregated from a diverse pool of stakeholders as opposed to heterogeneous pools , can generate useful information about complex so cial - ecological interde pendencies, thereby filling in knowledge gaps in li ght of unavoidable uncertainty. Finally, in chapter 4, an example of U.S. Atlantic coasts is used to scale up the CI approaches by crowdsourcing inputs from a very larg e population of stakeholders to predict climate change impacts on ocean fisheries and approximate their behavioral responses to these changes. This study empirically demonstrates that internet - based crowdsourcing approaches can produce accurate patterns of collective perceptions and behavioral responses which are highly aligned with empirical biogeographic patterns of climate change across east coast. These findings, and particularly that human responses to climate change varies regionally and is linked wit h ecosystem changes, are especially important as society continues developing scientific and management plans that consider climate change. Moreover, this work represents one of the largest studies involving stakeholder mental models and overcomes many of the common logistical constraints (e.g., time and effort of in - person interviews) that have typically limited the scale and sp atial coverage of past studies. 8 REFERENCES 9 REFERENCES 1. Ostrom, E. A diagnostic approach for going beyond panaceas. Proceedings of the National Academy of Sciences of the United States of America vol. 104 15181 15187 (2007). 2. Ostrom, E. A general framework for analyzing sustainability of social - ecological systems. Science (80 - . ). 325 , 419 422 (2009). 3. Liu, J. et al . Complexity of coupled human and natural systems. Science vol. 317 1513 1516 (2007). 4. Levin, S. et al. Social - ecological systems as complex adaptive systems: Modeling and policy implications. Environ. Dev. Econ. 18 , 111 132 (2013). 5. Shannon, C. E. A m athematical theory of communication. Bell Syst. Tech. J. 27 , 379 423 (1948). 6. Johannes, R. E. The case for data - less marine resource management: Examples from tropical nearshore finfisheries. Trends Ecol. Evol. 13 , 243 246 (1998). 7. Arlinghaus, R. et al . Governing the recreational dimension of global fisheries. Proceedings of the National Academy of Sciences of the United States of America vol. 116 5209 5213 (2019). 8. Huntington, H., Callaghan, T., Fox, S. & Krupnik, I. Matching traditional and scientif ic observations to detect environmental change: a discussion on Arctic terrestrial ecosystems. Ambio 18 23 (2004). 9. Moller, H., Berkes, F., Lyver, P. O. & Kislalioglu, M. Combining science and traditional ecological knowledge: monitoring populations for co - management. Ecol. Soc. 9 , (2004). 10. Arlinghaus, R. & Krause, J. Wisdom of the crowd and natural resource management. Trends Ecol. Evol. 28 , 8 11 (2013). 11. Barnes, M. L., Lynham, J., Kalberg, K. & Leung, P. Social networks and environmental outcomes. Proc. Natl. Acad. Sci. 113 , 6466 6471 (2016). 12. Carruthers, T. R. et al. Landscape - scale social and ecological outcomes of dynamic angler and fish behaviours: processes, data, and patterns. Can. J. F ish. Aquat. Sci. 76 , 970 988 (2019). 13. Anadón, J. D., Giménez, A. & Ballestar, R. Linking local ecological knowledge and habitat modelling to predict absolute species abundance on large scales. Biodivers. Conserv. 19 , 1443 1454 (2010). 14. Beaudreau, A. H. & Levin, P. S. Advancing the use of local ecological knowledge for assessing data - poor species in coastal ecosystems. Ecol. Appl. 24 , 244 256 (2014). 15. Gray, S. et al. Harnessing the Collective Intelligence of Stakeholders for Conservati on. Front. Ecol. Environ. (2020). 10 16. Aminpour, P. et al. Wisdom of stakeholder crowds in complex social -- ecological systems. Nat. Sustain. 1 9 (2020). 17. Lynam, T., De Jong, W., Sheil, D., Kusumanto, T. & Evans, K. A review of tools for incorporating com munity knowledge, preferences, and values into decision making in natural resources management. Ecol. Soc. 12 , (2007). 18. Strager, M. P. & Rosenberger, R. S. Incorporating stakeholder preferences for land conservation: Weights and measures in spatial MCA. Ecol. Econ. 57 , 627 639 (2006). 19. Lukyanenko, R., Parsons, J. & Wiersma, Y. F. Emerging problems of data quality in citizen science. Conserv. Biol. 30 , 447 449 (2016). 20. Bonter, D. N. & Cooper, C. B. Data validation in citizen science: a case study fr om Project FeederWatch. Front. Ecol. Environ. 10 , 305 307 (2012). 21. Kao, A. B. et al. Counteracting estimation bias and social influence to improve the wisdom of crowds. J. R. Soc. Interface 15 , 20180130 (2018). 22. Kurvers, R. H. J. M., Wolf, M., Naguib , M. & Krause, J. Self - organized flexible leadership promotes collective intelligence in human groups. R. Soc. open Sci. 2 , 150222 (2015). 23. Surowiecki, J. The wisdom of crowds: Why the many are smarter than the few and how collective wisdom shapes business. Econ. Soc. Nations 296 , (2004). 24. Reid, C. R. et al. Army ants dynamically adjust living bridges in response to a cost -- benefit trade - of f. Proc. Natl. Acad. Sci. 112 , 15113 15118 (2015). 25. Magurran, A. E. The adaptive significance of schooling as an anti - predator defence in fish. in Annales Zoologici Fennici 51 66 (1990). 26. Landemore, H. Democratic reason: Politics, collective intellig ence, and the rule of the many . (Princeton University Press, 2017). 27. Navajas, J., Niella, T., Garbulsky, G., Bahrami, B. & Sigman, M. Aggregated knowledge from a small number of debates outperforms the wisdom of large crowds. Nat. Hum. Behav. 2 , 126 132 (2018). 28. Malone, T. W., Laubacher, R. & Dellarocas, C. The collective intelligence genome. MIT Sloan Manag. Rev. 51 , 21 (2010). 29. Bonney, R. et al. Citizen Science: A Developing Tool for Expanding Science Knowledge and Scientific Literacy. Bioscience 59 , 977 984 (2009). 30. Kao, A. B. & Couzin, I. D. Decision accuracy in complex environments is often maximized by small group sizes. Proc. R. Soc. B Biol. Sci. 281 , 20133305 (2014). 31. Galton, F. Vox populi. Nature vol. 75 450 451 (1907). 32. Page, S. E . Making the difference: Applying a logic of diversity. Acad. Manag. Perspect. 21 , 6 20 (2007). 33. Krause, J., Ruxton, G. D. & Krause, S. Swarm intelligence in animals and humans. Trends 11 Ecol. Evol. 25 , 28 34 (2010). 34. Rosenberg, L. & Pescetelli, N. Amp lifying prediction accuracy using swarm AI. in 2017 Intelligent Systems Conference (IntelliSys) 61 65 (2017). 35. Page, S. E. The diversity bonus: How great teams pay off in the knowledge economy . (Princeton University Press, 2019). 12 CHAPTER 1 1 BECOMING INTELLIGENT ABOUT CO LLECTIVE INTELLIGENC E AND PUBLIC POLICY ABSTRACT Twenty - four centuries ago, Protagoras is generally credited for first taking seriously the 1 an idea echoed in Proverbs 11:14 centuries after: there is no counsel, the people fall; but in the multitude of counselors there is (King James Version of the Bible). Today, thanks to 21 st Century technologies, two heads and the counsel of the multitude are possible at the scale of thousa nds, millions, and beyond. Paradoxically, however, even with these technologies in place, the vast potential of collective counsel is widely underused by policy - makers. Additionally, significant fragmentation across the academic fields that study collectiv es in humans and non - human animals limits the cross - disciplinary advancements that have far - reaching implications for policy - making. So how can researchers and policy - makers work together to harness the power of collectives to address ing health and environmental problems? A vast increase in the study of Here we present an ove rarching, state - of - the - art framework for CI that provides guidance for policymakers, communities and researchers in developing new forms of CI for better addressing problems that societies face. 13 1.1 THE POWER OF THE COL LECTIVE Collective intelligence (CI) is a term used to describe a group property that emerges from the int eractions of various individuals such that the group ends up being more intelligent, i . e ., more capable of solving problems , making decisions, or answering questions than any individual within the group. Under the bridge of this definition, CI can be first thought to be a natural phenomenon, common to many species like ants and honeybees but also increasingly as a key form of success in human communities 2 . In many ways, this philosophy acknowledged a natural precursor to modern civilizations leading human societies to thrive culturally and economically. This type of outcome is frequently achieved in human groups through processes ranging from face - to - face de liberation to l arge - scale judgment aggregation to decentralized problem - solving. Based on CI observations from nature, various forms of human organization, and theoretical and experimental advances, our knowledge about successful CI conditions is fast expa nding in the biological sciences, cognitive and behavioral sciences, political and management sciences. Each of these fields offers a different stream of insight into how collectives navigate both simple and complex problems leading to better outcomes. For example, biologists have demonstrated how groups of modestly capable individuals can collectively succeed in highly complex tasks such as nest construction, navigation of an unfamiliar environment, and cohesive migration, e.g., see ref. 3 , or social scien tists have demonstrated how humans, once formed into a collective, can amplify their cognit ive capability, e.g., see ref. 4. At the same time, studies specific to computer and information science have offered several opportunities to deliberately design s ocial or cyber - infrastructures to allow collectives to address a particular problem. This enormous potential, however, is not yet fully accessible to policy - makers because of the lack of an overarching framework to inform the generation of new 14 collectively intelligent systems given a specific objective. Structuring such an overarching framework requires re sponding to several questions: What types of public - policy questions can CI support? What is the nature of the collective and what knowledge is appropriat e for different types of questions? How should this knowledge of the collective be integrated to ensure CI emerges and we avoid the madness of mobs? Here we present (outline) a framework that can guide the design of new collectively intelligent systems and Our outline focuses on three primary components: the nature of the policy problem or challenge, the nature of the collective, and the nature of the aggregation mechanism. 1.2 COLLECTIVE INTELLIGENCE FRAMEWO RK 1.2.1 The problem Defining the public need or challenge with clarity (i.e., the purpose ) is fundamental to designing a CI system. Such a system can be aimed at addressing a wide range of problems for which individuals have to accomplish va rious tasks such as data collection, observation, labor services or cognitive tasks such as processing new information (i.e., acquire and organize knowledge), retrieving that information from memory, and use that information at a later time (e.g. , for esti mation and prediction, making a decision, conducting an analysis, etc.) (Fig ure 1. 1). In addition, the complexity of the problem should be taken into account. Here we use a three - point continuum of complexity (i.e., simple, complex, and wicked) to classi fy problems: Simple problems are clearly defined with an ideal solution that can be obtained in a linear participants only have to decide on a single variable value (e.g., numerical estimate of a 15 quantity). Complex problems can eventually be clearly defined, but unlike simple problems, solutions to complex problems are not well - understood. Such problems are not solvable by reductionist or sequential techniques, and solutions to them are often adaptive and can lead to other problems and unintended consequences. Finally, wicked problems are complex problems which are neither clearly defined nor well - understood. These types of problems involve multiple stakeholders with different values and beliefs. Intelligent systems typically seek to manage wicked problems rather than definitively solve them 5 . 1.2.2 The collective Theorists have suggested various characteristics by which collectives may effectively construct a CI sys tem: The diversity of members of a collective, for example, has been demonstrated to serve a critical role in collective problem - solving 6 . Especially for more complex problems, cognitive diversity is a critical driver of collective performance. Here we us e Hong and Page (2004) method 6 d how they solve it. Further, the skills/expertise of individual participants are of importance to the collective outcome. Three categories can be used to classify the level of expertise people have in CI systems: lay public (individuals who do not necess arily have intimate knowledge, experience, and professional training in the subject of the problem); local stakeholders/communities (individuals likely to be affected by a management decision, action, or a problem); and subject - matter experts (individuals who possess specialized or professional knowledge of a subject). Additionally, one important group characteristic is the group size which has been hypothesized to influence group CI. While some traditional models of collective decision - 16 making, e.g., Condo proposed that collective accuracy should increase monotonically with group size 7 , more recent studies have demonstrated that group size differently impacts the accuracy of collective decisio n - making given the complexity of environmental cues and the correlation of information driving individual decisions 8 . P engagement , or their level of effort and motivation to solve the problem at hand 9 , can influence the design and implement ation of a CI system. Here we expand on Malone CI framework 10 and classify engagement into four overarching categories: ( 1 ) Monetary incentives; ( 2 ) Social responsibility, concerns, and civic duty; ( 3 ) Enjoyment, satisfaction and recognition; and ( 4 ) Legitimate right, ownership, and liability. A final relevant factor is the task management process, which explains how a collective manages the distribution of labor or intellectual contributions. A collective is either self - governed ( decentralized) with autonomous agents or hierarchically controlled (centralized). 1.2.3 The aggregation mechanism Group formation can take place once a collective of individuals are either sampled or self - n is aggregated, however, depends largely on two factors. First, the level of social influence among individuals, which ranges from highly influenced with collaborative and synchronous interactions to highly independent with no social interactions. In thi s case, social influence can take various forms: individuals can either communicate through face - to - face dialogue or through online platforms, referred to as artificial Swarm platforms 11 , which allow users to interact concurrently to make collective decis ions. These interactions are synchronous, meaning that users can explore decision - spaces together in real - time. On the other hand, social interactions can occur 17 asynchronously, meaning that individuals are independently exposed to information about and/or their collective responses, or they receive correlated environmental cues 8 (e.g., people are exposed to the same social media outlets). Given the type of policy problem, social influence may undermine or improve collective performance. In simple estimation tasks, for example, a dominating belief is that social influence may drive out beneficial diversity 12 . Those upholding this belief con tend that connected range of individual judgments 12 . While this can be problematic in more centralized networks, recent studies, e.g., refs. 4,13 have dem onstrated that, in decentralized networks, connected individuals outperform disconnected ones due to the benefits of collective learning. In addition, and especially for complex and uncertain problems, innovation entails social interactions whereby ideas n eed to be recombined. Second, a CI solution requires an aggregation method by which individual inputs are combined. We have identified five general aggregation rules: average rule (i.e., using a central tendency measure); addition rule (i.e., pooling or crowdsourcing information); majority rule (i.e., using voting mechanisms); convergence rule (i.e., reaching a consensus by deliberation or convergence of opinions); and emergence rule (i.e., self - organized recombination of individual inputs emerges to inno vations or better outcomes). 18 Figure 1.1. Collective intelligence framework for policy - making. This framework provide s insights into new hypothetical pathways to aggregate information (i.e., units of knowledge) from a group of individuals and thereby offer solutions that are more optimal than how any single individual could have addressed a particular problem ( examples are provided in the Appendix, Figure 1.S1) . 19 1.3 POLICY IMPLICATIONS While we are only at the beginning of exploring the potential impact of CI methods on public policy, we envision at least three important areas of impact. The first is interdisciplinarity. Choosing the right CI intervention for a given policy challenge will require engaging with fields of knowledge that did not traditionally intersect. The design of a citizen consultation at the level of a large city may benefit from cognitive science and social psychology in question formulation; network science to identify the right diffusion channels; data science and mac hine learning in the treatment of open - text citizen contributions; and design thinking with behavioral economics to redesign a public service. And while the field of CI has long been allied with quantitative disciplines such as computer science 14 , the nex t wave of experiments will benefit from the insights of social science disciplines such as participatory democracy, social psychology, management, peacebuilding, and complex mediation. To this end, we see a natural convergence between the framework for CI studies presented here and the closely related discourses of crowdlaw and public entrepreneurship 15 , epistemic democracy, complex systems, organizational change, and behavioral insights or nudge theory 16 . Second, a common CI framework should allow policy makers to eliminate many of the false choices that dominate current political debates. Harnessing the CI of a community or country does not necessarily mean calling a referendum or overturning a government. On the contrary, the field of CI provides a multi tude of methods and techniques at the disposal of policymakers: some relevant to decision - making, but others to collective observation, interpretation, prediction, or preservation of common knowledge. Practitioners of CI may include those seeking radical c hanges in existing institutions, but so too can they be faithful stewards of 20 them. In all cases, institutionalizing CI methods will require an understanding of how to supplement and not necessarily replace existing representatives and intermediaries. Fina lly, a common CI framework can serve as a basis for new coalitions of scientists, policymakers and citizens that will be necessary to take the most promising CI methods to scale. Many promising CI pilots never achieve the necessary institutional buy - in to create a long - term impact. As such, an increasing amount of attention is being given to the conditions for institutionalizing CI processes, including the need to link new participatory channels to performance indicators of managers and public servants 17 . To name only a few, the work of the NYU GovLab, NESTA Centre for Collective Intelligence Design, OECD Future of Democracy Network, World Bank Open Government Unit, Democracy R&D Network, and EU Horizon 2020 CI fund seek to develop the link between scientif ic research into what works and a hard - nosed understanding of what lasts. As this research agenda expands, the network of researchers exploring CI principles should itself embody those principles. This means creating more diverse data - gathering channels, i Intelligence in Morocco. It means more opportunities to pool knowledge in innovative ways, such as the virtual CI conference in June 2020 hosted by Northeastern Un iversity and Copenhagen Business School, and it means the epistemic humility practiced by researchers and dialogue facilitators alike: in shaping this new discipline for policy - making, we must be vigilant against our own biases and ever - ready to overturn o ur presumptions if new evidence comes to light. What is the future of CI? At a minimum, these methods have already shown the promise of a more agile and inclusive policy - making framework, in which current priorities are more 21 easily achieved and existing in stitutions reap the benefit of higher effectiveness and greater trust. Conversely, it may entail a more profound paradigm shift in which existing models give way to more radically decentralized or distributed systems. B ut whether we are able to define, stu dy and implement the field of CI will determine if we can collectively address our shared problems or not . 22 APPENDIX 23 APPENDIX SUPPLEMENTARY INFORM ATION S1 Collective Intelligence Examples Here we have identified 21 unique approaches that exemplify most of the CI systems that - world problems. These examples include citizen science 18 , micro - task markets 19 (e.g., Amazon Mechanical Turk ), human swarm intelligence 20 (e.g. , Swarm AI technology ), traditional wisdom of crowds 7 , wisdom of decentralized networked crowds 13 , knowledge co - production 21 , distributed governance 22 ( Blockchains technology), epistemic democracy 23 , social bookmarking 24 (Folksonomy), Delphi methods 25 , prediction markets 26 , adaptive co - management and community engaged studies 27 , open innovation and broadcast search 28 (e.g., idea competitions), open problem - solving 29 (e.g., MIT Climate CoLab ), commons - based peer production 30 , deliberative democracy 31 , mass collaborati on 32 (e.g., Linux ), collective memory 33 (e.g., Wikipedia ), wisdom of stakeholder crowds in complex problems 34 (e.g., social - ecological modeling), wisdom of crowds in combinatorial problem - solving 35 (e.g., traveling salesperson problem and minimum spann ing tree problem ), and diversity trumps ability theorem 6 . These examples are shown in Figure 1.S1, each demonstrates a unique array of sub - components from three main CI components: the problem, the collective, and the aggregation mechanism. In addition, t he Sankey diagram displayed in Fig ure 1.S2 is a flow diagram, in which the width of the arrows represents proportionally the flow quantity between two sub - components of the CI framework, based on 21 aforementioned examples. 24 S 2 Supplementary Figures Figure 1.S1. Examples of collective intelligence (CI). Each example demonstrates a unique approach to harness the CI of a collective by aggregating individual inputs to solve a particular problem. See Figure 1.1 for more information about sub - ca tegories (i.e., PU, CX, DV, EX, GS, EN, TM, GF, SI , and AG ). 25 Figure 1 .S2. The Sankey diagram showing the flow between sub - components of CI framework. C onsider ing examples from Figure 1.S1 , the diagram shows where a CI can come from and where it can end up, w ith possible intermediate steps, where the width of the connections between two nodes visualizes the quantity of examples used these pairs of nodes (i.e., sub - components). 26 REFERENCES 27 REFERENCES 1. Hatzistavrou, A. Aristotle on the Authority of the Many: Politics III 11, 1281a40 b21. Apeiron (2019). 2. Sapolsky, R. M. Behave: The biology of humans at our best and worst . (Penguin, 2017). 3. Gordon, D. M. Collective Wisdom of Ants. Sci. Am. 314 , 44 47 (2016). 4. Kurvers, R. H. J. M., Wolf, M., Naguib, M. & Krause, J. Self - organized flexible leadership promotes collective intelligence in human groups. R. Soc. open Sci. 2 , 150222 (2015). 5. Rittel, H. W. J. & Webber, M. M. Dilemmas in a general theory of planning. Policy Sci. 4 , 155 169 (1973). 6. Hong, L. & Page, S. E. Groups of diverse problem solvers can outperform groups of high - ability problem solvers. Proc. Natl. Acad. Sci. 101 , 16385 16389 (2004). 7. Surowiecki, J. The wisdom of crowds: Why the many are smarter than the few and how collective wisdom shapes business. Econ. Soc. Nations 296 , (2004). 8. Kao, A. B. & Couzin, I. D. Decision accuracy in complex environments is often maximized by small group sizes. Proc. R. Soc. B Biol. Sci. 281 , 20133305 (2014). 9. Bonabeau, E. Decisions 2.0: The power of collective intelligence. MIT Sloan Manag. Rev. 50 , 45 (2009). 10. Malone, T. W., Laubacher, R. & Dellarocas, C. The collective intelligence genome. MIT Sloan Manag. Rev. 51 , 21 (2010). 11. Rosenberg, L. & Pescetelli, N. Amplifying prediction accuracy using swarm AI. in 2017 Intelligent Systems Conference (IntelliSys) 61 65 (2017). 12. Lorenz, J., Rauhut, H., Schweitzer, F. & Helbing, D. How social influen ce can undermine the wisdom of crowd effect. Proc. Natl. Acad. Sci. U. S. A. 108 , 9020 9025 (2011). 13. Becker, J., Brackbill, D. & Centola, D. Network dynamics of social influence in the wisdom of crowds. Proc. Natl. Acad. Sci. 114 , E5070 -- E5076 (2017). 1 4. Mulgan, G. Big mind: How collective intelligence can change our world . (Princeton University Press, 2018). 15. Noveck, B. S. Crowdlaw: Collective intelligence and lawmaking. Anal. Krit. 40 , 359 380 (2018). 16. Thaler, R. H. & Sunstein, C. R. Nudge: impr oving decisions about health. Wealth, and Happiness 6 , (2008). 17. Peixoto, T. & Steinberg, T. Citizen Engagement: Emerging Digital Technologies Create New Risks and Value. (2019). 18. Bonney, R. et al. Citizen Science: A Developing Tool for Expanding Scie nce Knowledge 28 and Scientific Literacy. Bioscience 59 , 977 984 (2009). 19. Kittur, A., Chi, E. H. & Suh, B. Crowdsourcing user studies with Mechanical Turk. in Proceedings of the SIGCHI conference on human factors in computing systems 453 456 (2008). 20. Ro senberg, L. B. Human Swarms, a real - time method for collective intelligence. in Artificial Life Conference Proceedings 13 658 659 (2015). 21. Jasanoff, S. States of knowledge: the co - production of science and the social order . (Routledge, 2004). 22. Ølnes, S., Ubacht, J. & Janssen, M. Blockchain in government: Benefits and implications of distributed ledger technology for information sharing. (2017). 23. List, C. & Goodin, R. E. Epistemic democracy: Generalizing the Condorcet jury theorem. J. Polit. Philos. 9 , 277 306 (2001). 24. Golder, S. A. & Huberman, B. A. Usage patterns of collaborative tagging systems. J. Inf. Sci. 32 , 198 208 (2006). 25. Linstone, H. A., Turoff, M. & others. The delphi method . (Addison - Wesley Reading, MA, 1975). 26. Wolfers, J. & Zit zewitz, E. Prediction markets. J. Econ. Perspect. 18 , 107 126 (2004). 27. Armitage, D., Berkes, F., Dale, A., Kocho - Schellenberg, E. & Patton, E. Co - management and the co - Glob. Environ. Chang. 21 , 995 1004 (2011). 28. Jeppesen, L. B. & Lakhani, K. R. Marginality and problem - solving effectiveness in broadcast search. Organ. Sci. 21 , 1016 1033 (2010). 29. Malone, T. W. et al. Putting the pieces back together again: Contest webs for large - scale pro blem solving. in Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing 1661 1674 (2017). 30. Benkler, Y. & Nissenbaum, H. Commons - based peer production and virtue. J. Polit. Philos. 14 , 394 419 (2006). 31. Lande more, H. Deliberative democracy as open, not (just) representative democracy. Daedalus 146 , 51 63 (2017). 32. Benkler, Y., Shaw, A. & Hill, B. M. Peer production: A form of collective intelligence. Handb. Collect. Intell. 175 , (2015). 33. Garcia - Gavilanes, R., Mollgaard, A., Tsvetkova, M. & Yasseri, T. The memory remains: Understanding collective memory in the digital age. Sci. Adv. 3 , e1602368 (2017). 34. Aminpour, P. et al. Wisdom of stakeholder crowds in complex social -- ecological systems. Nat. Sustain. 1 9 (2020). 35. Yi, S. K. M., Steyvers, M., Lee, M. D. & Dry, M. J. The Wisdom of the Crowd in Combinatorial Problems. Cogn. Sci. 36 , 452 470 (2012). 29 CHAPTER 2 2 WISDOM OF STAKEHOLDE R CROW DS IN COMPLEX SOCIAL - ECOLOGICAL SYSTEMS This chapter is a reprint of an original peer - reviewed article published in Nature Sustainability in 2020, volume 3, on pages 191 - 199. The original article can be found at: https://doi.org/10.1038/s41893 - 019 - 0467 - z . ABSTRACT Sustainable management of natu ral resources requires adequate scientific knowledge about complex relationships between human and natural systems. Such understanding is difficult to achieve in many contexts due to data scarcity and knowledge limitations. We explore the potential of harn essing the collective intelligence of resource stakeholders to overcome this challenge. Using a fisheries example, we show that by aggregating the system knowledge held by stakeholders through graphical mental models, a crowd of diverse resource users prod uces a system model of social - ecological relationships that is comparable to the best scientific understanding. We show that the averaged model from a crowd of diverse resource users outperforms those of more homogeneous groups. Importantly, however, we fi nd that the averaged model from a larger sample of individuals can perform worse than one constructed from a smaller sample. However, when averaging mental models within stakeholder - specific subgroups and subsequently aggregating across subgroup models, th e effect is reversed. Our work identifies an inexpensive, yet robust way to develop scientific understanding of complex social - ecological systems by leveraging the collective wisdom of nonscientist stakeholders. 30 2.1 INTRODUCTION Many environmental problems tha t influence human well - being, such as climate change, biodiversity loss, and overexploitation of natural resources , are caused by a combination of social and ecological factors that occur in coupl ed systems across scales 1 . Managing resources under such com plexity requires adequate syste m representation (i.e., models ) 2 , so that the before action is taking. However, the sheer number of intimately linked social - ecological systems that require management, limited knowledge and resources, and the difficulty to enumerate many s ystem elements cause scientific model creation to lag behind decision - making needs in most natural resource contexts 3,4 . This limits the effectiveness of natural resource management and contributes to the inevitable collapse of many exploited systems , such as fisheries 5 . To address knowledge limitations and data gaps, resource managers frequently receive input and decision - making supp ort from resource stakeholders 6,7 . Resource users sample the natural environment through their routine interactions with social - ecological systems (e.g. , while fishing or hunting) 4 and thus accumulate and refine knowledge and observations over years and, frequently, in different locations (e .g., anglers moving among lakes ) 8 . Therefore, m onitoring and assessment of natural resource dynamics may be improved by leaning on the knowledge of diverse resource stakeholder s (e.g., fishers) 7 in ways that harness their collective intelligence (CI) 9 the ability of a group to solve problems effectively. For example, natural resource management increasingly uses citizen scientists 10 to collect and aggregate observational data (e.g., by observing bird distribution a nd abundance ) 11 . Importantly, the CI held by a group can also be harnessed by pooling judgments, rather than observations, from large The so - called wisdom - 31 of - crowds (WOC) phenomenon was discovered more than a hundred years ago, when the average judgment of the crowd of observers accurately estimated the weight of a dead ox 12 . This phenomenon frequently leads to surprisingly accurate point estimates by averaging the judgments of a large collective 1 3 . In addition to simple estimation tasks, a W OC effect has also been researched in cases of higher solution complexity , such as combinatorial problems 14,15 . U nderstanding the complex social - ecological interactions in natural resource ecosystems , however, constitutes a considerably more difficult problem than counting the number of birds 10 , guessing the weight of an animal 12 , or solving a Euclidean traveling salesperson problem 15 . Natural resource managers frequently have to predict future system states (e.g., in response to a planned management intervention ), which requires more complex knowledge about the structure, connection, and dynamic behavior of natural resource systems , often associated with high solution . It is currently unclear if the WOC approach can harness CI for such complex problem - solving conditions . In this work, we explore if the WOC can be leveraged to provide accurate system knowledge about natural resources. Specifically, u sing a case from fisheries, we ask: c an crowds of non - scientist resource user s provide representations of the ecological and social cause - and - effect relationships that drive resource stock dynamics and mirror the best scientific understanding of the same s ocial - ecological context ? Given the urgent need to effectively manage globally declining fish stocks 16 , 17 , this is a question of utmost relevance: if stakeholder crowds can provide accurate representations of complex social - ecological relationships , then by using the CI of stakeholders we could create a more complete coverage of localized social - ecological processes than any team of scientists can ever achieve when traditional scientifically - 32 driven assessments are limited and cannot cover the universe of l ocal environmental and social interactions. One way to elicit system representations from stakeholders is through cognitive maps. These are graphical models of system elements (concepts) and their causal connections (represented as signed arrows) . T hey rep resent of external reality, referred to as mental models 18 . M ental models of complex systems can be represented in special semi - quantitative form s of cognitive map s called Fuzzy Cognitive Maps (FCM) 19,20 . Importantly, i ndividual mental models elicited by FCMs can be aggregated mathematically to create a model that represents the insights of all subjects 19,20 . However, there is a lack of ntal models about complex social - ecological relationships, such as human interactions with natural fish populations 21 . In this study, we explore using WOC principles to establish a presumably accurate understanding of natural resource dynamics by proposin g and testing a novel approach for aggregating individual mental models collected from non - scientist stakeholders. We use an example of a recreational fishery ecosystem and independently generated mental models, represented by FCMs, from diverse resource u ser s, composed of individuals who interact with fishery resources in different ways , either through exploiting fish populations (anglers ), managing resources ( fisheries managers ) or governing communities of resource users (angling club managers) . I n gene ral, and especially for complex problems with many interrelated components, incorporating diverse knowledge and expertise into collective problem - solving improves the 22 24 . Similarly, diversity of perspectives has been identified as a critical 33 driver of WOC 13 . Building on earlier theoretical reasoning 22 24 , we hypothesize, first, that, a system model generated by aggregating the mental models of a crowd of diverse resource users outperforms the models of more homogeneous groups (H 1 ). Y et, it is realistic to assume that users of the same social - ecological system are most likely to be socially influenced by their peers in real life , especially by those from the same stakeholder category (e.g., anglers, club managers, and fisheries manager s) , due to similarities in the ways they use and interact with the natural resources. Such interactions can be direct through face - to - face communications or indirect through sharing knowledge, information and assumptions over media and through being expose d to a similar set of information sources (e.g., educational material codified in books). Socially influenced subgroups of individuals, however, tend to accumulate and represent correlated knowledge. Despite potentials for social learning and improving the accuracy of the collective judgment s, p rior WOC studies 25 27 have show n that under such conditions, averaging data points from a larger crowd of individuals increases the risk of am plifying biased knowledge that drives from direct or indirect exposure to social influences, thereby potentially diminishing the WOC effect 25 27 . Therefore, we hypothesize, second, that, when arithmetically averaging mental models of stakeholders with plausible real - life social influence, larger samples of mental mo dels may amplify the negative effect social influence can have on WOC, thereby deteriorating collective performance as crowd size increases (H 2 ). To deal with the latter issue, past t heoretical and empirical WOC s tudies 26,28 have suggested that, once the of individuals whose opinions are more likely to be directly or indirectly influenced by their subgroup peers) , the WOC can be enhanced by averaging responses across modules 26,28 . Ass uming that the crowd is suffering from the possible negative effect social influence can have 34 on WOC 27 and b uilding on prior theoretical work 26 , we hypothesize, third, that a multi - level aggregation method that averages mental models within modules (i.e. , subgroups of stakeholders from the same user type category), followed by a subsequent aggregation across modules, can dampen the negative effect of social influence (H 3 ). This multi - level aggregation approach may compensate for the possibly harmful biase s as a result of social influence, thereby allowing larger crowds to demonstrate an improved WOC effect. Our work tests the above mentioned three hypotheses and thereby establishes that WOC can be leveraged to crowdsource system knowledge of social - ecological and other complex systems , while also offering methodological guidance for aggregating the input of crowds of resource users to generate high - quality system models similar to those developed by trained scientists . Our findings provide the basis for managing and planning interventions in complex social - ecological systems that are data - poor or even data - deficient, but that have an abundance of local knowledge from resource users. 2.2 EXPERIMENTAL DESIGN We collected g raphical mental models o f 218 stakeholders characterized as recreational anglers, angling club managers, and fisheries managers through a fuzzy cognitive mapping task in a series of workshops in angling clubs recruited from north - western Germany. The FCMs represented participant understanding of the fish ecology and fishery management regarding the northern pike ( Esox luciu s) fishery (see a previous publication for more details ) 21 . T he individual ly collected mental models graphically displayed the perceived cause - and - effect relat ionships of ecological and social concepts affecting each other ( see Appendix , Fig ure 2.S 1). Additionally , we ran two FCM workshops with 17 fishery scientists , each of whom had formal training and sci entific knowledge in fishery resource dynamics and pike ecology , to create a 35 scientific reference mental model representing the best scientific understanding about the same ecosystem. W e experiment ed on various ways to draw and aggregate mental models fr om a population of stakeholders to explore the impact of diversity, possible biases raised because of real - life social influences, and aggregation methods on the WOC . The effects were quantified by comparing the aggregated mental models against the scientific reference mental model (i.e., e . We used two aggregation methods: (a) Single - level that is accomplished by arithmetically averaging the weights of all individually contributed links in FCMs of group members (see previous publication s for more details) 19,20 , and (b) Multi - level that first di vides the stakeholders into separate modules (i.e., smaller subgroups) and arithmetically average s the edge weights of all contributi ng maps within each module, and t hen in the second level, it use s the median t o aggregate the maps across the modules (see Methods) . We proposed to use median in the second level of aggregation because the median has been shown to outperform the arithmetic mean in likely skewed distributions 12,29,30 . We used the single - level aggregation method t o form the averaged mental models of stakeholder - specific groups with members only from one stakeholder category (i.e., homogeneous groups of anglers, club managers, and fisheries managers). We also aggregated all 218 individual mental models usin g the multi - level aggregation method to construct a crowd mental model composed of diverse stakeholders. To create the scientific reference model we aggregated the mental models generated by 17 scientists (i.e., experts) using single - level aggregation meth od . We compared stakeholder - model (i.e., reference model ) in terms of their (a) centrality of concepts representing pike 36 ecology a nd management, (b) strong cause - and - effect relationships , (c) network geometr ic structures, and (d) dynamic behavior ( see Methods ). - models made by a random graph generator using the probability distribution of edge weights drawn from th We used this null - unwise model to test that any observed WOC is not simply an artifact of averaging mental models, and is - world relevant knowledge. Finally, to test the imp act of accumulated biases of socially influenced individuals on the WOC effect and the success of different aggregation methods in filtering out these biases, we formed numerous samples of individuals randomly drawn from the entire population of 218 stakeh olders with different sample sizes. For each random sample of individuals we aggregated their mental models using two aggregation methods: single - level and multi - level. We then computed an overall performance error by comparing the aggregated mental model against the expert s mental model (see Methods for details). 2.3 RESULTS We find that the structural properties of the crowd mental model match scientific understanding about the social - ecological relationships driving pike fisheries, This was evidenced by eva luating agreement between the crowd model and the scientific model using three metrics : (a) centrality index ( which represents the relative importance of a concept in the mental model ) , (b) strong causal patterns ( which represents the arrangement of strong cause - and - effect relationships ) , and (c) graph eigenvalues (which represent hidden fundamental patterns of geometric structure that has implications for the networked functionality of a mental model ). The centrality measures ( see Methods) indicated that the three stakeholder - specific groups were 37 biased toward specific management strategies ( e.g., anglers were biased toward angling pressure being particularly impactful for pike, and fisheries and club managers were biased toward enhancement of habitat quality promoting pike ) (Fig ure 2. 1). However, in support of our first hypothesis (H 1 ), when the mental models of all diverse stakeholders were aggregated, the crowd model demonstrated remarkable similarity to the experts ( i.e., the reference mo del) regarding the centrality of six important concepts for possible impacts of fishery management decisions on pike population (Figure 2. 1) . Figure 2.1. Centrality profiles of different groups (in color) and the expert reference model (in black/grey). Axes in the radar charts show the centrality of system elements that are important for fishery management decisions. Katz index is used to measure the centrality (see Methods ). The crowd also showed the highest agreement with the reference model regarding the strongest cause - and - effect relationships in pike ecology and management (Figure 2.2 ) . Additionally, the eigenvalue similarity index ( see Methods) also indicated that the structure of the crowd mental model had the most similar fundamental characteristics to the experts, 38 suggesting yet again signif icant structural agreement (Figure 2. 3) . We , therefore, conclude that the structure of the mental model of the crowd is very similar to the one produced by experts and thus, a WOC effect is demonstrated. Figure 2.2. Agreement on strong causal patterns in the FCM of stakeholder - specific groups, the crowd and the experts. The crowd map has the highest degree of matched patterns (~70% matched) with experts; the stake holder - specific groups perform substantially better (among 53% to 63% of correct matches) than the null - unwise model (only ~30% correct matches). Weak relationships with an edge weight less than 0.33 (the first tertile in zero to one continuum, correspondi ng to the weak interval) were removed from the maps to get the strong causal patterns (see Appendix , Figure 2. S 2 ). Error bars display standard errors. 39 Figure 2.3. Eigenvalue similarity index. Within each group (x axis), each point represents one individu al and is placed according to the eigenvalue similarity index (y axis). The similarity index represents the structural mismatch with the experts mental model. The swarm plots reflect the density of points around any distance value, while black squares repr esent the aggregated mental models for each group. The crowd model has the smallest distance from (highest similarity with) the experts model. Interestingly, for all stakeholder groups, aggregated model is located below the densest area of the plot, illust rating the WOC effect (the average model outperforms most individuals). Yet, this effect is notably higher in the crowd. Structure does not necessarily provide insights into how the fishery might react under changing social - ecological conditions . We , there fore, assessed the dynamic ( i.e., functional) behavior of the FCMs by simulating how changes in one or more system element s of the mental models impacted the state of all system elements ( see Methods ). We find again in support of H 1 that the functional pro perties of the crowd mental model accurately match scientific understanding about pike ecology (Fig ure 2. 4). We revealed this agreement using a measure of dynamic distance , which represents the mismatch between two models in terms of the outcomes they produce as a result of changes in the state of one or more concepts . The functional properties of the mental models generated by the crowd and experts aligned, where the mean of 40 the dynamic distance between experts and the crowd was the lowest compared to all stakeholder - specific groups (Figure 2. 4). Figure 2.4. The dynamic distance between the experts model and the stakeholder - derived models based on 10,000 randomly generated scenarios (experiment s). Each experiment randomly selects a set of concepts (nodes in the FCMs) and changes their values to produce outputs (see Methods ). ( a ) Each cell in the color - bar graph represents a random scenario with colors denoting the dynamic distance. ( b ) Boxplots illustrate the distribution of these dynamic distances for each group in 10,000 experiments. The mean of dynamic distances from the reference model is the smallest in the crowd. We also find that the impact of biases deriving from real - life social influenc es exhibited two distinct behaviors across different aggregation methods affecting WOC performance (Figure 2.5). For the models built by single - level aggregation larger samples of stakeholders amplify the 41 accumulation of biases, and thus group performance error increases at larger sizes in agreement with H 2 (Fig ure 2. 5 a). By contrast, confirming H 3, f or the models built by multi - level aggregation larger samples of stakeholders cancel out the biases, and therefore group performance error decreases monotonic ally as more data point s are drawn from the population of mental models (Fig ure 2. 5 b). Consistent with prior theoretical and em pirical works 26,28 , we collectively show that the WOC effect is indeed observed for the large crowd s, which consist of multiple socially influenced subgroups of stakeholders (i.e., modules), but only if multi - level aggregation , as opposed to single - level aggregation, is used (Figure 2. 5). This result supports previous experimental studies 15,27 , implyin g that social influences and their resulting biases , if not appropriately harnessed 31 , will undermine the WOC effect in large crowds when averaging individually collected mental models about natural resource dynamics, at least under the conditions of our study context. Figure 2.5. The sampling and averaging effect on performance error in crowds built by drawing and aggregating mental models using two aggregation methods . ( a ) Single level aggregation. ( b ) Multi - level aggregation. Samples were formed by randomly drawing individuals from all 218 participants. Data are shown for 100 repeats per sample size. (Test of > 100 random crowd assignment show no significant difference). 42 2.4 DISCUSSION This study advances the science of CI by examining how the WOC can be leveraged to crowdsource mental models of social - eco logical systems . We demonstrate that a large - enough group of diverse and informed stakeholders, who pool their mental models, can provide system descriptions that mirror t he representations of system knowled ge by scientific experts (Figures 2. 1 - 2. 3). This is an important finding because there is a widespread lack of monitoring data and scientific models in many freshwater and small - scale coastal fisheries and other exploite d ecosystems with which resource users regularly interact 5 . Our work supports an earlier hypothesis 4 that the knowledge of these local resource stakeholders can be mined to provide the insights necessary for sustainably managing exploited ecosystems and p revent ing their collapse. diversity theorem 22 , we found that the system model generated by a crowd of diverse individuals could potentially outperform the model s of stakeholder - specific groups . However, assuming that the cr owd is suffering from the negative effect social influence can have on WOC , and consistent with recent theoretical work 25,26 , we also demonstrate that larger crowds do not necessarily perform better . Instead, a ggregating more data points (i.e., individual mental models) in larger crowds may decrease performance under certain conditions (Fig ure 2. 5 a ) . Our work thus extends prior theoretical work 25,26 by providing empirical evidence that shows the importance of knowledge distribution and aggregation methods in WOC tasks where correlated information could decrease performance with increasing group size. The multi - level aggregation method that we presen t offers a solution : it first creates sub group - level models for user groups assumed to be under group - specific social influences, based on the arithmetic mean (i.e., filtering out system aspects the subgroups did not agree on, 43 thus reducing variance) . It s ubsequently aggregates those initially formed sub group models, using the median of the mean values (i.e., reintroducing variance) . This approach benefits from both modularity that creates a situation of optimum knowledge variation 26 and compensate s for a skewed distribution of opinions through using the median rather than mean , thereby enabling improved performance in larger crowds (Fig ure 2. 5 b ). This is a very important finding for guiding the application of the methods we present in a natural resource c ontext. Even though simple aggregation can provide accurate models with an optimal (small) sample size (Figure 2.5 a) , in our case, this optimal sample size that minimizes the group performance error 25 is not theoretically quantifiable due to the unknown correlation between individual beliefs. Our multi - level aggregation method addresses th is issue by triggering a WOC effect that monotonically improves with relatively larger sample sizes . This is of practical relevance for sustainable n atural resources management , where often no robust criterion exists for exclusion of some stakeholders , and contrarily , an unbounded encouraged for democratic reasons 32 (see Appendix , Supplementary Dis cussion and Figure 2.S 3 ). A few limitations are worth outlining. First, w e used a particular format, namely FCM 33 , to capture , represent , and aggregate mental models , as well as to explore the structure and dynamic behavior of the system these models represent. Other system formats may lead to different results. Also, while great care went into the selection of experts, knowledge elicitation, modeling, and model testing, we cannot claim that the reference model is the best possible representation of t does it necessarily represent the best known science. H owever, because any existing limitation of the reference model equally applies to the other models in this study , our conclusions regarding the WOC for obtaining system knowledge from stakeholders remain robust . A further limitation is that our 44 findings were generated from a specific natural resource management context with a unique governance system : in Germany, local level - angling clubs are self - managing their privately owned fishery resources 34 , and both managers and anglers have to pass training that exposes them to concepts of aquatic ecology, fisheries management and conservation and fisheries legislation 35,36 . This, in addition to the workshop settings for data collection, which likely attracted more avid and experienced anglers, means that our sample likely included ecologically interested and rather educated anglers. It is therefore uncertain whether our results translate one - to - one to other natural resource systems, where resource users are less heavily engaged in the local management of resource systems. Also, in our experiments, all participants were provided with a standardized list of system components (see Methods) in favor of m odel comparability 37 . Therefore, the extent to which our findings would apply to situations where there are considerable debates concerning the constituents of a system is unknown. Finally, while our findings demonstrate that WOC can be leveraged to provi de accurate system representations, it is unknown whether the crowd has the ability to quantify the status of natural resources, assess human pressures on them, and derive sustainable harvest rates all of which are critical components of sustainable mana gement 4,5 . These are important directions for future research. not only approximated the structure, but also the dynamic behavior of the scientist - provided model in resp onse to changes (Figure 2. 4). This is very relevant for designing inclusive processes and adaptive co - management practices that require stakeholders, managers, and scientists first model likely outcome scenarios and then jointly agree on possible managemen t actions for uncertain ecosystems 6,38 . While frequently proposed to manage uncertainty in social - ecological systems, such adaptive management approaches often suffer from a lack of readily available 45 simulation models 6,39 . Based on our work, we instead r ecommend proactively involving local stakeholders in system simulation by aggregating individual mental models resulting from online or other survey means . To conclude, we found that robust scientific information of com plex ecosystem dynamics can be genera ted from a group of informed stakeholders. In fact, when done at the right scale and for the appropriate problem, leveraging the CI of stakeholders through a crowd - sourcing approach can be a stepping - stone for fostering institutional fit 40 and accommodati ng nested governance in environmental decision - making 41,42 . For example, certain natural resource problems are local in orientation (e.g., overfishing of a coastal fishery for non - migratory fish, such as coastal pike), but are still data - deficient and in need of urgent conservation action that is agreed - upon by local communities of resource users. Here, harnessing local stakeholder knowledge through a systematic approach, as proposed in our study, can provide much - needed information for sustainability. Whe n this information is paired with also granting local users sovereignty for making local conservation decisions, we can anticipate increased legitimacy of the resulting management actions 17 . Management authorities at larger scales (e.g., regional, nationa l, or international) can, in turn, focus on environmental problems operating at those scales, yet their decisions might also be influenced by harnessing the CI of regionally operating stakeholders. Ultimately, collecting system understanding may operate in a nested fashion by first organizing understanding at lower levels through user group - specific mental models, with ultimate decisions being coordinated at higher levels through across - group models (see Supplementary Discussion) . Despite its promise , our work also clearly shows the importance of carefully designing WOC approaches in natural resource contexts. In particular , if the wrong aggregation method is 46 chosen , increasing sample size can produce a solution worse than one produ ced by intermediate sampl may become unreliable. Fortunately, perhaps, when using the right aggregation method, as we show, a large enough crowd of diverse stakeholders can produce a science - li ke understanding of even complex social - ecological dynamics . While further research is needed to confirm the application of WOC to natural resource contexts, it has considerable potential for addressing pertinent problems of unsustainable natural resource use and biodiversity loss. 2.5 METHODS 2.5.1 Description of study system and context Many global fisheries are in trouble 5,43 . Harvest regulations and stocking practices have been promoted as a common management response in inland and marine fisheries 44 . While sto cking is a common management practice for freshwater fisheries around the world, researchers have recently beg u n questioning the sustainability of these decisions , given their negative consequences and the highly uncertain context in which many of these de cisions are made 45 . Alternative and complementary management options to stocking include social wellbeing - oriented measures (e.g., decreasing angling pressure through input controls) and habitat rehabilitation policies (e.g., increasing spawning habitat, increasing refuge, and increasing riparian vegetation) 46 . The degree to which different fishery decision - makers understand the ecological and social tradeoffs of management decisions is currently not well understood 47 , and there is abundant documentation that fisheries stakeholders and managers find themselves in disagreement about which policy to follow 48 . Moreover, it is notoriously difficult to understand social - ecological interactions and how various ecological factors affect the productive capacity of renewable natural resources striving in the natural ecosystem. The 47 problem is elevated in inland fisheries given the multitude of ecosystems that exist in water - rich landscapes. The multifaceted origin of the fisheries system gives rise to a complex soc ial - ecological problem with substantial data - deficiencies , which lends itself for an investigation of WOC effect for complex system modeling: stakeholders who either use or manage the fisheries interact with the system in different ways and thus accumulate diverse system knowledge that results in different mental models of the structure and function of the system 21 . These different mental models could be mined in WOC applications to harness their CI . Germany offers a compelling case for application of WOC as many local recreational fisheries are managed by angler communities , organized in angling clubs 34 . As opposed to open access systems in the USA and other regions of the world, in Germany as in much of central Europe, angler communities own or lease fishing rights from water owners and in this position have sovereignty to engage in certain management a ctions (e.g., stocking, increasing harvest regulations). Angling clubs range in the number of 10, 000 in Germany alone, meaning that there are 10.000 or more individual decision makers born out the natural resource user community themselves. Roles in anglin g clubs differ with some anglers becoming elected as club managers, mainly tasked with running the voluntary body. On the other hand, selected anglers take training courses in fisheries management and become fisheries managers or water bailiffs taking over the management tasks. A further group entails ordinary anglers who in Germany also have to pass a 30 hour training course to acquire a fishing license and be allowed to join angling clubs. The content of the angling course is mainly directed to legal and practical issues 35,36 fisheries knowledge and education metrics assessed using questionnaires after the mental model exercises (see Appendix , Table 2.S 3). 48 2.5.2 Mental model s Mental models about social - ecological systems, and in fact any type of system, can be elicited and represented as Fuzzy Cognitive Maps (FCM) 33 . These can be analyzed with regard to structure and dynamic behavior of the system 49 . Moreover, FCMs from individuals can be aggregated into a larger FCM that represents the collect ive knowledge of all contributors 20 and thus provide a tool for WOC. In this study we used the FCM format to collect data from a crowd of 218 stakeholders who manage their own lake and river section fisheries in Germany 34 : recreational anglers, who are organized in clubs, fishery club managers, and fisheries managers, who are responsible for the entire ecosystem. In addition, we collected the system models from 17 fishery scientist and used their model for comparison. Between 10 and 20 anglers, managers and club heads of Lower Saxony, Germany, were invited to one of our 17 workshops (for details see a previous publication) 36 , where graphic mental model representations of the ecology and fishery management of the model species lected through Fuzzy Cognitive Mapping technique. We used pike populations as an example case because it is a valuable species in the study region in high demand by anglers 21 . To standardize the collection of FCMs for this study, all participants received the same set of ecological concepts, which represented key factors affecting pike population dynamics. These factors were derived from independent focus groups with anglers and mental model pre - tests with both anglers and experts to identify key concepts relevant to the pike fishery. We also completed a thorough review of the pike literature to identify key aspects of their life history and what determines population dynamics (e.g., macrophyte abundance) 50 . We added human - centered concepts represented ang ling impacts (e.g., fishing pressure) to outline a social - ecological, rather than merely an ecological, system. The task was to arrange the 49 concepts and draw connections between them based on their own understanding and knowledge ors of importance to the pike population biology and their relationships freedom to add additional concepts (participants received blank cards to be able to outline concepts not mentioned so far) and instructed that not all concepts had to be used in their model ( see Appendix , Table 2.S 1 for a complete list of concepts). The fi nal drawings were photographed for further analyses ( Appendix , Figure 2.S1). I t is worth no ting that t he mental models were obtained at the beginning of the workshop s before any influence could have happened by the team of researchers and workshop organizers and before any other type of information was exchanged with the stakeholders. The visualizations that result from FCM modeling (see Appendix , Fig. 2.S 1) are similar to so - called causal maps, which can be structurally explored in terms of network characteristics. Furthermore, FCM models are also quantitative simulation models that ca n be used to assess the dynamic behavior of the system under study. FCM computation shows the changes in the state of 51 : when one concept increases (or decreases) t his triggers a cascade of changes to other system elements until the system converges to a so - 52 questions, such as how an increase in one concept (e.g. , angling pressure) a ects all other elements in th e system 52 . In a nutshell, FCMs are directed graphs, and therefore, using graph theory, the y can be analyzed structurally to represent system knowledge regarding the elements and connections of the system. Also, to represent how the system behaves in res ponse to input changes, FCM can be analyzed dynamically (i.e., functionally), based on fuzzy causal algebra fo r simulating causal 50 propagation 33 ( see Methods, Dynamic analysis and Inferences ). Moreover, FCM from different participants can be mathematically aggregated if their matrices are brought to the same size and thus include information about every system element that is mentioned in any of the contributing maps. 2.5.3 Mental model a ggregation Individual FCMs can be aggregated mathematically to create a mode l that represents the insights of all study participants and thus provide a tool for testing WOC. There are two aggregation methods used in this study to build the crowd model: (a) Single - Level aggregation ; a ggregation is obtained in one step : where is the adjacency matrix used to represent the FCM of participant p , N is the total number of participants, and indicates the element of this matrix with the value equals to the weight of the edge between node i and j . represents the crowd FCM with the corresponding adjacency matrix . A nd (b) Multi - level aggregation ; a ggregatio n is obtained in two steps: S tep (1) is c omputin g the mean FCM of each subgroup: Where represents the aggregated FCM of sub group G and indicates the element o f adjacency matrix with the value equals to the weight of the edge between node i and j . 51 A nd s tep ( 2) is averaging subgroup means. We can use the arithmetic mean of subgroup means to average them; h owever, forming sub groups which consist of individuals with the same role in the fishery club carries the risk of amplifying stakeholder - specific biases in each sub group and can be expected to increase the skewness of subgroup models distribution . Biases are likely to exist under standing of fishery management 53,54 , which is the largest group in our dataset. Most importantly, to further remove the effect of biases, to form collective solutions, rather than using the arithmetic mean of sub group means, we propose to aggregate sub group means using the median. Earlier studies, in which the crowd is asked to provide single variable estimates, and in which there are significant biases in individual jud gments, show that the median outperforms the arithmetic mean 12,29,30 . Thus we use d the median to combine group means in the second level of the aggregation . Additionally, to remove the effect of subgroup biases, w e can also use weighted - mean and geometric mean in the second level of aggregation based on prior theoretical and empirical studies 27,55 57 . We measured the performance of the crowd model built by d ifferent averaging methods in the second level of aggregation , and our result showed that the median had the best performance amongst other aggregation methods (see Appendix , Table 2 .S2 ). 2.5.4 FCM a nalyses FCM c oncepts (nodes) represent the qualitative characteristics of the system with an absolute value between 0 and 1, characterizing their so - 52 . Arrows (edges) are characterized by a number in the interval of [ - 1, +1], corresponding to the 52 strength, direction, and sign of causal relationships between concepts. The steady state that an FCM reaches in response to an input change (i.e., a forced change in the activation of one or more of its concepts), depends on how the activated concept(s) is connected to other conc epts in the system. How nodes and edges are arranged is thus of great importance and frequently used to centrality shows the contribution of this concept in a cogniti ve map which is determined by accumulating the strength of causal relationships linking this node to the other nodes 52 . One individual considers concepts with higher centrality more important since they are more strongly linked to the other system element s and consequently play more important roles in the dynamic of the system. Comparing the centrality of particular set s of concepts in different cognitive maps translates the differences in the system definition and its important components. In this study, we used Katz centrality index 58 , since it is expected to provide the most appropriate centrality measurement for comparing aggregated maps with higher density and presumably higher abundance of feedbacks 59 . 2.5.4.1 Structural a nalysis In this study, we compared the structure of FCMs using three approaches: The first approach is to compare the centrality of six concepts of central relevance to fishery management Each centrality profile displays the Katz centrality of these six concepts in a radar chart (Figure 2.1) . We calculate the Katz centrality of each node i with: 53 where is the Katz centrality of node i , is the adjacency matrix of FCM, is the Attenuation factor, and is the extra weight attributed to the immedia te neighborhood. What Katz centrality measures is the relative influence of a node within the FCM by taking into account the weight of the immediate neighbors and also all other nodes in the FCM that connect to the node through these immediate neighbors. E xtra weight would be given to the nodes located in the immediate neighborhood through parameter (in our case ). Connections made with distant neighbors are penalized by the attenuation factor (in our case ). The Katz centrality of each no de is a function of the Katz centrality of other nodes. Thus, this centrality computation is an iterative process (in our case maximum number of iterations is , and the error tolerance used to check convergence is ). The second approach to analyzing and comparing the structure of FCMs is an investigation of agreement of strong causal patterns. This patterns emerge when we remove weak edges with absolute weights less than 0.33 from aggregated FCMs (Figure 2.2) . The remaining edges illustrate the strong causal patterns used for model description. where is the set of strong edges with in the FCM of group , is the set of strong edges with in the FCM of experts, is the 54 intersection of strong edges in FCM s of the group g and experts (i.e., set of matched edges) , is the union of stron g edges in FCM s of the group g and experts , and is the proportion of matched edges between group g and experts. Furthermore, we can compare the network structure of FCMs with regards to the quantitative aspects of their graph geometric shapes. In this study, we evaluate combinatorial and geometric properties 60,61 . Given two graphs, this index evaluates how similar they are in terms of the important featu res of their structures. Therefore, it provides a comparison between each FCM and the expert FCM regarding their fundamental structure. In fact, eigenvalue similarity index measures the Euclidean distance between two graphs in a new coordinate system where in coordinates represent eigenvalues. In this coordinate system, each graph is determined by a point and the distance between two points demonstrates the structural similarity between these two graphs. The shorter the distance, the more similar the graphs are in terms of the essential components of their structures (Figure 2.3) . To measure eigenvalues similarity index, we first calculate the eigenvalues of Laplacian of adjacency matrices of both FCMs. For each FCM the Laplacian matrix is calculated by. where L is the Laplacian matrix, D is the diagonal matrix, and A is the adjacency matrix. Then, for each Laplacian matrix, we find the smallest k such that the sum of the k largest eigenvalues constitutes at least 90% of the sum of all o f the eigenvalues 60 . If the values of k are different between the two graphs, we use the smaller one. Thus, the eigenvalues similarity index is the sum of the squared differences between the largest k eigenvalues of the group g and 55 experts FCMs. This give s us a number in the range [0, ), where values closer to zero are more similar: where is the eigenvalue graph similarity index, , is the eigenvalue of the Laplacian matrix of experts FCM, , is the eigenvalue of the Laplacian matrix of group g FCM. 2.5.4.2 Dynamic a nalysis and i nferences In addition to network structure, we analyze the dynamic behavior of FCMs. As prior studies sugge sted, the dynamic behavior of FCMs can be assessed through analyzing their 49,62 . To do so, in each scenario, we change the value of one or more concepts (i.e., nodes) in a map and record the alterations of the system state from the 52 . The value of each concept in the s teady state is calculated using: where is the value of concept at iteration step k+1, is the value of concept at iteration step k, is the value of concept at iteration step k, and is the weight of the edge relationship between and . Function values at each step 49 . Our threshold f unction is a sigmoidal function: 56 where is a real positive number (in our case ) which determines the steepness of the function . or multiple concepts and us e (Eq. 2.10 ) to compute the value of other concepts. The scenario results are the differences between when the system is self - administered (i.e., steady state) and when it is bounded by fixed manipulations in the state of some concepts (i.e., scenario) . For each concept the change in its value as a result of running a scenario is: where is the change in the value of concept , is the value of concept in the steady state, and is the value of concept after converging into a new steady state while scenario concepts are clamped on fixed values. Comparing the scenario outcomes in different FCMs gives us a clear p icture of how differently the system dynamic behavior is perceived by different mental models . To compare dynamic behavior of each group mental model with experts (i.e., reference model), w e compute the Euclidean distance between the ir outputs of a scenari o (Figure 2.4) . The mean of these distances in all of the scenarios (i.e., 10,000 random scenarios in our case ) represents the degree of agreement on simulation outcomes and therefore compare their dynamic behavior: 57 where is the dynamic distance between group G and experts, is the result of scenario j in concept in experts map, is the result of scenario j in concept in the group map, and N is the tot al number of scenarios. 2.5.4.3 Normalized e rror and p erformance The normalized dynamic and structure errors are respectively the standardized dynamic and structure distances between the crowd and expert models: where is the normalized structure error, and is the normalized dynamic error. The normalized total error is the mean of normalized dynamic and structur e error: Finally, the Normalized Performance is calculated by subtracting the normalized total error from one: 58 APPENDIX 59 APPENDIX SUPPLEMENTARY INFORM ATION S1 Supplementary Methods S1.1 Alternative multi - level a ggregation methods In Multi - level method the aggregated model is obtained in two steps: (1) Computing the mean FCM of each subgroup, where is the adjacency matrix used to represent the FCM of participant p , indicates the element of this matrix with the value equals to the weight of the edge between node i and j , and G is the set of individuals in the subgroup. And (2) c ombining subgroup means : The simplest way to combine subgroup means is to use the arithmetic mean, which is called Multilevel Mean - Mean S1.1.1 Multilevel Mean - Mean This method uses the arithmetic mean of subgroup means to aggregate the maps. Most importantly, to further remove the effect of biases, to form collective solutions, rather than using the arithmetic mean of sub group means, we can aggregate sub group means by alternative aggregation techniques in the second level. 60 S1.1.2 Multilevel Mean - W Mean Firstly, w e can weigh sub - approach builds on the works of Mannes et al. 55 and Budes cu and Chen 56 . However , we were not able to identify justifiably reliable criterion to calculate contribution weights, instead , and based on Kao and Couzin 26 suggestion, we simply weighted the sub groups by the reverse order of their proportional size. It uses the weighted mean of subgroup means to aggregate the maps. S.1.1.3 Multilevel Mean - Geo Mean: Secondly, t o account for the fact that estimates of edge weights are not necessarily - Lorenz et al. 27 and van Dolder & van den Assem 57 . This can be expected to perform better than the arithmetic mean when the data is right skewed because most people estimate small causal strengths and a few estimate very strong effects. It uses the geometric mean of sub group means to aggregate the maps. where N is the total number of participants, and indicates the element of this matrix with the value equals to the weight of the edge between node i and j . represents the aggregated FCM of group G . represents the crowd FCM. is the number of different subgroups 61 and is the contribution weight of group G used in weighted mean calculation (in our case ). The re is no quantitative criterion to compute contribution weights, and the weights are qualitatively chosen in the opposite order of proportional group sizes. We measured the performance of the crowd model built by different aggregation methods, and our resu lt showed that the Multilevel Mean - Med had the best performance amongst other aggregation methods (see Appendix , Table 2.S2 ). S1.2 demographics and education The three stakeholder groups (anglers, club managers and fisheries managers) as well as model exercises. The educational variables that were measured included three levels of formation: (1) school - level education (secondary education) , (2) work - related education (tertiary education), and (3) , in Germany, anglers are legally obliged to take a 30 - hour training course in principles of aquatic ecology, legal condit ions and how to treat fish from a welfare perspective (fisheries related education and training). Fisheries managers elected from angling clubs are further obliged to receive specific training in principles of fisheries management, usually offered by angle r associations and assisted by fisheries agencies in each of the 16 German states. In addition, anglers, managers and club heads can also self - teach themselves in ecological principles. We therefore specifically assessed the degree of ecology - related train ing outside formal educational formation through schools or professional training for the job market. In addition to assessing education at three levels, in the three angler groups (anglers, club managers, fisheries managers), we also measured the self - rat ed knowledge over a range of fish 62 ecological and fisheries management topics to measure self - perception of ecological knowledge. - perception of their knowledge about a quatic ecology , f ish stocking in general , Pike ( Esox luc ius ) stocking practices , Carp ( Cyprinus carpio ) stocking practices , and m easures for the sustainable care and management of water - bodies . To describe the differences among the four surveyed groups, we conducted statistical tests using ANOVA on metrical var iables and Chi² - test for distributional variables. Importantly, in addition to examining mean values we were interested in the within group heterogeneity of variables to index the degree of diversity present in each of the four groups we surveyed. S1.2.1 D emographics The four groups did not statistically differ in average age, and all four samples were heavily biased towards males (>94% of all surveyed people, which is the default in the study population). However, there was substantial more within group va riation in age in the angler sample (as indexed by SD), and age variation was also higher in club managers relative to the more homogenous groups of fisheries managers and fisheries scientists. S1.2.2 Education The angler group revealed the largest hetero geneity in the distribution of the highest school education degree compared to the other three groups. On the other extreme, the scientists were the most homogenous sample with over 94% of the respondents holding a university - entrance qualification (Abitur ) the highest school degree possible in Germany. Fisheries managers were more homogenous compared to club managers in terms of school education. A similar pattern was revealed in terms of the distribution of the highest degree of professional training. W hile the fisheries scientists were most homogenous (predominantly having either a 63 university degree or a PhD), the anglers exhibiting the most spread in tertiary degrees compared to club managers or fisheries managers. Finally, in terms of specific officia l or voluntary education in natural scientific and ecological topics related to aquatic systems and fisheries, fisheries managers showed the largest and most homogenous degree of ecological training, followed by club managers and then the angler group. For example, while only 7% of anglers had completed a one - week fisheries management training course, 18% of club managers and 85% of fisheries managers that responded to our survey completed this training. Similarly, 21% of anglers regularly attended public s eminars on fish biology topics, while 32% of club managers and 64% of fisheries managers acknowledged such training. Overall, the degree of ecological training in fish ecology question thus was most homogenous and more pronounced in fisheries managers, fol lowed by club managers and lastly anglers. S1.2.3 Self - rated ecologica l knowledge on fisheries topics Following the training in fish biological topics, the mean self - rating index of ecological knowledge was significantly highest among fisheries scientists, followed by fisheries managers, club managers and anglers. Importantly, however, the variation in self - rated knowledge (as indexed by SD) was higher in the angler group than in the fisheries manager group. Interestingly, also the scientists sho wed quite high variation in the self - rated knowledge, most likely because the self - rated knowledge with very specific domains (such as stocking the species of carp or pike) was assessed, which is unlikely to be something fisheries scientists regularly enga ge with in their practical world. Overall, a picture emerged that the angler group was the most heterogeneous of all three stakeholder groups and the fisheries manager group was the most homogenous in relation to education, with club managers ranging in be tween. On the other extreme , fisheries scientists overall wer e mainly academically trained ( Appendix , Table 2.S 3). 64 S2 Supplementary Discussion : S2.1 Distinct impact of different aggregation methods on WOC The distinct impact of different aggregation methods on WOC can be explained with biases resulting from real - life social influences. Despite potentials for learning from others and thus developing more integrated understanding of the system, in our study, participants were socially influenced only th rough their real - life interactions, where we had no control on the social network structure to avoid accumulation of biases. Becker et al. 31 theoretically and experimentally demonstrated that under certain conditions, and in highly decentralized networks, social influence may produce learning dynamics that potentially improve s the WOC; however, we cannot assert that real - life social network structure is decentralized. Thus, we hypothesized that social processes may undermine the WOC effect, and the negativ e impact can be aggravated in larger crowds where the accumulation of biases magnifies the skewness of the distribution of individual data points . In a crowd built by single - level aggregation, as we draw and average more mental models, the group performanc e initially approaches its optimal point owing to the benefits of information pooling, but drawing more mental models undermines the crowd performance because knowledge that is shared by many members of the group (i.e., commonly agreed upon knowledge that relates to easily observable system elements) is downplaying the specialized mental models that deviate from this common knowledge (see Appendix, Figure 2.S 3). By contrast, we show that in a crowd built by multi - level aggregation, the group performance imp roves monotonically as more mental models are drawn and averaged because the specialized knowledge is not downplayed in favor of commonly agreed upon knowledge (see Appendix , Figure 2.S 3 ) . The multilevel process first acknowledges group - specific knowledge using the mean (i.e., it reduces within - group variability). Then, in the second level of 65 aggregation, it reintroduces some levels of variability by aggregating maps across different stakeholder groups ; however, it uses median which reduces the negative inf luence of group - stakeholder groups agreed upon). S2.2 Fostering institutional fit and nested governance Our WOC approach has potential implications for operationaliz ing local institutions to fit the complex social and ecological aspects of large - scale coupled human - natural systems by suggesting a novel structure for a nested governance system. This nested governance system integrates actions at local levels (i.e., agg regating local knowledge of stakeholders to create role - based subgroups) and coordinates decisions at higher levels with larger scales management authorities (i.e., aggregating across sub - groups). This application can be supported by findings of the recent works of Bodin and Nohrstedt 63 on collaborative management networks and McGlashan et al. 64 , demonstrating how actions in complex system components could be directly related to how a multitude of actors collaborate to collectively represent a complex system by identifying parts of the system on which they can intervene. 66 S3 Supplementar y Figures Figure 2.S1. An example of pike ecology and management fuzzy cognitive map generated by one participant in the workshops: The individually collected mental models graphically display the perceived cause - and - effect relationships of ecological an d social concepts affecting each other (e.g., how habitat quality affects juvenile pike that later grow into harvestable size, or how fish - eating birds, stocking, or angling pressure affect the pike population). Note that the 67 Figure 2.S2. The aggregated fuzzy cognitive maps of pike ecology and management in different groups of stakeholders : ( a ) Map generated ( b ) Map generated by ag ( c ) Map generated by aggregating fisher models, ( d ) Map generated by aggregating all 218 individual models using multi - level aggregation method, and ( e reference model. Red arrows represent negative r elationships, and blue arrows represent positive relationships between concepts. Weak relationships with a weight less than 0.33 were removed from the maps for a more clear illustration. 68 Figure 2.S3. Distribution of group emphasis on different concepts a nd its variation with group size : model. Given each specific group size , we randomly sampled 100 groups by drawing individual mental models. Each chart has 19 box p lots (one for each concept in the model), each shows the distribution of degree centrality of a concept in 100 random samples. Degree centrality represents the perceived importance of the concept, based on its connections to other concepts 69 (i.e., the numbe r of inward and outward facing arrows). The x - axis shows the 19 concepts coded from 0 to 18 (see Appendix , Table 2.S 1 for the name of the concepts). The y - axis indicates the deg ree centrality of each concept. As group size increases, initially high and lo w - centrality concepts move further apart in the bottom row (single - level aggregation), but much less so in the top row (multi - level aggregation). In the case of single - level aggregation, initially highly central concepts become relatively much more emphasi zed than other concepts, thus crowding out more specialized knowledge. These emphasized concepts are easily - observable and more correlated knowledge, such as node 3 = baitfish/prey fish, node 7 = spawning, node 9 = riparian plants like reeds, node 11 = zoo plankton, node 17 = hiding places and refuges, and node 18 = surface area of a body of water. These concepts (i.e. , nodes) can be expected to be part of all or most contributing models. However, their centrality is outsized in comparison to other concepts as group size increases. By contrast, in multi - level aggregation, the larger groups do not intensively overemphasize common knowledge shared by the majority, nor do they underemphasize specialized, and yet important, concepts. In this case, as crowd size i ncreases, it quickly achieves a stable, and yet approximately unbiased, distribution of emphasis on different concepts, and this stable pattern does not significantly cha nge in relatively larger sizes. 70 S4 Supplementary Tables Table 2.S1. The list of all concepts used to build fuzzy cognitive maps. The list of factors were derived from independent focus groups with anglers and mental model pre - tests with both anglers and experts, to identify key concepts relevant to the pike fishery. Participants were give n the freedom to add additional concepts and the final list of all identified concepts was 19 concepts coded from node 0 to node 18 in all fuzzy cognitive maps . Node number 0 pike population (adult, over the legal size limit) 1 stocked pike (adult, over the legal size limit) 2 stocked pike, young fish (under the legal size limit) 3 baitfish, prey fish 4 other predatory fish 5 Algae 6 depth of a body of water 7 spawning grounds 8 wild pike, young fish (under the legal size limit) 9 emergent riparian plants (e.g. , reeds and other bank vegetation) 10 Benthic invertebrates (snails, crustaceans etc.) 11 zooplankton 12 submerged aquatic plants 13 cormorant 14 plant nutrients 15 turbidity of water 16 angling pressure 17 hiding places, refuges 18 surface area of a body of water 71 Table 2.S2. The performance of the crowd model generated by different aggregation methods. The last column shows the overall performance of the crowd models generated by different aggregation methods. The overall performance is calculated by subtracting the normalized total error from one. The normalized total error itself is the mean of normali zed dynamic and structure errors. The normalized dynamic and structure errors are respectively the standardized Euclidian dynamic and structure distances between the crowd and expert models as described in Methods section. Therefore, the normalized perform ance serves as an interpretive criterion to rank the accuracy of aggregated models in approximating the structure and dynamic behavior of the scientific expert model. Aggregation method Method details Normalized Structure Error Normalized Dynamic Error Normalized Total Error Overall Performance Single - level The arithmetic mean of all individuals 0.337 0.94 0.64 0.36 Multilevel Mean - Mean The arithmetic mean of subgroup means 0.164 0.721 0.44 0.56 Multilevel Mean - W Mean The weighted mean of subgroup means 0.124 0.687 0.41 0.59 Multilevel Mean - Geo Mean The geometric mean of subgroup means 0.093 0.667 0.38 0.62 Multilevel Mean - Med The median of subgroup means 0.084 0.623 0.35 0.65 72 Table 2.S3. Fisheries knowledge and education metrics assessed using questionnaires after the mental model exercises. of 0.84. Statistical test on mean differences is based on ANOVA, with Post - hoc test Tukey B for homogenous variances, and Dunnett - T - 3 for heterogeneous variances. Anglers Club Managers Fishery Managers Scientists Age (years) 48.5(14.13) 47.8(13.28) 46.3(11.6) 40(9.8) Gender 98% Male 2 % Female 96% Male 4% Female 97% Male 3% Female 94% Male 6% Female Self - rated Ecological Knowledge Mean(SD) N=135 Mean(SD) N=52 Mean(SD) N=31 Mean(SD) N=17 Ecological Knowledge index 13.6(3.2)a*** 15.2(3.0)ab*** 17.7(2.1)b*** 19.7(4.2)c*** Aquatic ecology 2.9(0.8)a*** 3.1(0.7)b*** 3.7(0.5)c*** 4.2(0.7)d*** Fish stocking in general 2.7(0.7)a** 3.2(0.7)b** 3.7(0.5)c*** 4.35(0.6)d*** Pike ( Esox lucius ) stocking practices 2.5(0.8)a*** 2.8(0.8)b** 3.2(0.7)c*** 3.9(1.1)d*** Carp ( Cyprinus carpio ) stocking practices 2.6(0.8)a*** 2.9(0.8)b** 3.4(0.5)c** 3.3(1.4)c** management of water - bodies 3.0(0.7)a*** 3.3(0.9)b** 3.8(0.6)c** 4.0(0.9)c** Fisheries Related Education and Training Educational course in preparation for the state angling exam 89% 94% 85% NA Training as a fisheries manager 7%a*** 18%b*** 85%c*** NA University degree in biology or ecology 0% 0% 5% NA Attendance of fisheries biology presentations, or other natural science presentations 21%a*** 32%b*** 64%*c*** NA Self - education by means of technical literature (books, magazines, internet) 50% 69% 100% NA Secondary Education Certificate of secondary education 24.8%a*** 23.8%a*** 37.5%*** 0.0%b*** General Certificate of secondary education 24.8%a*** 23.8%a*** 18.8%a*** 0.0%b*** Advanced technical college certificate 10%a** 16.3%a** 3.1%b** 5.9%b** University entrance qualification (Abitur) 7.8%a*** 2.5%a*** 6.3%a*** 94.1%b*** Did not complete school degree 0.7% 0% 0% 0.0% Tertiary Education Apprenticeship 33.3% 35.0% 37.5% 0.0% University degree of technical degree of higher education 7.8%a*** 6.3%a*** 3.1%a*** 41.2%b*** No tertiary education 0.7% 0.0% 0.0% 0.0% Master Craftsman 10.6%a*** 15.0%*** 9.4%*** 0.0%b*** Technician 6.4%a*** 8.8%a*** 12.5%a*** 0.0%b*** PhD 0.0%a*** 0.0%a*** 3.1%a*** 58.8%b*** Still studying 1.4% 1.3% 0.0% 0.0% 73 REFERENCES 74 REFERENCES 1. Liu, J. et al. Complexity of coupled human and natural systems. Science vol. 317 1513 1516 (2007). 2. Levin, S. et al. Social - ecological systems as complex adaptive systems: Modeling and policy implications. Environ. Dev. Econ. 18 , 111 132 (2013). 3. Johannes, R. E. The case for data - less marine resource management: Examples from tropical nearshore finfisheries. Trends Ec ol. Evol. 13 , 243 246 (1998). 4. Arlinghaus, R. & Krause, J. Wisdom of the crowd and natural resource management. Trends Ecol. Evol. 28 , 8 11 (2013). 5. Fisheries 27 , 6 17 (2002). 6. Walters , C. Adaptive Management of Revewable Resources. (Macmillan Publishing Company, 1986). 7. Arlinghaus, R., Tillner, R. & Bork, M. Explaining participation rates in recreational fishing across industrialised countries. Fish. Manag. Ecol. 22 , 45 55 (2015). 8. Carruthers, T. R. et al. Landscape - scale social and ecological outcomes of dynamic angler and fish behaviours: processes, data, and patterns. Can. J. Fish. Aquat. Sci. 76 , 970 988 (2019). 9. Woolley, A. W., Chabris, C. F., Pentland, A., Hashmi, N. & Malon e, T. W. Evidence for a collective intelligence factor in the performance of human groups. Science (80 - . ). 330 , 686 688 (2010). 10. Irwin, A. Citizen science: A study of people, expertise and sustainable development . (Psychology Press, 1995). 11. Bonney, R. et al. Citizen Science: A Developing Tool for Expanding Science Knowledge and Scientific Literacy. Bioscience 59 , 977 984 (2009). 12. Galton, F. Vox populi. Nature vol. 75 450 451 (1907). 13. Surowiecki, J. The wisdom of crowds: Why the many are smarter than the few and how collective wisdom shapes business. Econ. Soc. Nations 296 , (2004). 14. Yi, S. K. M., Steyvers, M., Lee, M. D. & Dry, M. J. The Wisdom of the Crowd in Combinatorial Problems. Cogn. Sci. 36 , 452 470 (2012). 15. Bernstein, E., Shore, J. & Lazer, D. How intermittent breaks in interaction improve collective intelligence. Proc. Natl. Acad. Sci. 115 , 8734 8739 (2018). 16. Worm, B. et al. Rebuilding Global Fisheries. Science (80 - . ). 325 , 578 585 (2009). 17. Arlinghaus, R. et al. Governing the recreational dimension of global fisheries. Proceedings of the National Academy of Sciences of the United States of America vol. 116 5209 5213 (2019). 75 18. Craik, K. J. W. The nature of explanation . vol. 445 (CUP Archive, 1952). 19. Jetter, A. J. & Kok, K. Fuzzy Cognitive Maps for futures studies A methodological assessment of concepts and methods. Futures 61 , 45 57 (2014). 20. Gray, S., Chan, A., Clark, D. & Jordan, R. Modeling the integration of stakeholder knowledge in social -- ecological decision - making: benefits and limitations to knowledge diversity. Ecol. Modell. 229 , 88 96 (2012). 21. Gray, S., Hilsberg, J., McFall, A. & Arlinghaus, R. The structure and function of angler mental models about fish population ecology: The influence of specializa tion and target species. J. Outdoor Recreat. Tour. 12 , 1 13 (2015). 22. Page, S. E. Making the difference: Applying a logic of diversity. Acad. Manag. Perspect. 21 , 6 20 (2007). 23. Hong, L. & Page, S. E. Groups of diverse problem solvers can outperform gr oups of high - ability problem solvers. Proc. Natl. Acad. Sci. 101 , 16385 16389 (2004). 24. Boudreau, K. J., Lacetera, N. & Lakhani, K. R. Incentives and problem uncertainty in innovation contests: An empirical analysis. Manage. Sci. 57 , 843 863 (2011). 25. Kao, A. B. & Couzin, I. D. Decision accuracy in complex environments is often maximized by small group sizes. Proc. R. Soc. B Biol. Sci. 281 , 20133305 (2014). 26. Kao, A. B. & Couzin, I. D. Modular structure within groups causes information loss but can im prove decision accuracy. Philos. Trans. R. Soc. B Biol. Sci. 374 , 20180378 (2019). 27. Lorenz, J., Rauhut, H., Schweitzer, F. & Helbing, D. How social influence can undermine the wisdom of crowd effect. Proc. Natl. Acad. Sci. 108 , 9020 9025 (2011). 28. Navajas, J., Niella, T., Garbulsky, G., Bahrami, B. & Sigman, M. Aggregated knowledge from a small number of debates outperforms the wisdom of large crowds. Nat. Hum. Behav. 2 , 126 132 (2018). 29. Hooker, R. H. Mean or median [2]. Nature vol. 75 487 488 (1 907). 30. Perry - Coste, F. H. The ballot - box [2]. Nature vol. 75 509 (1907). 31. Becker, J., Brackbill, D. & Centola, D. Network dynamics of social influence in the wisdom of crowds. Proc. Natl. Acad. Sci. 114 , E5070 -- E5076 (2017). 32. Jentoft, S. Fisheries co - organizations. Mar. Policy 13 , 137 154 (1989). 33. Kosko, B. Fuzzy cognitive maps. Int. J. Man. Mach. Stud. 24 , 65 75 (1986). 34. Arlinghaus, R. A Property Rights - Based View on Management of Inland Recreational Fisheries: Contrasting Common and Public Fishing Rights Regimes in Germany and the United States. in American Fisheries Society Symposium vol. 75 13 38 (2011). 35. Hickley, P. & Tompkins, H. Recreational fisheries: social, economic, and management aspects . (Fishing News Books Oxford, UK, 1998). 76 36. Fujitani, M., McFall, A., Randler, C. & Arlinghaus, R. Participatory adaptive management leads to environmental learning outcomes extending beyond the sphere of science. Sci. Adv. 3 , e16025 16 (2017). 37. Gray, S. A., Zanre, E. & Gray, S. R. J. Fuzzy cognitive maps as representations of mental models and group beliefs. in Fuzzy cognitive maps for applied sciences and engineering 29 48 (Springer, 2014). 38. FAO Technical Guidelines for Respons ible Fisheries. No. 13. Recreational Fisheries. http://www.fao.org/3/i2708e/i2708e00.htm (2012). 39. Townsley, P. Social issues in Fisheries. FAO. Fish. Tech. Pap. 93 (1998). 40. Folke, C., Pritchard Jr, L., Berkes, F., Colding, J. & Svedin, U. The problem of fit between ecosystems and institutions: ten years later. Ecol. Soc. 12 , (2007). 41. Ostrom, E. Governing the commons: The evolution of institutions for collective action . (Cambridge university press, 1990). 42. Ostrom, E. A diagnostic approach for goi ng beyond panaceas. Proceedings of the National Academy of Sciences of the United States of America vol. 104 15181 15187 (2007). 43. Worm, B. & Branch, T. A. The future of fish. Trends in Ecology and Evolution vol. 27 594 599 (2012). 44. Arlinghaus, R., Mehner, T. & Cowx, I. G. Reconciling traditional inland fisheries management and sustainability in industrialized countries, with emphasis on Europe. Fish Fish. 3 , 261 316 (2002). 45. Lorenzen, K., Beveridge, M. C. M. & Mangel, M. Cultured fish: Integrative biology and management of domestication and interactions with wild fish. Biological Reviews vol. 87 639 660 (2012). 46. Cooke, S. J., Arlinghaus, R., Johnson, B. M. & Cowx, I. G. Recreational fisheries in inland water s . (Wiley - Blackwell Oxford, UK, 2016). 47. Sandström, A. Navigating a complex policy system - Explaining local divergences in Swedish fish stocking policy. Mar. Policy 35 , 419 425 (2011). 48. Dorow, M., Beardmore, B., Haider, W. & Arlinghaus, R. Using a nove l survey technique conservation programs. Biol. Conserv. 142 , 2973 2982 (2009). 49. Gray, S. A. et al. Using fuzzy cognitive mapping as a participatory approach to analyze c hange, preferred states, and perceived resilience of social - ecological systems. Ecol. Soc. 20 , (2015). 50. Craig, J. F. A short review of pike ecology. Hydrobiologia 601 , 5 16 (2008). 51. Giabbanelli, P. J., Gray, S. A. & Aminpour, P. Combining fuzzy cogni tive maps with agent - based modeling: Frameworks and pitfalls of a powerful hybrid modeling approach to understand human - environment interactions. Environ. Model. Softw. 95 , 320 325 77 (2017). 52. knowledge: a multi - step fuzzy cognitive mapping approach. Ecol. Modell. 176 , 43 64 (2004). 53. Dorow, M. & Arlinghaus, R. The relationship between personal commitment to angling and the opinions and attitudes of German anglers towards the conservation and management of the European eel Anguilla anguilla. North Am. J. Fish. Manag. 32 , 466 479 (2012). 54. Baer, J., Blasel, K. & Diekmann, M. Benefits of repeated stocking with adult, hatchery - reared brown trout, Salmo trutta, to recreational fisheries? Fish. Ma nag. Ecol. 14 , 51 59 (2007). 55. Mannes, A. E., Soll, J. B. & Larrick, R. P. The wisdom of select crowds. J. Pers. Soc. Psychol. 107 , 276 299 (2014). 56. Budescu, D. V. & Chen, E. Identifying expertise to extract the wisdom of crowds. Manage. Sci. 61 , 267 280 (2015). 57. Van Dolder, D. & Van Den Assem, M. J. The wisdom of the inner crowd in three large natural experiments. Nat. Hum. Behav. 2 , 21 26 (2018). 58. Katz, L. A new status index derived from sociometric analysis. Psychometrika 18 , 39 43 (1953). 59. Lavin, E. A., Giabbanelli, P. J., Stefanik, A. T., Gray, S. A. & Arlinghaus, R. Should we simulate mental models to assess whether they agree? in Proceedings of the Annual Simulation Symposium 6 (2018). 60. Koutra, D., Parikh, A., Ramdas, A. & Xiang, J. A lgorithms for graph similarity and subgraph matching. in Proc. Ecol. Inference Conf vol. 17 (2011). 61. Koutra, D., Shah, N., Vogelstein, J. T., Gallagher, B. & Faloutsos, C. DELTACON: Principled massive - graph similarity function with attribution. ACM Tran s. Knowl. Discov. Data 10 , 1 43 (2016). 62. Yoon, B. S. & Jetter, A. J. Comparative analysis for Fuzzy Cognitive Mapping. in PICMET 2016 - Portland International Conference on Management of Engineering and Technology: Technology Management For Social Innov ation, Proceedings 1897 1908 (Institute of Electrical and Electronics Engineers Inc., 2017). doi:10.1109/PICMET.2016.7806755. 63. Bodin, Ö. & Nohrstedt, D. Formation and performance of collaborative disaster management networks: Evidence from a Swedish wil dfire response. Glob. Environ. Chang. 41 , 183 194 (2016). 64. McGlashan, J., de la Haye, K., Wang, P. & Allender, S. collaboration in complex Systems: Multilevel network Analysis for community - Based obesity prevention interventions. Sci. Rep. 9 , 1 10 (2019 ). 78 CHAPTER 3 3 THE DIVERSITY BONUS IN POOLING LOCAL KNO WLEDGE ABOUT COMPLEX PROBLEMS A revised version of t his chapter is in press for publication in Proceedings of the National Academy of Sciences (PNAS) ( https://www.pnas.org/ ) . ABSTRACT Recently, theoreticians have hypothesized that diverse groups , as opposed to groups that are homogeneous , produce several assets all of which lead to better performance in solving complex problems. As such, understanding complex environmental or social issues , for which scientific information is typically limited, would likely benefit from the integration of diverse types of local expertise. Yet, capturing knowledge distributed across diverse types of local experts is not straightforward and consequently rarely evaluated, which often hinders applying knowledge - pooling to sustainability decision - making. To address these challenges, here we show how emerging internet technologies, semi - quantitative cognitive mapping techniqu es, and principles of collective intelligence theory can merge into a novel crowdsourcing approach to aggregate diverse expertise. Using a case of striped bass fisheries in Massachusetts, we show how our approach can be used to pool local knowledge of reso urce stakeholders to produce a model of complex social - ecological interdependenc i es. First, subjective evaluation of stakeholder models revealed improved performance of the diverse group compared to more homogeneous ones, as evidenced by blind reviews cond ucted by an expert panel . Second, objective evaluation of stakeholder models using a stochastic network analysis indicated that a dive rse group more adequately modeled complex interdependencies and feedbacks where homogeneous groups were more likely to fai l. Our work empirically validates the previous 79 theoretical assumption that knowledge diversity and pooling are important for understanding complex problems, while also highlight ing that diversity must be moderated through an aggregation process leading to more complex yet parsimonious models. 3.1 INTRODUCTION Determining which management strategies lead to sustainable outcomes in social - ecological systems (SES) is challenging, and are often made based on the best available sciences 1 . Specifically , in natural r esource management, a number of laws and regulations mandate the use of best scientific information available (BSI A ) in decision - making ( e.g. , the Magnuson - Stevens Fishery Conservation and Management Act of 1976 ; and the 2012 Forest Service Planning Rule ). In many cases , B SI A is comprised of empirical data, peer - reviewed information , expert knowledge, and local and traditional knowledge. This scientific information can be used to implement management strategy evaluation (MSE) 2 a decision - support tool (e.g. , computational models) to examine the implications of altern ative management and policy scenarios before any action is taken. However, in data - poor cases across a wide range of local sustainability contexts, scientific information is frequently inadequate for effective decision - making 3,4 . In such cases, traditional or local knowledge (LK) of people who interact with local ecosy stems can constitute a rich source of scientific information and plays a key role in decision - making 5 8 . However, there are two p rimary challenges associated with pooling LK from local stakeholders that need to be addressed: First, it is often difficult to quantify the scientific or management uncertainties associated with utilizing LK, which limits its formal or even legal use in e nvironmental assessments and decision - making 6,9 1 and thus cannot be easily 80 integrated with scientific assessments, which are often quantitative 6,10,11 . This limitation undermines the potential use of LK to develop statistically rigorous inferences or computationally executable simulations with the ability to represent the true condition of the system and predic t system responses to various management strategies or natural and anthropogenic perturbations. Second, LK held by stakeholders demonstrates considerable variations across different groups that represent biased and sometimes conflicting perceptions of comp lex social - ecological interdependencies. These variations may be linked to differences in preferred adaptation strategies 12,13 ; divergin g beliefs and values 14 ; disparate experiences and interactions with ecosystems 15 9 of integrating LK into management strategies 16 18 . In this article, we draw on collective intelligence (CI) theory 19 to test if LK once aggregated from diverse stakeholders produces accurate and reliable scientific information for complex problem - solving. CI is typically defined as a group phenomenon, enabling a group to accomplish complex tasks where individuals or any subset within it fail 20 . This group phenomenon may emerge when a collective of individuals either collaborate or indepen dently pool their knowledge to address a problem 19,21,22 . The group may therefore benefit f rom a larger, more refined, or recombined body of knowledge, whereas the aggregation mechanism filters out errors and biases , es , or allows for recombining the pool of knowledge in new ways that can result in innovative solutions, which is unlikely that any of the individual members would be able to come up with (e.g., ref. 23 25) . CI has been a growing area of investigation with implications for improving decision - making in different fields. Importantly, new information technologies have substantially increased human capacities to pool knowledge 81 and participate in decision - making 20 . For example, online crowdsourcing technologies such as Human DX and Sermo facilitate medical collaboration by build ing partnerships between medical societies and the public to improve medical training and clinical decision - making 26,27 . Prediction Markets, as another example, leverage internet technologies to harness the CI of online crowds and accurately predict the probabilities of various events occurring 21 . A growing body of literature now suggests that diversity of knowledge, once properly harnessed, is of utmost importance to improve CI in a group, thereby helping a group achieve better performance at the aggregate level 21,23,28 32 . In general, and especially for complex problem - solving such as those related to the sustainability of local SESs, theoretical and empirical evidence has demonstrated that knowledge diversity is a critical driver of collective performance 4,33 35 . But, how can we harness the CI of natural resource stakeholders via pooling LK to model social - ecological i nterdependencies? How does the factor diversity impact the group collective performance in modeling a complex system? Here we explore how emerging internet technologies, semi - quantitative cognitive mapping techniques, and CI theoretical pr inciples can be integrated into a novel crowdsourcing approach to address the challenges associated with using LK as an accurate and reliable source of information to understand local sustainability issues. Our approach results in the aggregation of LK tha t is elicited from diverse groups of natural resource stakeholders through mental modeling. This knowledge elicitation and aggregation mechanism can yield a computationally executable representation (i.e. , model) of social - ecological dynamics that combines local To implement this approach, we used an example of striped bass ( Morone saxatilis ) population dynamics in Massachusetts (MA), U.S.A. The striped bass fishery is a n important component of coastal economies throughout the eas t coast and is composed of 82 comm ercial and recreational fishers . While various stakeholder groups interact differently with the fishery, they each construct diverse knowledge about resource dynamics including both ecological dimensions (e.g., predator - prey relationships) and human dimensions (e.g., commercial and recreational fishing pressures), as well as their interrelationships. We crowdsourced mental models using a semi - quantitative technique called Fuzzy Cognitive Mapping (FCM) 36 to represent each ind - ecological relationships that influence striped bass populations and fisheries. We collected these FCMs using an online mental modeling technology (www.mentalmodeler.com), in the form of digitalized graph drawings from a diverse c rowd of local stakeholders, including recreational fishers, commercial fishers, and fisheries managers ( see Appendix , Fig ures 3.S1 and 3.S 2). The the aggregated kn owledge of stakeholders. These aggregated models can be analyzed in terms of their qualitative compositions (i.e., what concepts are represented), network structure of causal relationships (i.e., how concepts are connected), and dynamic behavior (i.e., how changes in the state of one or multiple concepts initiate a cascade of changes in other concepts) (see Materials and Methods ) . Given the numerous social and ecological concepts potentially influencing the striped bass population and the likely difference Aggregation took place once all individual mental models were transformed into adjacency matrices a mathematical repr esentation of a directed graph 37 . We first combined individual mental models by stakeholder types to form homogeneous, stakeholder - specific models using the arithmetic mean of their adjacency matrices elements 38 (see Materials and Methods ). 83 Subsequently, the more diverse crowd model (including all stakeholder types) was created through aggregating stakeholder - specific models using the median of their adjacency matrices elements - (see Appendix , Figure 3.S3) has been previously identified as the most reliable and effective method for combining the mental models of stakeholders to mediate the diversity of models and cancel out group specific biases 33 . We evaluated the performance of stakeholder - driven models , b oth homogeneous and diverse by ( i ) acquiring subjective judgment s from scientific experts, and ( ii ) analyzing the network structure of their aggregated mental models (see Materials and Methods ). Our findings demonstrate that aggregating knowledge from a diverse group of stakeholders produces a CI model of social - ecological i nterdependenc i es that can generate outcomes similar to scientific methods in anticipating the structure of natural resource systems and their response to management strategies and ext ernal perturbations, and these outcomes in general, outperform those of more homogeneous groups. Our study, therefore, provides tools and methods for synthesizing the knowledge held by diverse groups of local stakeholders, which can advance the formal use of LK in understanding and making decisions about integrated environmental, health, and social issues and can potentially enhance our ability to resolve local sustainability problems. 3.2 RESULTS A total of 32 individuals completed the online mental modeling s urvey including 13 recreational fishers, 11 commercial fishers, and 8 fisheries managers. To allow for standardization , but also capture knowledge diversity, participants were asked to include 5 concepts in their model s (recreational fishing, commercial fi shing, striped bass population, prey abundance, and water temperature) while other components in their concept maps could be freely associat ed based on their perceptions (s ee Appendix , Table 3. S1). 84 Aggregation of individual mental models by stakeholder ty pes resulted in three averaged FCMs representing the overall perception of each homogeneous group ( see Appendix , Figure 3. S 4 - 3.S6) . Group aggregated FCMs varied widely in the number of nodes and connections, as well as the qualitative composition of concep ts used to represent social - ecological relationships. For example, recreational fishers focused on social concepts influencing fish population s , while commercial fishers tended to incorporate biological concepts, and managers emphasiz ed management aspects (s ee Appendix , Figure S 8 ). Aggregation across diverse stakeholder groups using the median of group means yielded connections) (see Appendix , Figure 3. S 7 of all three stakeholder - specific groups by preserving a moderated level of information presented from each of them, blended knowledge diversity, and represented the overall understanding of the whole community abou t striped bass dynamics. The review of concepts in three stakeholder - specific models and the diverse crowd model revealed that there were 15 overlapping core concepts shared by all 4 models; however, these concepts were conne cted to each other differently in different models (Figure 3.1 ). Expert evaluations of model structure were conducted by a panel of fisheries scientist s based on the patterns of causal relationships among those 15 concepts to examine and compare the performance of four models. This eval uation was cond ucted in five steps: examining 1) striped bas s predator - prey relationships, 2) the effect of fishi ng pressures on striped bass, 3) striped bass conn ection to ecology and habitat, 4) social drivers affecting th e striped bass population, and 5) environmental drivers affecting the striped bass population. 85 Figure 3. 1. The representation of cause - and - effect relationships between 15 overlapping concepts shared by all stakeholder groups and the diverse crowd. Ecological components are green and social components are purple. The aggregated graphs of ( a ) commercial fishers, ( b ) recreational fishers, ( c ) fisheries managers, and ( d ) the diverse crowd were evaluated by scientific experts to assess their accuracy in terms of causal relationships and fe edback loops. Evaluations were con ducted in 5 steps as shown in ( d ). In addition, the four aggregated models were computationally manipulated to determine how perceived social - ecological dynamics vary for each model (see Appendix ). We computed 86 prediction of changes under six scenarios: increased inclement weather for fishing, increased water temperature, decreased water quality, increased price of fish, increased demand, and increased poachin g and illegal activities (Figure 3.2 ). Figure 3. 2. Fishery response (i.e., relative normalized change s in value of concepts) to six different scenarios simulating ( a ) increased inclement weather for fishing, ( b ) increased water temperature, ( c ) decreased water quality, ( d ) increased price of fish, ( e ) incr eased demand, and ( f ) increased poaching and illegal activities. Expert evaluations of model functionality were conducted to examine and compare 87 dynamic behavior revealed that the diverse crowd model outperformed stakeholder - specific models of homogeneous groups (i.e., recreational fishers, commercial fishers, and fisheries managers). Experts, on average, rated the crowd model as the most accurate map among the four models because it most adequately represented the causal relationships and feedback loops in striped bass fishery SES. Overall, scientific experts assessed that the structural performance of the crowd model was 65% accurate, followed by 55% accuracy for fisheries managers, 48% for recreational fishers, and 43 % for commercial fishers (Figure 3.3 experts rated the crowd model as the most accurate map among four blinded models owing to second in this category with 50% accuracy. The models of commercial fishers and recreational fishe rs were assigned 39% accuracy by experts according to the mod el's dynamic performance (Figure 3.3 b). This implies that aggregating diverse knowledge of different stakeholder groups may improve the overall accuracy of their combined LK and can potentially lead to higher scientific alignment (Figure 3.3 c) . 88 Figure 3. 3. Expert evaluation of aggregated models. The box plots represent the distribution of a ) structure and ( b ) dynamic performance. The performance was measured using a 7 point Likert scale for eac h item of interview sheets (see Appendix ). The assigned accuracies of structural items (i.e., five sub - structures illustrated in Figure 3. 1 ) were averaged and normalized to a scale between 0 and 1. Similarly, th e assigned accuracies of dynamic items (six scenarios illustrated in Figure 3. 2 ) were averaged and normalized to a scale between 0 and 1. The 2D scatter plot in ( c ) shows the overall score given to four models by each expert, where x - axis is the accuracy r - axis is the accur In addition to 15 overlapping concepts that appeared in all four aggregated models, we asked experts to examine the other concepts that did not appear in all 4 models. Expert 89 positives (i.e., not including necessary system components and including unnecessary ones , respectively c knowledge of the considered superfluous in modeling the striped bass population. This qualitative assessment of egated models revealing that, on negative, which is the smallest among all other stakeholder - specific models with overall false errors ranging from 32% to 55% ( see Appendix , Figure 3. S9 ). The stochastic network analysis of model structures revealed the prevalence of complex motifs (i.e., bi - directionality, indirect effect, multiple effects, and feedback loop micro - structures) in a model (see ref. 39) . We quantified the expected value of counts for complex motifs given the size and density of networks of each group FCM (see Materials and Methods ). Deviations of motif counts from their expected value were used as measu prevalence (Figure 3 . 4). Our results demonstrate that the FCM of the diverse crowd has a higher prevalence for all complex motifs compared to the expectation, thereby representing a higher perception of complex causality. 90 Figure 3. 4 . Deviation of the prevalence of complex c ausal motifs in aggregated models relative to uniform random graphs for ( a ) bi - directionality, ( b ) indirect effect, ( c ) multiple effects, and ( d ) feedback loops. Black dots represent 10,000 random graphs and the blue line shows the expected value of motif counts. Red dashes represent the deviation of eac h model from the expected value. 3.3 DISCUSSION This study advances the use of LK for understanding complex problems by leveraging the assets of knowledge diversity (i.e. , 29 . We draw on CI th eoretical principles interrelationships in social and environmental problems t hat human societies face . Our study 91 offers a novel approach to collect and aggregate knowledge div ersity across various groups of individuals to provide an improved understanding of complex problems. We used an example of striped bass fisheries in MA to empirically test our approach. Our results demonstrate that once harne ssed properly, pooling diverse LK would likely yield accurate representations of complex interdependencies between humans and the environment that govern a natural resource sy stem. The resulting aggregated model can also generate reliable and accurate predictions of system responses to natural and ant hropogenic perturbations (Figure 3.3 ). This study, therefore, adds to a growing body of literature that investigates the use of LK in scientific assessments and the management of natural resources 8,10,11,40,41 . Stakeholders interact with natural resources differently , and this may lead them to construct diverse perceptions about social - ecological interdependencies. The semi - quantitative FCM approach we used here enables us to highlight these variations in stakeho of perceived system composition, structure of interdependencies, and dynamic behavior 42,43 . Even though a large body of literature has been dedicated to measuring these variations using FCMs (e.g. , ref. 44) , few studies have explored th e benefits of knowledge aggregation 33,38,45,46 . Here we focused on how diversity impacts stakeholders' collective perception of a complex problem. Consistent with past theoretical studies (e.g. , 28 , we found that the ag gregation of LK obtained from a diverse group of stakeholders produces a system representation that outperforms those of homogeneous groups. However, to be successful, the aggregation needs to be mediated, filtering out the biases associated with each grou 29 . As such, the 92 aggregation method we used here is based on a tw o - step averaging mechanism (F igure 3.S3 ) to let stakeholder - specific biases cancel each other out and also knowledge insufficiencies be complemented by pooling diverse expertise (see ref. 33 for more details about the aggregation method). In this study, w e used both subjective and obj ective evaluations t o measure the performance of aggregated FCMs. We asked a group of experts with a wide range of scientific knowledge and expertise to assess the performance of the stakeholder - driven models based on their personal opinions and knowledge. Even though experts represented a wide range of academic disciplines (e.g., fisheries ecology, economics, etc.) and professional expertise, a clear majority of experts rated the diverse crowd model as the most accurate one, compared to stakeholder - specifi c , homogeneous models. Additionally, stochastic network analysis provided an objective evaluation of the model ' s performance in represent ing complex causalities driving the system . The FCM of diverse crowd demonstrated a higher prevalence for all complex m otifs compared to the expectation. The aggregated map of recrea tional fishers also demonstrated high prevalence for all tested motifs while managers demonstrate d low prevalence of motif indirect effects , which represents a lower appreciation of cascading impacts 45 . The aggregated map of commercial fishers, on the other hand, demonstrated low prevalence for all tested complex motifs indicating that commercial fishers tend to perceive the system as more linear with hierarchal casual struct ures (Figure 3.4) . Additionally, managing uncertainty is a key challenge for policy and decision - making . In natural resources management, t wo common types of uncertainties include scientific uncertainty (related to data sources) and management uncertainty (related to the ability to predi ct management success/outcomes). To address the former, laws and regulations frequently mandate 93 the use of BSIA (e.g. , Magnuson - Stevens Fishery Conservation and Management Act of 1976 ). However, the interpretation of such laws has led natur al resource management to overvalue minimizing scientific uncertainty, often to the detriment of properly handling management uncertainty. Similarly, researchers have suggested that managing the latter requires that stakeholders, managers, and scientists f irst predict how systems respond to management strategies through scenario analysis and then collectively achieve a shared understanding about possible management actions 47 . Such adaptive co - management practices, however, often suffer from a lack of ready - to - use simulation models, in addition to the high amount of time and resources necessary to elicit diverse knowledge. Here we demonstrate that the use of internet technology to crowdsource LK through an online mental modeling platform can help achieve eff orts to manage both types of uncertainties . Even though these mental modeling practices are commonly organized through workshops (e.g., ref. 43 ) and interviews (e.g., ref. 44 ) , we demonstrated that once provided with simple instructions (e.g., short video s and written directions) (see Appendix ) , participants can comfortably interact and familiarize themselves with the online platform, and thus knowledge elicitation process can be automated. Our study demonstrates a novel CI approach for aggregating and int modeling systems for understanding a nd promoting the sustainability of complex social - ecological systems. 3.4 METHODS 3.4.1 Mental models and fuzzy cognitive maps In this study, we used FCM s t 94 many researchers have suggested the importance of eliciting and measuring mental models 18,38,44,48,49 . However, many mental model elicitation techniques often yield qualitative representations of associative rules between concepts/ideas and logical chains of reasoning, with few standardized methods to analyze them as computational simulations of the syste m they represent 38 . Here we used FCM a semi - quantitative technique to bridge the divide between highly computational system modeling and easy - to - construct qualitative cognitive or concept ception showing a network of cause and effect relationships (edges) among different concepts (nodes) and, yet, can be computationally manipulated due to the numerical parametrization of the strength of causal relationships. These models are therefore simul knowledge about dynamics of the system they represent 37 . By increasing or decreasing a - simulated using the a uto - associative neural network method 50 (see Appendix ) . 3.4.2 Online crowdsourcing implementation We collected mental models from diverse groups of stakeholders including commercial fishers, recreational fishers, and fisheries managers. We used a contact list of recreational and commercial fishers including all MA licensed fishers. In addition, we used a contact list of fisheries managers including individuals from NOAA, Massachusetts Division of Marine Fisheries, and Atlantic States Marine Fisheries Commission - striped bass board. Random sampling methods were used to select 100 individuals from each l ist. Individuals who indicated their willingness to participate were received instructions through email. Each individual participated independently in an online mental modeling survey, where they used an online 95 mental modeling technology ( www.mentalmodele r.org ) to make an FCM about striped bass population dynamics and social - ecological factors that impact fish population and fishery . Participants were given a written step - by - step manual (see Appendix ) and a series of short videos instructing them how to br ainstorm, identify, and add components via an online graphical interface, representing all concepts that they believe impact either their fishing effort and/or the striped bass population. Participants were then asked to use this modeling technology to dra w lines between concepts and assign a relative value between 0 and 1 (either positive or negative) to each link based upon the degree to which one component affects another. This exercise was completed when the participant could no longer think of addition al relevant concepts or linkages among concepts. Participants had to save their mental model contributions and send them to the 3.4.3 Collective intelligence and knowledge pooling To harness the CI of local stakeholders for natural resou rce system modeling we expanded a well - documented method 21,51 . WOC refers to the finding that groups of people, under certain conditions, are collectively smarter than individuals in problem - solving, decision making, in novating, and predicting. For example, in simple estimation, the average of individual judgments often outperforms the judgment of the majority of the contributing individuals and sometimes even the best individual judge 21 . WOC has been applied to many si tuations from people contributing to medical diagnostics 52 to predicting the winners of major sporting events 53 , often with high rates of success. A theoretical explanation for this phenomenon is that there is an error associated with each individual jud gment, and taking the average over a large number of responses filters out the noise of gross over - and under - estimates, thus moving the aggregate response closer to the ground truth 21,54 . 96 We used WOC principles to aggregate mental models of stakeholders about the striped bass SES . According to Surowiecki 21 , crowd - based s olutions, can be reliable when (a ) the study participan ts represent diverse opinions, (b ) make their judgments independent of each other and ) are able to dr aw on their local knowledge, and (d ) there exist some aggregation mechanisms to combine individual contributions into a collective response. stakeholders whose LK and perc eptions were elicited independently using an online mental modeling technology in the form of FCMs. Once the individual FCMs were standardized (i.e., using unique terminologies for similar concepts) (see ref. 55 ) , models were combined using their adjacency matrices and matrix algebra to create a model that represented the collective knowledge of stakeholder groups and thus provided a tool for leveraging WOC (see Appendix ). 3.4.4 To evaluate the accuracy and overall performance of the stakeholder - driven models we conducted in - depth interviews with fisheries experts. Experts were recruited from the National Oceanic and Atmospheric Administration (NOAA), Northeast Fisheries Science Center, Massachusetts Division of Mari ne Fisheries, and an academic institution. A purposeful sampling method was used to select a sample of fisheries scientist s with diverse scientific expertise and educational background also being involved in management, assessment, and conservation of stri ped bass fish stocks in MA. Eight experts participated with academic background in environmental sciences; natural resource management; ecology, evolution and marine biology; environmental conservation; environmental and natural resource economics; marine sciences and fisheries biology; and social sciences. Interviews with experts were semi - structured with a combination of pre - established questions and a series of interactive model evaluation practices 97 requiring scientific experts to examine the accuracy of four aggregated models: three models from homogeneous groups (recreational fishers, commercial fishers, and managers), and one diverse crowd model. Models were blinded (i.e. , experts had no information about which model represented which group). Each expe rt independently interact ed with the stakeholder - driven models and express ed dynamics using a 7 point Likert scale (1 = very inaccurate, 7 = very accurate) as a proxy measurement for m 3.4.5 Network analysis of stakeholder - driven models To identify the extent to which each aggregated model represented complex causal processes we used stochastic network analysis of causal micro - structures. Building on network theory and cognitive map analyses of complex causal structures developed by Levy et al. 39 , we compared the aggregated FCMs according to th eir network motifs (i.e., micro - structures that are constructed by two or three nodes and some unique patterns of connections between them, which shape the underlying elements of perceived causation in a cognitive map). The extent to which one cognitive map can represent complex interdependencies among social and ecological components of a natural resource system is thus linked to the distribution of complex micro motifs within its network. Theoretical and empirical studies have frequently suggested that four particular motifs exemplify more complex patterns of causation 39,45,46,49,56 58 ; therefore, their prevalence in a cognitive map indicates higher perception of complex interdependencies: bi - directionality, multiple effects, indirect effect, and feedback loop (see Figure 3.4 and ref. 39 ) . The prevalence of each motif was measured using u niform random graph tests, which compared the count of motifs in a network with the expected value of counts in randomly generated networks of the same size and density with uniform distribution of edges 59 . We measured the 98 etwork and how this count compared to the expected value of counts in 10,000 randomly generated networks of the same size and density with uniform distribution of edges. 99 APPENDIX 100 APPENDIX SUPPLEMENTARY INFORM ATION S1 Supplementary Methods S1.1 Mental Models and Fuzzy Cognitive Maps (FCM) Mental models 60 are simplified internal representations of reality that allow humans to perceive patterns of cause - and - effect relationships through reasoning and to make decisions. Mental models consist of bel iefs and subjective knowledge that are constructed as individuals observe, interact with, and experience the world around them and concurrently develop an internal model to understand and predict how it functions 61 . As such, they synthesize knowledge that is acquired through experientia l, social, and formal learning. Mental models that represent causal knowledge (e.g. , how social and ecological components are interconnected in a natural resource system) can be elicited through cognitive mapping 62 . Cog nitive maps are representations of mental models in the form of directed graphs. Nodes represent concepts that are part of the mental model and edges (arrows) are used to show the causal relationship between the concepts. Fuzzy Cognitive Maps (FCM) 36 extend causal cognitive maps in order to add a dynamic component to their analysis. These are graphical models of system components (nodes) and their causal relationships (edges), forming a weighted directed graph (Figure 3.S1) . Relationships (edges) are c haracterized by a number in the interval of [ - 1, +1], corresponding to the strength and sign of causal relationships between nodes. They, therefore, provide a semi - quantitative system modeling technique, based on auto - associative neural networks and fuzzy set theory that make cognitive maps computable (see FCM computation section of this A ppendix ). 101 A total of 32 individuals completed the online mental modeling survey including recreational fishers, commercial fishers, and fisheries managers , each creating their own FCM (Figure 3.S2) . Table S1 shows the number of participants from each stakeholder type. In addition, the mean and standard deviation of number of concepts (i.e. , nodes) and connections (i.e. , edges) used by individuals to construct FC M represent ing their mental models about striped bass population dynamics are shown in Table S1 . S1.2 Mental Models Aggregation S1.2.1 Stakeholder - specific models (homogenous groups) Individual mental models represented as FCMs can be aggregated mathematically using matrix algebra operations on their adjacency matrices. The s e aggregated models also referred can be used to represent the knowledge and perception of a group of participants and thus provide a tool for knowledge - pooling 63 . To combin e mental models of a homogenous group with individuals from a specific stakeholder type (e.g. , recreational fishers, commercial fishers, or managers) we calculated the arithmetic mean (i.e. , simple average) of edge weights that are shared in all FCMs (see also ref. 64 for more details ) : where is the adjacency matrix of the FCM of participant p , N is the total number of participants in a group, and indicates the element of this matrix with the value equals to the weight of the edge between node i and j . represents the aggregated FCM of a group with the corresponding adjacency matrix . 102 We used the above aggregation method to creat e s takeholder - specific (homogenous ) models of recreational fishers (Figure 3.S4), commercial fishers (Figure 3.S5), and fisheries managers (Figure 3.S6). S1.2.2 Crowd model (diverse group) To build an aggregated mental model of diverse stakeholders (i.e. , the crowd model), we used a multi - level aggregation technique (Figure 3.S3) . The first level of aggregation was achieved by adding mental models of individuals from the same stakeholder type and averaging the weigh ts of shared edges (see Eq. S1). At the s econd level, we aggregated the averaged stakeholder - specific models. At this level, we could have used the arithmetic mean of averaged maps to aggregate across the stakeholders; however, forming stakeholder - specific models that consist of same - type individ uals could likely amplify the accumulation of stakeholder - specific biases. To address this issue, and similar to what described in Aminpour et al. (2020), here we used the median of stakeholder - specific averaged models to further remove the effect of biase s: where indicates the element of the adjacency matrix of crowd model with the value equals to the weight of the edge between node i and j . In our case, there are three types of stakeholders: recreational fishers, commercial fishers and fisheries managers. Thus, we used the median of edge - weights across three arithmetically averaged stakeholder - specific maps (i.e. , , and ) to build the diverse crowd model (Figure 3.S7). 103 S1.3 Concept Categorization We categorized concepts used by participants into two main categories: (1) Ecological - dimension and (2) Human - dimension. The ecological - dimension was divided into two sub categories of biological concepts and habitat related concepts. In addition, the human - dimension was divided into two sub categories of social concepts and management related concepts. We measured the frequency and relative percentage of each sub - category across stakeholder types to determine stakeholder - specific biases (Figure 3.S8). S1.4 FCM computation FCM models are semi - quantitative simulation models 65 that can be used to assess the perceived dynamic behavior of the system they represent 63, 66,67 . Here, we used FCM computational analysis to demonstrate how stakeholders, based on their collective perceptions , striped bass population) given an initial change in one or combination of concepts (i.e. , scenario inputs) (e.g. , water quality or water temperature) ( also see ref. 68 for details about scenario analysis ). An increase (or a decrease) in a concept initiates a cascade of changes to other system concepts ( typically n ormalized between 0 and 1), and this iterative propagation of the initial change evolves into a so - 69 . By comparing the system states (i.e. , the value of concepts) before and after initiation of a change, FCM can be used to implem scenario analysis, and therefore represent perceived dynamic behavior of the system (in this case, striped bass fisheries). To run a scenario, the value of one or more concepts (i.e., scenario nodes) in a FCM was changed and forced to stay at either +1 (an increase) or - 1 (a decrease). This initial change passes 104 through the network of nodes and connections including feedback loops until the system reaches a new state. The consequent alterations in the state of other system concepts were calcul ated by subtracting their initial values from their values after the scenario was introduced and system evolved into a new state . The initial value of each concept also known as steady state is calculated using the following formula: where is the value of concept at iteration step k+1, is the value of concept at iteration step k, is the value of concept at iteration step k, and is the weight of the edge relationship between and . Function the concept values at each step to a normalized interval between - 1 and 1. In this study, we used a hyperbolic tangent functi on (see ref. 70 for more details about hyperbolic tangent function) : where is a real positive number (in our case ) which determines the steepness of the function . The value of each concept under a scenario was computed using the same formula (Eq . S3), but this time scenario nodes were forced to take fixed values (either +1 or - 1). The scenario s when the system was self - administered and when it was forced by fixed manipulations in the state of scenario concepts 63,69 . For each concept the change in its value as a result of running a scenario is: 105 where is the change in the value of concept , is the value of concept in the steady state, and is the value of concept after converging into a new state while scenario concepts are clamped on fixed values. S1.5 Onl ine mental modeling instructions The individuals who participated in online mental modeling survey were given a step - by - step instruction how to build a FCM model using the online mental modeling technology. Mental Modeler online tool is modeling software t hat helps individuals and communities capture their knowledge in a standardized format that can be used for scenario analysis. Based in FCM , users can develop semi - quantitative models of complex social and environmental issues by d efining the important com ponents of a system and also the relationships between these components 71 . The step - by - step direction showing in Figure 3.S10 was used to instruct participants. 106 S2 Supplementary Figures Figure 3.S1. An example of a fuzzy cognitive map (FCM) representing a mental model about striped bass fishery. The FCM was created using Mental Modeler online platform at www.mentalmodeler.org . Boxes demonstrate system concepts defined by the individual modeler and arr ows indicate causal relationships between concepts. 107 Figure 3.S2. All individual fuzzy cognitive maps (FCM) representing the mental models of 32 participants about striped bass fishery in Massachusetts. 108 Figure 3.S3. Multi - level aggregation method. A t the first level, individual maps are aggregated the second level, the resulting group means are aggregated using the median of their edge weights to produce t he crowd model. 109 Figure 3.S4. Aggregated mental model of recreational fishers. Circles demonstrate unique system concepts mentioned by the individuals of type recreational fisher. Ecological - dimension concepts are green and human - dimension components are purple . Weighted blue/red arrows indicate positive/negative causal relationships between concepts. The arrows thickness represents the strength of the causal relationships ranged from - 1 to +1. The weight of the arrows are computed using equation 3.S1 . 110 Figure 3.S5. Aggregated mental model of commercial fishers. Circles demonstrate unique system concepts mentioned by the individuals of type commercial fisher. Ecological - dimension concepts are green and human - dimension components are purple . Weighted blue/red arrows indicate positive/negative causal relationships between concepts. The arrows thickness represents the strength of the causal relationships ranged from - 1 to +1. The weight of the arrows are computed using equation 3. S1. 111 Figure 3.S6. Aggregated mental model of fis heries managers . Circles demonstrate unique system concepts mentioned by the individuals of type manager . Ecological - dimension concepts are green and human - dimension components are purple. Weighted blue/red arrows indicate pos itive/negative causal relationships between concepts. The arrows thickness represents the strength of the causal relationships ranged from - 1 to +1. The weight of the ar rows are computed using equation 3.S1 . 112 Figure 3.S7. Aggregated mental model of the diverse crowd . Circles demonstrate a parsimonious list of s ystem concepts mentioned by all individuals of all stakeholder types . This parsimonious list of system concepts is obtained by a multi - level aggregation method. Ecological - dimension concepts are green and human - dimension components are purple. Weighted blue/red arrows indicate positive/negative causal relationships between concepts. The arrows thickness represents the strength of the causal relationships ranged from - 1 to +1. The weight of the arr ow s are computed using equation 3.S1 . 113 Figure 3.S8. The frequency and the relative percentage of each category of system concepts across three stakeholder groups. The numbers on bar - graphs indicate the frequency of concepts under each specific category. x - axis shows the relative percentage. Figure 3.S9. , nodes). Evaluated concepts were those appeared in more than one model, but not all models. The opinion table in ( a ) shows whether a component is necessary (black), superfluous (white) or there is no consensus among experts (half - black, half - white). The percent of false errors b ). 114 Figure 3.S10. Step - by - step written ins tructions for participants to direct them how to use online mental modeling tool and create fuzzy cognitive maps repressing their perception of striped bass fisheries in MA and social - ecological relationships driving this system. 115 S3 Supplementary Tables Table 3.S1. The number of participants from each stakeholder type and the number of nodes and connections used in their mental models. The mean and standard deviation of number of concepts (i.e. , nodes) and connections (i.e. , edges) are shown by stakeholder types. Stakeholder group Number of Participants Number of Nodes Mean (SD) Number of Connections Mean (SD) Recreational fishers 13 11.54 (4.01) 29.85 (20.53) Commercial fishers 11 11.45 (2.84) 23.45 (12.41) Fisheries managers 8 12.00 (3.21) 27.25 (7.87) Total 32 11.63 (3.35) 27.00 (15.32) 11 6 REFERENCES 117 REFERENCES 1. Charnley, S. et al. Evaluating the best available social science for natural resource management decision - making. Environ. Sci. Policy 73 , 80 88 (2017). 2. Punt, A. E., Butterworth, D. S., de Moor, C. L., De Oliveira, J. A. A. & Haddon, M. Management strategy evaluation: best practices. Fish Fish. 17 , 303 334 (2016). 3. Gilchrist, G., Mallory, M. & Merkel, F. Can local ecological knowledg e contribute to wildlife management? Case studies of migratory birds. Ecol. Soc. 10 , (2005). 4. Arlinghaus, R. & Krause, J. Wisdom of the crowd and natural resource management. Trends Ecol. Evol. 28 , 8 11 (2013). 5. Berkes, F., Colding, J. & Folke, C. Redi scovery of traditional ecological knowledge as adaptive management. Ecol. Appl. 10 , 1251 1262 (2000). 6. Huntington, H., Callaghan, T., Fox, S. & Krupnik, I. Matching traditional and scientific observations to detect environmental change: a discussion on A rctic terrestrial ecosystems. Ambio 18 23 (2004). 7. Moller, H., Berkes, F., Lyver, P. O. & Kislalioglu, M. Combining science and traditional ecological knowledge: monitoring populations for co - management. Ecol. Soc. 9 , (2004). 8. Beaudreau, A. H. & Levin, P. S. Advancing the use of local ecological knowledge for assessing data - poor species in coastal ecosystems. Ecol. Appl. 24 , 244 256 (2014). 9. Esch, B. E., Waltz, A. E. M., Wasserman, T. N. & Kalies, E. L. Using best available science information: Determ ining best and available. J. For. 116 , 473 480 (2018). 10. Laborde, S., Imberger, J. & Toussaint, S. Contributions of local knowledge to the physical limnology of Lake Como, Italy. Proc. Natl. Acad. Sci. 109 , 6441 6445 (2012). 11. Ainsworth, C. H. Quantify ing species abundance trends in the Northern Gulf of California using local ecological knowledge. Mar. Coast. Fish. 3 , 190 218 (2011). 12. Otto - Banaszak, I., Matczak, P., Wesseler, J. & Wechsung, F. Different perceptions of adaptation to climate change: a mental model approach applied to the evidence from expert interviews. Reg. Environ. Chang. 11 , 217 228 (2011). 13. Game, E. T. et al. Fuzzy models to inform social and environmental indicator selection for conservation impact monitoring. Conserv. Lett. 11 , e12338 (2018). 14. St John, F. A. V, Steadman, J., Austen, G. & Redpath, S. M. Value diversity and conservation conflict: Lessons from the management of red grouse and hen harriers in England. People Nat. 1 , 6 17 (2019). 15. Freeman, J., Baggio, J. A. & Coyle, T. R. Social and general intelligence improves collective action in a common pool resource system. Proc. Natl. Acad. Sci. 117 , 7712 7718 (2020). 16. Adams, W. M., Brockington, D., Dyson, J. & Vira, B. Managing tragedies: understanding 118 conflict over common pool resources. Science (80 - . ). 302 , 1915 1916 (2003). 17. King, M. A. & Fairfax, S. K. Public accountability and conservation easements: learning from the Uniform Conservation Easement Act debates. Nat. Resour. J. 65 129 (2006). 18. Biggs, D. et al. Conserv. Lett. 4 , 169 183 (2011). 19. Woolley, A. W., Chabris, C. F., Pentland, A., Hashmi, N. & Malone, T. W. Evidence for a collective intelligence factor in the performa nce of human groups. Science (80 - . ). 330 , 686 688 (2010). 20. Mulgan, G. Big mind: How collective intelligence can change our world . (Princeton University Press, 2018). 21. Surowiecki, J. The wisdom of crowds: Why the many are smarter than the few and how collective wisdom shapes business. Econ. Soc. Nations 296 , (2004). 22. Krause, J., Ruxton, G. D. & Krause, S. Swarm intelligence in animals and humans. Trends Ecol. Evol. 25 , 28 34 (2010). 23. Hong, L. & Page, S. E. Groups of diverse problem solvers can outperform groups of high - ability problem solvers. Proc. Natl. Acad. Sci. 101 , 16385 16389 (2004). 24. Lorenz, J., Rauhut, H., Schweitzer, F. & Helbing, D. How social influenc e can undermine the wisdom of crowd effect. Proc. Natl. Acad. Sci. 108 , 9020 9025 (2011). 25. Kao, A. B. et al. Counteracting estimation bias and social influence to improve the wisdom of crowds. J. R. Soc. Interface 15 , 20180130 (2018). 26. Abbasi, J. Sha ntanu Nundy, MD: the human diagnosis project. Jama 319 , 329 331 (2018). 27. Sarasohn - Kahn, J. The wisdom of patients: Health care meets online social media. (2008). 28. Page, S. E. Making the difference: Applying a logic of diversity. Acad. Manag. Perspect. 21 , 6 20 (2007). 29. Page, S. E. The diversity bonus: How great teams pay off in the knowledge economy . (Princeton University Press, 2019). 30. Krause, S., James, R., Faria, J. J., Ruxton, G. D. & Krause, J. Swarm intelligence in humans: diversit y can trump ability. Anim. Behav. 81 , 941 948 (2011). 31. Boudreau, K. J., Lacetera, N. & Lakhani, K. R. Incentives and problem uncertainty in innovation contests: An empirical analysis. Manage. Sci. 57 , 843 863 (2011). 32. Bernstein, E., Shore, J. & Lazer , D. How intermittent breaks in interaction improve collective intelligence. Proc. Natl. Acad. Sci. 115 , 8734 8739 (2018). 33. Aminpour, P. et al. Wisdom of stakeholder crowds in complex social -- ecological systems. Nat. Sustain. 1 9 (2020). 34. Gray, S. et al. Harnessing the Collective Intelligence of Stakeholders for Conservation. 119 Front. Ecol. Environ. (2020). 35. Baggio, J. A. et al. The importance of cognitive diversity for sustaining the commons. Nat. Commun. 10 , 1 11 (2019). 36. Kosko, B. Fuzzy cogniti ve maps. Int. J. Man. Mach. Stud. 24 , 65 75 (1986). 37. - step fuzzy cognitive mapping approach. Ecol. Modell. 176 , 43 64 (2004). 38. Gray, S., Chan, A., Clark, D. & Jordan, R. Modeling the integration of stakeholder knowledge in social -- ecological decision - making: benefits and limitations to knowledge diversity. Ecol. Modell. 229 , 88 96 (2012). 39. Levy, M. A., Lubell, M. N. & McRoberts, N. The structure of mental models of sus tainable agriculture. Nat. Sustain. 1 , 413 420 (2018). 40. Anadón, J. D., Giménez, A. & Ballestar, R. Linking local ecological knowledge and habitat modelling to predict absolute species abundance on large scales. Biodivers. Conserv. 19 , 1443 1454 (2010). 41. Taylor, R. B., Morrison, M. A. & Shears, N. T. Establishing baselines for recovery in a marine reserve (Poor Knights Islands, New Zealand) using local ecological knowledge. Biol. Conserv. 144 , 3038 3046 (2011). 42. Gray, S. A. et al. Using fuzzy cognit ive mapping as a participatory approach to analyze change, preferred states, and perceived resilience of social - ecological systems. Ecol. Soc. 20 , (2015). 43. Gray, S., Hilsberg, J., McFall, A. & Arlinghaus, R. The structure and function of angler mental m odels about fish population ecology: The influence of specialization and target species. J. Outdoor Recreat. Tour. 12 , 1 13 (2015). 44. Stier, A. C. et al. Integrating expert perceptions into food web conservation and management. Conserv. Lett. 10 , 67 76 ( 2017). 45. Olazabal, M., Chiabai, A., Foudi, S. & Neumann, M. B. Emergence of new knowledge for climate change adaptation. Environ. Sci. Policy 83 , 46 53 (2018). 46. Hamilton, M., Salerno, J. & Fischer, A. P. Cognition of complexity and trade - offs in a wil dfire - prone social - ecological system. Environ. Res. Lett. 14 , 125017 (2019). 47. Armitage, D., Berkes, F. & Doubleday, N. Adaptive co - management: collaboration, learning, and multi - level governance . (UBC Press, 2010). 48. Jones, N. A., Ross, H., Lynam, T., Perez, P. & Leitch, A. Mental models: an interdisciplinary synthesis of theory and methods. Ecol. Soc. 16 , (2011). 49. Halbrendt, J. et al. Differences in farmer and expert beliefs and the perceived impacts of conservatio n agriculture. Glob. Environ. Chang. 28 , 50 62 (2014). 50. Reimann, S. On the design of artificial auto - associative neuronal networks. Neural Networks 11 , 611 621 (1998). 120 51. Galton, F. Vox populi. Nature vol. 75 450 451 (1907). 52. Kurvers, R. H. J. M. et al. Boosting medical diagnostics by pooling independent judgments. Proc. Natl. Acad. Sci. U. S. A. 113 , 8777 8782 (2016). 53. Rosenberg, L. & Willcox, G. Artificial human swarms outperform vegas betting markets. in Advances in Intelligent Systems and Computing vol. 880 721 729 (Springer Verlag, 2019). 54. Stat. Sci. 29 , 420 424 (2014). 55. Siqueiros - García, J. M., Lerner, A. M., Eakin, H. C. & Hernández Aguilar, B. A standardization pro cess for mental model analysis in socio - ecological systems. Environ. Model. Softw. 112 , 108 111 (2019). 56. Grotzer, T. A. & Shane Tutwiler, M. Simplifying Causal Complexity: How Interactions Between Modes of Causal Induction and Information Availability L ead to Heuristic - Driven Reasoning. Mind, Brain, Educ. 8 , 97 114 (2014). 57. White, P. A. Naive ecology: Causal judgments about a simple ecosystem. Br. J. Psychol. 88 , 219 233 (1997). 58. Trope, Y. & Liberman, N. Construal - Level Theory of Psychological Dist ance. Psychol. Rev. 117 , 440 463 (2010). 59. Butts, C. T. Social network analysis: A methodological introduction. Asian J. Soc. Psychol. 11 , 13 41 (2008). 60. Johnson - Laird, P. N. Mental models: Towards a cognitive science of language, inference, and consc iousness . (Harvard University Press, 1983). 61. Mohammed, S. & Dumville, B. C. Team mental models in a team knowledge framework: Expanding theory and measurement across disciplinary boundaries. J. Organ. Behav. Int. J. Ind. Occup. Organ. Psychol. Behav. 22 , 89 106 (2001). 62. Eden, C. On the nature of cognitive maps. J. Manag. Stud. 29 , 261 265 (1992). 63. Gray, S. A., Zanre, E. & Gray, S. R. J. Fuzzy cognitive maps as representations of mental models and group beliefs. in Fuzzy cognitive maps for applied s ciences and engineering 29 48 (Springer, 2014). 64. Jetter, A. J. & Kok, K. Fuzzy Cognitive Maps for futures studies A methodological assessment of concepts and methods. Futures 61 , 45 57 (2014). 65. Voinov, A. et al. Tools and methods in participatory mod eling: Selecting the right tool for the job. Environ. Model. Softw. 109 , 232 255 (2018). 66. Stylios, C. D. & Groumpos, P. P. Modeling complex systems using fuzzy cognitive maps. IEEE Trans. Syst. Man, Cybern. A Syst. Humans 34 , 155 162 (2004). 67. Jetter, A. J. & Sperry, R. C. Fuzzy cognitive maps for product planning: using stakeholder knowledge to achieve corporate responsibility. in 2013 46th Hawaii International 121 Conference on System Sciences 925 934 (2013). 68. Giabbanelli, P. J., Gray, S. A. & Aminpour, P. Combining fuzzy cognitive maps with agent - based modeling: Frameworks and pitfalls of a powerful hybrid modeling approach to understand human - environment interactions. Environ. Model. Softw. 95 , 320 325 (2017). 69. Papageorgiou, E. I. & Salmero n, J. L. Methods and algorithms for fuzzy cognitive map - based modeling. in Fuzzy cognitive maps for applied sciences and engineering 1 28 (Springer, 2014). 70. Harmati, I. Á. & Kóczy, L. T. Notes on the Dynamics of Hyperbolic Tangent Fuzzy Cognitive Maps. in 2019 IEEE International Conference on Fuzzy Systems (FUZZ - IEEE) 1 6 (2019). 71. Gray, S. A., Gray, S., Cox, L. J. & Henly - Shepard, S. Mental modeler: a fuzzy - logic cognitive mapping modeling tool for adaptive environmental management. in 2013 46th Hawai i International Conference on System Sciences 965 973 (2013). 122 CHAPTER 4 4 CROWDSOURCING MENTAL MODELS FOR PREDICTIN G BEHAVIO RAL RESPONSES TO CLIMATE CHANGE This chapter is in r eview for publication in Global Environmental Change ( https://www.journals.elsevier.com/global - environmental - change ) . ABSTRACT Understanding and modeling human behavioral responses to changing environmental conditions is difficult, especially at large social and environmental scales. This is due less to scientific understanding of how environmental conditions are predicted to chan ge, and more of an issue of how environmental change is perceived by humans and how these perceptions are integrated with intended behavioral responses. We developed a method for utilizing the collective knowledge and perceptions of stakeholders to predict local scale responses to climate change. Specifically, by crowdsourcing mental models of 1,464 recreational fishers across a large social - ecological gradient along the U.S. Atlantic coast, we show that simulations of warming waters and increased stormines s reveal mental model predictions about environmental change that explain divergent behavioral responses across regions, measured as the number of intended days fishing. Importantly, these diverging responses align with empirical patterns of environmental change. More broadly, our approach could be applied to predict human behavioral responses to environmental or even social changes across biogeographic scales and social - ecological contexts. 123 4.1 INTRODUCTION Climate change is projected to impact oceans throu gh a wide range of environmental changes and pulses of disturbance which vary widely at local scales 1,2 . While the environmental impacts of climate change have received significant attention, there is considerable uncertainty regarding the social impacts resulting from climate change, especially in understanding how ecological gradients 3 6 . Here, we show that crowdsourcing mental models of 1,464 recreational fishers distributed across a large social - ecological gradient from U.S. north to south Atlantic reveals latitudinal patterns of perceptions and intended behavioral responses (i.e., fishing days) to climate change (e.g. , increased storminess and warming water temp eratures). Harnessing local - with ecosystems may provide considerable insight, decreasing uncertainty on how society and ecosystems may react to climate change. Predicting how individual fishers and fishing communities may be impacted or are able to adapt to the consequences of climate change has been an increasingly high priority for fisheries social scientists 7,8 . For instance, warming ocean waters are expected to induce biogeographic shifts for many fish species 9 11 , which may impact fishing communities through the decline or disappearance of traditionally predominant species, as well as the increasing prevalence of formerly rare or novel species 7 . In addition, sea lev el rise and increased storminess may directly impact fishing communities by decreasing the number of fishing trips and damaging facilities. However, such complex scenarios could promote a multitude of social and economic outcomes for fishing communities th at are difficult to predict and may therefore increase uncertainty regarding climate change. 124 Many studies have increasingly demonstrated that the behaviors and responses of stakeholders to environmental and management changes are often complex and can be s trongly influenced by a number of factors including knowledge, perceptions, and concerns about these changes 12 . Likewise, understanding the dynamics of human institutions is often essential for predicting outcomes of coupled systems such as fisheries, as strongly influence ecosystem structure and function 13 . Understanding climate - influenced changes and creating appropriate adaptive management strategies to optimize trade - offs will require integrated modeling of numerous ecologi cal and social variables. However, uncertainties associated with human behavioral responses to climate - driven changes, such as storms and ecosystem changes, are compounded by inadequate tools and methods to quantify them 14 . We developed an online survey m ethod with fuzzy - cognitive mapping (FCM) to crowdsource mental models of climate change among 1,464 recreational fishers across the states of Massachusetts (MA), North Carolina (NC), and Florida (FL). By defining positive or negative pairwise relationships between components in a networked structure, individual FCMs represent individual - level perceptions about the social and ecological impacts of climate change, as well as intended behavioral responses 15 (see Methods). Additionally, once mathematically agg regated, these individual mental model representations can be scaled up to represent community beliefs, 16 (see Methods -- mental model aggregation). Importantly, causal connections in FCMs are numerically parameterized using fuzzy logic 17 . These mental models are therefore quantitative simulation tools that can be used to assess individual or scenario (i.e. , an activation vector that makes changes in one or a set of components, which 125 triggers a cascade of changes in other system components until the system reaches a new attractor) (see Methods). ow climate change may impact coastal recreational fisheries across three states representing higher to lower latitudes of the U.S. Atlantic Coast (Figure 4.1). Specifically, this paper focuses on three questions: 1) to what extent do survey responses regar - influenced ecological changes align with empirical patterns of climate change disturbances?, 2) how do recreational anglers perceive the individual and combined effects of warming coastal waters and increased storminess on their primary target species?, and 3) how might fishing behaviors change under these same scenarios? 4.2 RESULTS 4.2.1 Overall Climate Concern We compared the individual responses regarding the concerns of recreational fishers in MA, NC, and FL about (1) ocean w arming, (2) severe storms, and (3) fish declines. Across the three states, respondents in MA demonstrated significantly higher concerns about global warming and increased ocean temperature (Figure 4. 2 a) , while NC respondents were most concerned about incr eased severe storms (Figure 4. 2 b). In terms of the fish decline and the status of fisheries, FL respondents demonstrated the lowest concerns , while inter - state comparisons revealed a latitudinal gradient in concern that increased from south to north (Figu re 4. 2 c ). In addition, we used empirical data for changing ocean patterns over the past 20 years at the regional ecosystem scales to obtain latitudinal patterns of (1) sea surface temperature (SST) trends, (2) the trends of frequency of stormy days in coa stal regions, and (3) the proportion of 126 fish stocks that are overfished/experiencing overfishing as a proxy measure for fish declines (Figure 4.2 d - f) . All of these data support the differential angler perceptions that we measured. In particular, water tem perature increases have been greatest in MA (Figure 4.2 d), while NC has experienced the highest increase in frequency of stormy days (Figure 4.2 e), and MA is in the federal fisheries management region (New England) that has experienced the highest percen tage of overfished stocks. In general, survey responses largely aligned with observed empirical patterns, suggesting a strong conformity between subjective stakeholder perceptions and objective measures of environmental changes. Figure 4.1 . Community maps of three study regions representing Florida (FL), North Carolina (NC), and Massachusetts (MA) built by aggregating individual FCMs from each region. The inset shows the NC community model with details (see Appendix , Figure 4. S1 for details about other states). Blue/red arrows indicate positive/negative causal relationships between concepts. Edge weights represent perceived strength of the causal links. 127 Figure 4.2. a ), storminess ( b ), and fish decline ( c ) alongside patterns of empirical data on water temperature ( d ), storminess ( e ), and fisheries stock status ( f ) trends. Levels of significance are illustrated in ( a - c ) by asterisks ( p - value< 0.05, p - value< 0.01, and p - value< 0.001 are shown by one, two, and three asterisks respectively). (Note: stock status trends are missing 1997 - 98 data points due to the unavailability of information for the number of overfishing stocks. In addition, stock status data are classified based on NOAA fisheries regions: MA i s included in New England; NC is partially included in the mid and south Atlantic; and FL is partially included in the south Atlantic and Gulf). 128 4.2.2 Fishing Characteristics One question of the online survey documented the primary target species of respondents, which was later used as a concept in the mental model section of the survey. Primary target species varied considerably across our study regions. As shown in (Figure 4.3 a ), in the Northeast (i.e. , MA), Striped Bass dominates recreational fisheries with 6 3.2% of respondents listing it as their primary target species. The next closest fish species, Bluefish, represents 4.5% of primary target species. In the Mid - Atlantic (i.e. , NC), the most targeted species of Red Drum represents 23% of all target species. The next closest fish species, Summer Flounder and Spotted Seatrout, represent 10.9% and 10.1% respectively. In the Southeast (i.e. , FL), where empirical measures of fish diversity demonstrate higher species richness, the most targeted species of Snook, Re d Drum, Red Snapper, and Spotted Seatrout respectively represent 15.4%, 13.3%, 10.7%, and 10.3% of all target species. To quantitatively measure diversity of target species in each state, we used Shannon diversity index ( ) by accounting for both the numb er of unique b ). Quantification of index indicates that the diversity of primary target species increases from North to South Atlantic, which aligns with the bioge ographical patterns increasing species richness with decreasing latitude. 4.2.3 Simulating Climate Change in Mental Models We aggregated mental models by states to build regional fishing community maps. Arrays of scenario analyses with various activations of wat er temperature and water storminess were carried out to show the mental model predictions of changes in target species abundance (Figure 4.4) and intended fishing days (Figure 4.5) under a range of climate change scenarios. We find simulations of warming w aters on the community map of FL recreational fishers 129 generally produce favorable perceived outcomes of increased abundances of target species. In contrast, warming water scenarios on community maps of MA and NC yield more negative perceptions with decreas ing abundance of target species. Moreover, simulations of increased water storminess on the community mental model of NC produce negative outcomes of drastic declines in target species abundances, while these undesirable outcomes are smoother in FL and are completely flattened in MA (Figure 4.4). Figure 4.3. Diversity of target species across regions. ( a ) Species accumulation curves shows the cumulative percentage of total primary target species reached by a given number of unique species. ( b ) Circular c hart for each region shows the target species and their percentage. Only target species with more than 10% are labeled for each region: Striped Bass ( SB ), Red Drum ( RD ), Summer Flounder ( SF ), Spotted Seatrout ( SS ), Snook ( SK ), and Red Snapper ( RS ). The hor izontal bar charts show the calculated Shannon diversity index (H). 130 Figure 4.4. target species abundance. For each state results are shown for various combinations of water tem perature and storminess jointly ( a - c ); water temperature individually ( d - f ); and water storminess individually ( g - i from high decrease ( - 1) to high increase (+1). Heat map shows the perceive d changes in target species abundance from high decrease (dark red) to high increase (dark blue). In addition, patterns of behavioral responses to climate change vary across regions. Specifically, increased water temperature is predicted to variably alter intended fishing days across all three states, with the NC map indicating a stronger positive relationship and the FL map having almost no sensitivity to warming water temperature. However, decreased water 131 temperature is perceived to slightly raise intende d fishing days in MA, while the NC and FL community maps predict declines in fishing days, and these declines are more abrupt in FL when water temperature drops considerably. Moreover, simulation of increased water storminess is likely to lead to decreased fishing days across all three states, with these changes being smoother in FL and more intense in NC (Figure 4.5). Figure 4.5. regarding the intended fishing days. For each st ate results are shown for various combinations of water temperature and storminess jointly ( a - c ); water temperature individually ( d - f ); and water storminess individually ( g - i from high decrease ( - 1) to high increase (+1). Heat map shows the intended number of days fished from high decrease (dark red) to high increase (dark blue). 132 4.3 DISCUSSION Predicting how people respond to climate change across spatial scales is extremely challenging and faces both conceptual and methodological barriers 6,7,9 . Our approach of using online surveys to crowdsource mental models provides a powerful tool for studying human behavior in complex social - ecological systems and therefore overcome these barriers. How ever, it should be noted that online surveys may limit the representation of all perceived important concepts for SESs due to the lack of freedom given to participants t o include additional customized concepts 16 . Although our fixed - concept approach to rep resent mental models may not capture the full complexity of the system, it provides a standardized way to collect variation across how stakeholders perceive socially and environmentally relevant interdependencies that influence their local - scale understand ing of system dynamics and how they behaviorally respond to scenarios of environmental change. Yet, the high level of alignment between stakeholder concerns, mental model variations, and empirical patterns adds confidence on the validity of the survey data collected to construct mental models . Our study demonstrates the power of harnessing local knowledge for both understanding the changing dynamics of fisheries and other resources , as well as predicting the individual and collective behavioral responses o f groups of stakeholders to environmental changes . More broadly, our study demonstrates a novel online approach for crowdsourcing the mental models of stakeholders to predict diverging patterns of how humans respond to climate change across scales, but als o has implications for understanding disparate behavioral responses given other large - scale social changes (e.g. , globalization, massive political changes, or large - scale pandemics) . From a methodological standpoint, our study demonstrates an approach to g reatly increase the scale of data collection including mental models. This approach allow ed us compare 133 mental models across largely distributed biogeographic regions with broad implications for other desired contexts that are typically infeasible or limite d by the labor intensive traditional approaches for representing mental models 15,16 . In the specific context of fisheries science and management, a key finding of the models and simulations in our study is that perceptions and intended behavioral respon ses to environmental change (i.e., water temperature, storminess) align with empirical patterns. In particular, we found that stakeholders in Florida seem more resilient to environmental change, with the exception of very cold conditions, than stakeholders in the mid - to - north Atlantic regions. In addition, stakeholders in the northeast perceived more negative impacts on fisheries as a result of increased water temperature (Figure 4.5). We hypothesize that varying patterns of environmental change and disturb ances, as well as biogeographical patterns of increasing species richness with decreasing latitudes, drive these regional differences in stakeholder perceptions and intended behavioral responses . For instance, along the U.S. Atlantic coast, ocean waters of f northeastern New England have experienced the greatest warming, up to 3°F, while ocean warming along the Florida Gulf coastal waters is about 0.5°F over the past century 18,19 . Meanwhile, the diversity of fish species is the highest in the southeast comp ared to mid and north Atlantic. In addition, t he implications of increased storminess for fisheries have multiple dimensions. N umerous studies have suggested that climate change is likely to increase the intensity of tropical cyclones 20 and may also incr ease their frequency 21 . In the wake of these extreme events, fishing communities, as well as fishing infrastructure and opportunities, may be severely disrupted. Coastal and marine fisheries are also often constrained by wind and weather patterns of much lesser intensities as studies have shown that wind speed is an effective predictor 134 of fishing effort, particularly for offshore recreational fisheries 22 . In our study, concern for storms was gr eatest in North Carolina (Figure 4. 2 b), which aligns with empirical patterns of having the highest increase in the frequency of stormy days reported by the National Weather Service between 1997 and 2017 (Figure 4.2 e). Moreover , our results indicate that concern for fisheries declines was greatest in New England, a region with a considerably higher proportion of fisheries stocks categorized as overfished or experiencing overfishing 23 . While there is a wealth of growing physical evidence for changing ocean patterns at the global and region al ecosystem scales 24 , the social impacts and behavioral responses are often not well - understood at local to regional scale s . Our study may fill this gap by demonstrating model predictions of behavioral responses to environmental changes and conditions. This study thus supports the idea that crowdsourced mental models can provide robust and valuable tools for predicting societal or stakeholder behavioral responses to climat e change and other scenarios. Such approaches to leveraging local knowledge, therefore, may be particularly valuable when empirical data is scarce or unavailable 25 . As climate change continues to reshape the dynamics of fisheries and other social - ecologic al systems, our study provides a methodology for understanding complex stakeholder perceptions and predicting human behavioral responses . Notably, fisheries stock assessments and management plans routinely highlight fishing behavior as among the largest co ntributors to management uncertainty 26,27 . Others have argued that participatory modeling and scenario analyses of mental models offer valuable tools for understanding the human dimension of fisheries, including behavioral intentions, as well as decreasin g overall uncertainty 28,29 . However, historically, the complexity and spatial coverage of these studies has been limited by 135 logistical constraints of conducting in - person interviews to elicit mental models. Our study demonstrates an internet - based approac h to overcome these limitations and collect robust mental model data for understanding complex social - ecological system s. Such internet - based approaches may provide a way to understand how perceptions of environmental dynamics vary across social or ecologi cal scales not previously possible. 4.4 METHODS 4.4.1 Ethics Statement Review Board (IRB #13 - 07 - 16) and electronic consent was acquired from all survey participants. 4.4.2 Survey Instrument 4.4.2.1 Overview The survey instrument was designed and administered using Qualtrics Survey Research Suite. The full survey instrument included 66 questions, and the data described in this paper represent the following core sections: Fishing Characteristics, Clima te and Hazard Concerns, Mental Models, and Demographics (e.g., education, income, gender, race, birth year). 4.4.2.2 Survey items for eliciting mental models A primary section of the survey was designed to collect data necessary to assemble individual fuzzy - logic impacts on marine ecosystems. Mental models are simplified internal representations of the external world that allow individuals to perceive patterns of cause - and - effect relationsh ips and associations. These internal mental models, therefore, enable humans to make decisions through internal processes of reasoning 30 . Mental models that represent causal knowledge (e.g. , how 136 social and ecological components are interconnected in a fis hery ecosystem) can be graphically obtained through cognitive mapping techniques 31 in the form of directed graphs. Graph nodes represent concepts (i.e. , system components) and graph edges (arrows) represent the causal relationships between the concepts. I n addition, Fuzzy Cognitive Maps (FCM) are augmented forms of conventional qualitative cognitive maps 32 , which are computationally executable and can thus perform dynamic simulations of the complex system they represent. These are semi - quantitative graphi cal models of system components and their causal relationships in the form of weighted directed graphs 33 . In FCMs, causal connections (i.e. , edges) are assigned a numerical value in the interval of [ - 1, +1], corresponding to the magnitude and sign of the relationships. In our case, responses to Likert - scale questions were mapped into numerical values to determine edge weights. These numerical parametrizations of causal relationships enable FCM computations to represent system dynamics based on neural netwo rks and fuzzy set theory 15 . Our survey instrument involved a series of questions designed to select FCM concepts and assign edge weights. First, we used a two - question series to assess the relative importance of individual target species. The first questi Appendix , Figure 4.S3). All that App Appendix , Figure 4.S4). 4.4.3 Survey D ata Collection We conducted email surveys of licensed recreational anglers in Florida, North Carolina, and Massachusetts. The data presented in this paper represent 1,464 responses from a split - 137 sample design of 3,000 total respondents across the coastal s tates of Florida, North Carolina, and Massachusetts (1,000 each). Email addresses were acquired from state managed license databases. Data collection occurred through an online survey of licensed anglers over an 8 week period in October and November 2017. We used an iterative sampling (4 waves) approach, involving an initial email contact and two reminder emails 34 , until we reached a desired sample size of 1,000 complete responses. We used a three stage process to assure data quality and validity including average completion time, failed to accurately complete attention check questions, or would not question. The adjusted response rate for the survey was 14.9% after adjusting for bounced, blocked, and unopened emails. 4.4.4 Empirical Data 4.4.4.1 Ocean Warming Data To visualize recent changes in ocean temperature, we mapped mean sea surface temperature (SST) and SST trends from 1997 - 2017. Daily mean sea surface temperature data were acquired from the NOAA OI SST V2 High Resolution Dataset. ( https://psl.noaa.gov/data/gridded/data.noaa.oisst.v2.highres.html ) SST trends for the three study states were determined from 1997 - coastline to the edge of the Exclusive Economic Zone (EEZ) ( Appendix , Figure 4.S2). Simple linear regression models were fitted on annual data points to determine the trends in SST over the 20 - year period. 138 4.4.4.2 Storminess Data For the purposes of our study, we estimated storminess as the annual number of days with a designated severe weather event in a coastal county from 1997 - 2017. Severe weather events r Data Inventory; non relevant items such as drought and wildfire events were excluded from analysis. Coastal counties were determined through NOAA Economics: Ocean Watch Now (ENOW). Simple linear regressions were performed through the annual mean data to determine annual trend. 4.4.4.3 Fisheries Stock Status Data Congress to visualize stock status from years 1997 to 2017 ( https://www.fisheries.noaa.gov/national/population - assessment s/fishery - stock - status - updates) . The data was assembled from the following fisheries management regions: New England Fisheries Management Council (NEFMC), Mid - Atlantic Fisheries Management Council (MAFMC), South Atlantic Fisheries Management Council (SAFMC ) and the Gulf of Mexico Fisheries Management Council (GMFMC). For each region, we quantified the total number of stocks assessed, the total number of stocks overfished, and the total number of stocks experiencing overfishing. Percent overfished and percen t overfishing were calculated by dividing the respective stock numbers by the total number of stocks assessed in that region. 4.4.5 Analyses We used the PyFCM package ( https://github.com/payamaminpour/PyFCM/wiki ) to conduct FCM aggregation and computational ana lyses. We conducted sensitivity analyses by simulating 10,000 scenarios of climate change on the community models across states. These scenario analyses were conducted to simulate the perceived impacts of climate changes 139 (individual and joint impacts of pe rturbations in water temperature and water storminess) on fish abundance and consequent behavioral responses regarding the intended number of days fishing. 4.4.5.1 FCM aggregation Individual mental models elicited by FCMs can be aggregated mathematically by avera ging the elements of their adjacency matrices a square matrix used to represent the FCM graph where elements of the matrix indicate the numerical values of connections (i.e. , edge weight) between pairs of nodes that are adjacent in the graph (see Appendix , Figure 4.S5). These models (i.e. , as a tool for harnessing the collective wisdom) from a group of individuals 16 and gregated knowledge and perception 17 . To build aggregated maps of different states, we calculated the median of edge weights that are shared in all FCMs of individuals who belong to the same state. In contrast to conventional aggregation mechanisms that us e arithmetic mean (i.e. , simple average) of edge weights to combine FCMs 35,36 , we use median here as an alternative measure of central tendency to avoid outliers (i.e. , maps with extreme deviations from the mean of the group). One main advantage of this a ggregation method is that the community maps built by the median more precisely represent group - specific biases, and therefore better highlight inter - group variations in comparisons. The adjacency matrix of aggregated FCM of each state was obtained as foll ows: Therefore : 140 where is the set of nodes (i.e. , concepts) used to build FCMs with unique concepts, is the adjacency matrix of the aggregated FCM, and is the adjacency matrix of the individual in the set of individuals who belong to the same state. 4.4.5.2 FCM Computation FCM models can be computationally manipulated to assess the perceived dynamic behavior of the system they represent. We used FCM computational analysis to demonstrate how fishers of a state, collectively, perceive/predict the changes in the abundance of their target species and the number of days they intend to fish, given an initial change in one or combination of climate change concepts (i.e. , water temperature and water storminess). In FCM formulation, concepts initiates a cascade of changes to other system concepts based on how they are erative propagation of the initial 15 . By comparing the system states before and after implementing a scenario, FCM can represent perceived dynamic behavior of the system. The initial acti vation of each concept also known as the activation of concepts in the is calculated using the following activation rule, namely Kosko rule 32 : where is the activation of concept at iteratio n step , is the activation of concept at iteration step , is the value of concept at iteration step , and is the weight of the edge relationship from to . Function s 141 used to squash the concept activations at each step to a normalized interval between 0 and 1. In this study, we used a sigmoid function as the most common squashing function used in FCM studies: where is a real positive number (in our case ) which determines the steepness of the function . The value of parameter was determined such that the system dynamics were optimally represented 37,38 . To run a scenario, the value of scenario concepts (i.e . , water temperature and/or water storminess) was forced to a fixed activation value, and the activation of other concepts were computed using equation ( 4. 3 ) . The scenario outcomes were then calculated as the differences between the activation of the syste new state the system evolved to as the result of forced manipulation of scenario concepts. For each concept , the change in its value as a result of running a scenario is: where is the change in the value of concept , is the value (i.e. , activation) of concept in the steady state, and is the value of concept after converging into a new state while scenario concepts are clamped on fixed values. 142 APPENDIX 143 APPENDIX SUPPLEMENTARY INFORM ATION S1 Code availability Codes for mental model aggregation and FCM analyses are publically available and can be obtained on GitHub at https://github.com/payamaminpour/PyFCM/wiki . 144 S2 Supplementary Figures (a) (b) Figure 4.S1. Community maps of study regions representing ( a ) Florida (FL) and ( b ) Massachusetts (MA) built by aggregating individual FCMs from each region. 145 (a) (b) Figure 4.S2. ( a ) Mean sea surface temperature for 2017. ( b ) Trends in monthly mean sea surface temperature from 1997 - 2017. 146 Figure 4.S3. Screensh ot of survey question used to determine survey respondents target species. The answer to this survey question was then populated into the subsequent mental model survey questions. 147 Figure 4.S4. Screenshot of survey question used to ascribe edge weight relationships among concepts. 148 Figure 4.S5. Adjacency matrix showing the corresponding relationships among model concepts derived from the online survey. 149 S3 Supplementary Tables Table 4. S1. Survey sample demographics and fishing characteristics of respondents within each state. FL NC MA Frequency Percent Frequency Percent Frequency Percent Education Less than high school 8 1.1% 4 0.5% 5 1.1% High school diploma or GED 97 12.8% 77 10.2% 62 13.1% Some college or 2 year degree 255 33.7% 245 32.4% 116 24.6% Bachelor's degree 253 33.5% 256 33.9% 165 35.0% Master's degree 95 12.6% 121 16.0% 78 16.5% Law or MD 31 4.1% 24 3.2% 23 4.9% Doctorate (PhD) 17 2.2% 29 3.8% 23 4.9% Income $25k or less 29 3.8% 18 2.4% 7 1.5% $25,001 to $35k 29 3.8% 17 2.2% 8 1.7% $35,001 to $50k 47 6.2% 47 6.2% 27 5.7% $50,001 to $75k 85 11.2% 86 11.4% 35 7.4% $75,001 to $100k 130 17.2% 128 16.9% 71 15.0% $100,001 to $150k 134 17.7% 186 24.6% 98 20.8% $150,000 to $250k 106 14.0% 102 13.5% 87 18.4% More than $250k 81 10.7% 60 7.9% 54 11.4% Prefer not to answer 115 15.2% 112 14.8% 85 18.0% 150 Gender Male 635 84.0% 658 87.0% 438 92.8% Female 109 14.4% 80 10.6% 25 5.3% Other 1 0.1% 2 0.3% 0 0.0% Prefer not to answer 11 1.5% 16 2.1% 9 1.9% Race White 659 87.2% 688 91.0% 408 86.4% Black or African American 1 0.1% 17 2.2% 8 1.7% American Indian or Alaska Native 11 1.5% 11 1.5% 2 0.4% Asian 10 1.3% 7 0.9% 12 2.5% Native Hawaiian or other Pacific Islander 2 0.3% 3 0.4% 1 0.2% Hispanic or Latino 34 4.5% 6 0.8% 9 1.9% Prefer not to answer 48 6.3% 37 4.9% 41 8.7% Age less than 21 6 0.8% 7 0.9% 9 1.9% 22 - 30 49 6.5% 38 5.0% 31 6.6% 31 - 40 108 14.2% 95 12.6% 67 14.2% 41 - 50 162 21.4% 170 22.5% 80 16.9% 51 - 64 379 50.1% 314 41.5% 192 40.7% 65+ 52 6.9% 132 17.5% 93 19.7% 151 REFERENCES 152 REFERENCES 1. Wilbanks, T. J. & Kates, R. W. Global change in local places: how scale matters. Clim. Change 43 , 601 628 (1999). 2. Arnell, N. W., Lowe, J. A., Challinor, A. J. & Osborn, T. J. Global and regional impacts of climate change at different levels of global temperature increase. Clim. Change 155 , 377 391 (2019). 3. Glaser, S. M. et al. Complex dynamics may limit p rediction in marine fisheries. Fish Fish. 15 , 616 633 (2014). 4. Clayton, S. et al. Psychological research and global climate change. Nat. Clim. Chang. 5 , 640 646 (2015). 5. Link, J. S., Nye, J. A. & Hare, J. A. Guidelines for incorporating fish distributi on shifts into a fisheries management context. Fish Fish. 12 , 461 469 (2011). 6. Pinsky, M. L. & Fogarty, M. Lagged social - ecological responses to climate and range shifts in fisheries. Clim. Change 115 , 883 891 (2012). 7. McClanahan, T. R. & Cinner, J. Ad apting to a changing environment: confronting the consequences of climate change . (OUP USA, 2012). 8. Marshall, N. A. & Marshall, P. A. Conceptualizing and operationalizing social resilience within commercial fisheries in northern Australia. Ecol. Soc. 12 , (2007). 9. Pörtner, H. O. & Peck, M. A. Climate change effects on fishes and fisheries: towards a cause - and - effect understanding. J. Fish Biol. 77 , 1745 1779 (2010). 10. Fodrie, F. J., Heck Jr, K. L., Powers, S. P., Graham, W. M. & Robinson, K. L. Climate - related, decadal - scale assemblage changes of seagrass - associated fishes in the northern Gulf of Mexico. Glob. Chang. Biol. 16 , 48 59 (2010). 11. Pinsky, M. L., Worm, B., Fogarty, M. J., Sarmiento, J. L. & Levin, S. A. Marine taxa track local climate velocities. Science (80 - . ). 341 , 1239 1242 (2013). 12. Cinner, J. E., Daw, T. & McClanahan, T. R. Socioeconomic factors that affect artisanal Conserv. Biol. 23 , 124 130 (2009). 13. Dietz, T., Ostrom, E. & Stern, P. C. The struggle to govern the commons. Science (80 - . ). 302 , 1907 1912 (2003). 14. USGCRP. Impacts, Risks, and Adaptation in the United States: Fourth National Climate Assessment, Volume II [Reidmiller, D.R., C.W. Avery, D.R. Easterling, K.E. Kunkel, K.L.M. Lewis, T.K. Maycock, and B.C. Stewart (eds.)]. USGCRP US Global Change Research Program Fourth National Climate Assessment vol. II https://www.globalchange.gov/nca4 (2018). 15. Özesmi, U. & Özesmi, S. - step fuzzy cognitive mapping approach. Ecol. Modell. 176 , 43 64 (2004). 153 16. Gray, S. A., Zanre, E. & Gray, S. R. J. Fuzzy cognitive maps as representations of mental models and group beliefs. in Fu zzy cognitive maps for applied sciences and engineering 29 48 (Springer, 2014). 17. Aminpour, P. et al. Wisdom of stakeholder crowds in complex social -- ecological systems. Nat. Sustain. 1 9 (2020). 18. Cheung, W. W. L., Watson, R. & Pauly, D. Signature of ocean warming in global fisheries catch. Nature 497 , 365 368 (2013). 19. Robson, J., Sutton, R., Lohmann, K., Smith, D. & Palmer, M. D. Causes of the rapid warming of the North Atlantic Ocean in the mid - 1990s. J. Clim. 25 , 4116 4134 (2012). 20. Dyer, C. L. Culture of Response, and the Fishing Peoples of South Florida and Louisiana. angry earth Disaster Anthropol. Perspect. 213 231 (1999). 21. Kossin, J. P., Knapp, K. R., Olander, T. L. & Ve lden, C. S. Global increase in major tropical cyclone exceedance probability over the past four decades. Proc. Natl. Acad. Sci. (2020). 22. Powers, S. P. & Anson, K. Estimating recreational effort in the Gulf of Mexico Red Snapper Fishery using boat ramp c ameras: Reduction in federal season length does not proportionally reduce catch. North Am. J. Fish. Manag. 36 , 1156 1166 (2016). 23. Fishery Stock Status Updates | NOAA Fisheries. https://www.fisheries.noaa.gov/national/population - assessments/fishery - stock - status - updates. 24. Clarke, L. et al. National Climate Assessment IV, Chapter 17: Sector Interactions, Multiple Stressors, and Complex Systems. in AGU Fall Meeting Abstracts (2018). 25. Sagarese, S. R. et al. Lessons learned from data - limited evaluations of data - rich reef fish species in the Gulf of Mexico: implications for providing fisheries management advice for data - poor stocks. Can. J. Fish. Aquat. Sci. 76 , 1624 1639 (2019). 26. Fulton, E. A., Smith, A. D. M., Smith, D. C. & van Putten, I. E. Human be haviour: the key source of uncertainty in fisheries management. Fish Fish. 12 , 2 17 (2011). 27. Patrick, W. S., Morrison, W., Nelson, M. & Marrero, R. L. G. Factors affecting management uncertainty in US fisheries and methodological solutions. Ocean Coast. Manag. 71 , 64 72 (2013). 28. Röckmann, C. et al. The added value of participatory modelling in fisheries management -- what has been learnt? Mar. Policy 36 , 1072 1085 (2012). 29. Gray, S., Hilsberg, J., McFall, A. & Arlinghaus, R. The structure and function of angler mental models about fish population ecology: The influence of specialization and target species. J. Outdoor Recreat. Tour. 12 , 1 13 (2015). 30. Johnson - Laird, P. N. Mental models: Towards a cognitive science of language, inference, and conscious ness . (Harvard University Press, 1983). 154 31. Eden, C. On the nature of cognitive maps. J. Manag. Stud. 29 , 261 265 (1992). 32. Kosko, B. Fuzzy cognitive maps. Int. J. Man. Mach. Stud. 24 , 65 75 (1986). 33. Voinov, A. et al. Tools and methods in participator y modeling: Selecting the right tool for the job. Environ. Model. Softw. 109 , 232 255 (2018). 34. Dillman, D. A., Smyth, J. D. & Christian, L. M. Internet, phone, mail, and mixed - mode surveys: the tailored design method . (John Wiley & Sons, 2014). 35. Gray , S., Chan, A., Clark, D. & Jordan, R. Modeling the integration of stakeholder knowledge in social -- ecological decision - making: benefits and limitations to knowledge diversity. Ecol. Modell. 229 , 88 96 (2012). 36. Jetter, A. J. & Kok, K. Fuzzy Cognitive Ma ps for futures studies A methodological assessment of concepts and methods. Futures 61 , 45 57 (2014). 37. Knight, C. J. K., Lloyd, D. J. B. & Penn, A. S. Linear and sigmoidal fuzzy cognitive maps: an analysis of fixed points. Appl. Soft Comput. 15 , 193 202 (2014). 38. Papageorgiou, E. I. & Salmeron, J. L. A review of fuzzy cognitive maps research during the last decade. IEEE Trans. Fuzzy Syst. 21 , 66 79 (2012).