EVOLUTIONOFCOOPERATIONINTHELIGHTOFINFORMATIONTHEORY By MasoudMirmomeni ADISSERTATION Submittedto MichiganStateUniversity inpartialentoftherequirements forthedegreeof ComputerScience-DoctorofPhilosophy Ecology,EvolutionaryBiologyandBehavior-DualMajor QuantitativeBiology-DualMajor 2015 ABSTRACT EVOLUTIONOFCOOPERATIONINTHELIGHTOFINFORMATION THEORY By MasoudMirmomeni Cooperationisubiquitousintbiologicallevelsandisnecessaryforevolutionto shapethelifeandcreatenewformsoforganization.Genescooperateincontrollingcells; cellsntlycollaboratetogethertoproducecohesivemulti-cellularorganisms;members ofinsectcoloniesandanimalclanscooperateinprotectingthecolonyandprovidingfood. Cooperationmeansthatmembersofagroupbearacost, c ,foranotherindividualstoearna b b .Whilecooperatorsofthegrouphelpothersbypayingacost,defectorsreceivethe bofthisaltruisticbehaviorwithoutprovidinganyserviceinreturntothegroup.To addressthisdilemma,hereweuseagametheoreticapproachtomodelandstudyevolutionary dynamicsthatcanleadtobehavior.Evolutionarygametheoryisanapproachto studyfrequency-dependentsystems.Inevolutionarygamestheofindividualsdepends ontherelativeabundanceofthevarioustypesinthepopulation.Weexploretstrate- giesandtgamessuchasiteratedgamesbetweenplayerswithconditionalstrategies, multiplayergames,anditeratedgamesbetweenfullystochasticstrategiesinnoisyenviron- mentstothenecessityconditionsthatleadtocooperation.Interestingly,weseethat inallofthesegamescommunicationisthekeyfactorformaintainingcooperationamong individuals.Weshowthatcommunicationandinformationexchangeisnecessaryfor theemergenceofcostlyaltruism,andtomaintaincooperationinthegroupthereshouldbe minimumrateofcommunicationbetweenindividuals.Wequantifythisminimumamount ofinformationexchange,whichisnecessaryforindividualstoexhibitcooperativebehavior, byanoisycommunicationchannelbetweentheminiteratedstochasticgamesand measuringthecommunicationrate(inbits)duringthebreakdownofcooperation. Idedicatethistomymother,father,brother,andmybelovedwife.Thanksforallthe andbeingalwaysthereforme. InmemoryofProf.CaroLucas.Youlefttsofgraceonmylife.Youshan'tbe forgotten. iv ACKNOWLEDGMENTS IshouldusethisopportunitytothankProf.ChrisAdamiandDr.ArendHintze.These yearshasbecomepartofme,andIwillhavethesepreciousexperienceswithme forever.Ineverforgettheexcitingmomentswehadaftersolvingaproblem.Thankyou forallyoursupport.Icouldnotachieveanythingwithoutyourhelpandguidance.Iindeed enjoyedtheseyearsworkingwithyou,thinkingwithyou,andlaughingwithyou. v PREFACE Amanisasuccessifhegetsupinthemorn- ingandgetstobedatnight,andinbetween hedoeswhathewantstodo. -BobDylan Igrewupamonganordinaryfamily.Allofushadcompletelytinterests.My fatherwasabankerandatotally\not-religious"Muslim!Ontheotherhand,mymother wasaveryreligiousschoolteacher.MybrotherandIweresomethingintermediate.Inthis family,mybrotherandIlearnedhowtochallengeeverythinganddonotbelieveeverything withoutasolidproof.MybrotherandIasthedescendantsofthisfamilychosedt careersinourlife.Mybrotherdecidedtostudyinmedicalschool;however,afterhe realizedthathedoesnothavethedevotiontothiscareer,hequitandbecameamovie critiquesincebeing\movie"runsinourfamily.I,ontheotherhand,wenttoan engineeringschool,andstudiedelectricalengineeringsincemyfatherwantedmetolearn hisVHSplayer!Insteadoflearninghowtoavideoplayer,thereIlearnedhow tomodelanydynamicwithaninput-outputsystem.Ialsopracticedcalligraphy,screen writing,andmoviedirectingandwroteshortstoriesandshortscreenplays;studiedcontrol engineeringandArIntelligence(A.I.)andtaughtmanyadvancedcoursesinelectrical engineeringandintelligence. SinceIbeganmystudyinUniversityofTehranasanA.I.graduatestudent,Iestablished anever-growingpassiontowardevolution.Asacontrolengineer,Iwastryingtoapply biologicallyinspiredmodelstomodelandpredictexoticphenomenainnature.Iwasamazed toseethecomplexityamongnaturalandbiologicaldynamics,andIwasalwaysseekingthe originofthiscomplexity.Iwantedtoknowhowspeciesexhibitcomplexbehaviorandinteract vi witheachotherinveryintricateways.Iwantedtounderstandunderwhatconditionssimple organismsevolvetocomplicatedmulti-cellularorganisms. ThroughyearsofmystudyattheUniversityofTehranundersupervisionoflate\Prof. CaroLucas"Igotfamiliarizetoevolutionaryprocessesandtformsofspeciesinter- actioninnature.JoiningBEACONatMichiganStateUniversityandthe\AdamiLab",I becameanecologistandevolutionarybiologist.Ecologyandevolutionarybiology issoimmenseandchallenging,andIstill,asanoviceresearcherinthisbroadneedto studyandexperiencealot.YetIhavelearnedalot.Fromacontrolengineerthatsimple designisalwaysoneofhisobjectivesandalwaystriestoapproximateandmodelanynonlin- earnaturaldynamicwithatime-invariantlinearinput-outputgreyboxmodel,nowItryto observespecies'interactionintheirnicheanddevelopanonlinearmodel,whichincludesthe nonlinearoftheenvironmentstothesespecies.Itrytounderstandhowcomplexity evolvesoverthecourseofevolution,hownaturalselectionthatfavorstomembersof apopulationmaintainscooperationandaltruisticbehavior,orhowcooperationemergesin agroupofanimalsandstabilizetheirdespoticsociety. Thesedays,Igetupearlyinthemorning,andgotobedverylateatnight,andin betweendowhatIalwayslikedtodo:investigatespeciesinteraction,hypothesizehowthey maintaincooperationintheirgroup,developamodelanddesignexperimentstotestthe validityofmodel,writeaprogramtotestmyhypotheses,analyzetheresults,comparethe resultswiththewell-establishedtheoriesinthecorpusofpublishedliterature,readabout evolutionarygametheory,generatecolorfulplotsforarticlesandpresentations,videochat tomybelovedwifeinCaliforniaandmyfamilybackinIran,andIfeelhappilysuccessful vii becausethesethingsgivemeenergyandhopetoliveandenjoythelife. MasoudMirmomeni June2015 TABLEOFCONTENTS LISTOFTABLES .................................... xiii LISTOFFIGURES ................................... xv Chapter1Introduction ............................... 1 1.1Motivation.....................................2 1.2Whatisevolution?................................5 1.3Informationexchangeandcommunicationchannel...............6 1.4Evolutionarygametheory............................7 1.4.1Kinselectionandinclusive....................8 1.4.2Directandindirectreciprocity......................9 1.4.3Spatialarrangements...........................10 1.5Evolutionarygamedynamics...........................10 1.6Organizationofthethesis............................11 Chapter2LiteratureReview ........................... 12 2.1Historyofevolutionarygames..........................13 2.2Inclusivetheoryandevolutionofcooperation..............16 2.3Evolutionofcooperationinstochasticgames..................22 2.4Altruisminmultiplayergame..........................25 2.5Altruisminspatialgames............................28 2.6Chaptersummary.................................29 Chapter3EvolutionofInformationContentinDigitalOrganisms .... 33 3.1Whatisinformationandhowisitrelatedtocomplexity?...........34 3.2ExperimentalSetup................................36 3.2.1Experiment1...............................37 3.2.2Experiment2...............................39 3.3Results.......................................39 3.3.1Priceequationandasatraitoverthecourseofevolution...39 3.3.2Evolutionoflogictasksascorrelatedcharactersto.......40 3.3.3Evolutionofinformationcontent....................53 3.3.4Priceequationanduncorrelatedcharacterstothe.......53 3.3.5Genomelengthasaweaklyselectabletraitoverthegenerations...66 3.4Discussion.....................................73 3.5ChapterSummary................................98 Chapter4EvolutionofCooperationinConditionalGames ......... 99 4.1Evolutionofcooperationinunconditionalgames:areview..........100 4.1.1IteratedPrisonersDilemma(IPD)....................107 ix 4.2Stabilityanalysisofedpointsinunconditionalgames:areview......108 4.2.1Iteratedprisonersdilemma........................111 4.3Evolutionofcooperationinconditionalgames.................111 4.3.1Conditionalcooperationiniteratedprisonersdilemma.........115 4.4Evolutionofcooperationinconditionalgames:stabilityanalysis.......116 4.4.1StabilityofconditionalcooperativestrategyinIPD..........118 4.5Communicationandinformationexchangeinconditionalgames.......119 4.5.1Communicationbetweenconditionalcooperatorandpuredefectorin noiselessenvironment...........................120 4.5.2Communicationbetweenconditionalcooperatorandpuredefectorin noisyenvironment............................122 4.5.3Communicationbetweenvigilantcooperatoranddefectorincondi- tionalgames................................123 4.6ChapterSummary................................127 Chapter5EvolutionofSuper-reciprocityinIteratedTrustDilemma(ITD)129 5.1IteratedTrustDilemma.............................130 5.2SecondarycooperationastheoptimalstrategyinITD.............133 5.2.1Optimalstrategyamongagroupofplayersthatcompletelyengagein secondarycooperationplay.......................134 5.2.2OptimalstrategyamongahomogenousgroupofplayersinbothIPD andITDregimes.............................136 5.3Quantifyingcommunicationandinformationexchangebycalculatingmutual informationbetweenrandomvariablesinconsecutiveiterationsofITDgame141 5.3.1Mutualinformationbetween G ( t ) andotherrandomvariablesat \ t +1 "..................................142 5.3.1.1Mutualinformationbetween G ( t ) and A S ( t +1) .....143 5.3.1.2Mutualinformationbetween G ( t ) and A O ( t +1) .....144 5.3.1.3Mutualinformationbetween G ( t ) and G ( t +1) ......146 5.3.2Mutualinformationbetween A S ( t ) andotherrandomvariablesat \ t +1 "..................................147 5.3.2.1Mutualinformationbetween A S ( t ) and A S ( t +1) ....147 5.3.2.2Mutualinformationbetween A S ( t ) and A O ( t +1) ....149 5.3.2.3Mutualinformationbetween A S ( t ) and G ( t +1) .....150 5.3.3Mutualinformationbetween A O ( t ) andotherrandomvariablesat \ t +1 "..................................151 5.3.3.1Mutualinformationbetween A O ( t ) and A S ( t +1) ....151 5.3.3.2Mutualinformationbetween A O ( t ) and A O ( t +1) ....153 5.3.3.3Mutualinformationbetween A O ( t ) and G ( t +1) .....154 5.3.4Mutualinformationbetween G ( t ) andotherrandomvariablesat \ t +1 "..................................156 5.3.5Mutualinformationbetween A S ( t ) andotherrandomvariablesat \ t +1 "innoisyenvironments......................159 5.3.5.1Mutualinformationbetween A S ( t ) and A S ( t +1) ....160 5.3.5.2Mutualinformationbetween A S ( t ) and A O ( t +1) ....162 x 5.3.5.3Mutualinformationbetween A S ( t ) and G ( t +1) .....164 5.3.6Mutualinformationbetween A O ( t ) andotherrandomvariablesat \ t +1 "..................................166 5.3.6.1Mutualinformationbetween A O ( t ) and A S ( t +1) ....166 5.3.6.2Mutualinformationbetween A O ( t ) and A O ( t +1) ....168 5.3.6.3Mutualinformationbetween A O ( t ) and G ( t +1) .....169 5.4ExperimentalSetup................................171 5.5Results.......................................173 5.5.1ofTemptationPay.......................174 5.5.2ofmutationrate..........................176 5.5.3ofenvironmentalnoise......................180 5.6Discussion.....................................180 5.7ChapterSummary................................201 Chapter6GameTheoreticModelforSocialBehaviorinAnimals .... 203 6.1Populationdynamicofpublicgoodsgame...................205 6.1.1Paymatrixrepresentationofthepublicgoodsgame........210 6.1.2Punishmentasasignalingmechanismbetweencooperatorsanddefec- torsinPGG................................213 6.1.2.1Communicationbetweenpunishingcooperatorandpurede- fectoroverroundsofpublicgoodsgameinnoiselessenviron- ment...............................214 6.1.2.2Communicationbetweenpunishingcooperatorandpurede- fectoroverroundsofpublicgoodsgameinnoisyenvironment217 6.1.2.3Communicationbetweenpunishingcooperatoranddefector inpublicgoodsgame......................218 6.2Collectivehuntinggame.............................222 6.2.1Collectivehuntinggamewithnopunishment..............225 6.2.1.1Solitaryhunting( =1)....................226 6.2.1.2Allornothing( = k +1)...................231 6.2.1.3Smallgrouphunting(1 < 0and BR C> 0, whichindicatestheriseinfrequencyofthecooperatortypeisdue tocostlycooperationbetweencloselyrelatedpartners.Withouthav- ingfurtherinformation,thereisnostatisticalorscienreasonto concludethatthisinterpretationiscorrect[1].............20 xv Figure2.2:Regressionisnotcapableofidentifyingthecausesoffrequencychange. InthisexampleAllenetal.demonstratedhowregressionmethods canbemisleadingincharacterizingthereasonofchangeinallele frequency.Theirthreehypotheticalexamplesdepicthowtheregres- sionmethodleadstowronginterpretations.(a)Ahanger-on(indi- catedbypurple)interactswithapartner.Theregression recipemisinterpretsthisbehaviorasmutuallybcooperation ( B> 0 ;C< 0).(b)Ajealousindividual(indicatedbyred)attacks anindividualofhighwhichreducestherecipientsfrom 5to4,andtheattackersfrom1to0.Theregressionrecipe misinterpretsthisattackascostlycooperation( B;C> 0).(c)A nurse(indicatedbyblue)helpsanindividualoflowess,whichas aresultincreasestherecipientsfrom0to0.5(thisgivesan extra50%chanceofhavingananddecreasesthenurses from1to0.5.Theregressionrecipeagainmisinterpretsthis aidascostlyharmingorspite( B< 0 ;C> 0)[1]...........21 Figure2.3:Thespatialiteratedprisoner'sdilemmadepictsmarvelouschaotic patters.Inthisexamplethesizeofarenais200 200.Theedges arewrappedaroundtoavoidboundaryconditions.Thecolorcode isasfollows:blueindicatescooperatortocooperatortransition;red representsadefectortodefectortransition;greenrepresentsacoop- eratorwithadefectorasparent,andyellowindicatesadefectorwith acooperatorasparent.Themoregreenoryellowinthepicture, themorevariationoccursfromgenerationtogeneration.Allblue andallredarenaiscompletelystatic.(a) b =1 : 35,thissituations isadvantageoustocooperator(b)1 : 7 c b where r isthedegreeofrelatedness(cocientofrelatedness), b isthebofthealtruism totherecipientofcooperation,and c isthecostofaltruismtothealtruist.Usingcostand bhelpedHamiltontohaveaneconomicviewpointtonaturalselectionasanoptimizer thattriestomaximizeyetletthealtruistsexistinthepopulation[81].Although Hamiltonbecameoneoftheleadingevolutionarybiologistsofthe20 th centurybecauseof 16 hisworksonkinshipandaltruism,nobodytookhisworkon\kinselection"seriouslyunless aningeniouschemist,whoquithisjobatIBMandbeganhisnewcareerasanevolutionary biologistattheGaltonLaboratory,Londonnoticedthispaperandusedthisruletodevelop hismarveloustheoryatmid-1970s.ThismanwasGeorgePriceandbecameHamilton'sclose frienduntilhecommittedsuicideinJanuary1975inhisapartment.HeappliedHamilton's ideatoextendFisher'sfundamentaltheorem,andtheresultwasanovelcovariancemodel forevolutionarychangesoftraits.Byapplyingthe\Priceequation"tocheckthevariation ofcooperativetraitsandaltruisticbehavior,wethatwhenaltruistsaresurroundedby manybloodkininthepopulationascarriersofaltruisticgenes,naturalselectionfavors altruismandnumberofproducedbyanindividualcarryingthealtruisticgenes increasesoverthegenerations.Inotherwords,ifthemeanrelatednesswithinthepopulation islessthanthemeangeneticrelatednesswithinthepopulation,thenmalevolentbehavior canevolve.Price'snovelideawaspublishedinNaturein1970andlaterwasusedbyDavid C.QuellertoextendtheHamilton'sinclusivetheory[81]. Despiteitsreputation,Hamilton'srulehasbeenoftencriticizedbeingambiguousorin- adequatelygeneral[17,82,83,84,85,86];thus,variouscorrectionshavebeengivento thistheory,whichontheonehand,increasethegeneralityofthetheory,andontheother hand,erodethesimplicityoftheHamilton'srule[13,87,88,89].Inmostoftheseextended versionsofHamilton'srule,assortmentandnon-additivitywereaddedtothemodelasne- cessityconditionstomaintaincooperation[13,90].Havingassortmentinthepopulation helpsaltruiststointeractmostlywithaltruists,whichmakesthebofcooperationbe distributedmostlytoaltruisticcarriers.Atthesametimedefectorsornon-cooperatorsare stuckinteractingwitheachotherandtheabsenceofcooperation.Non-additivity hassimilaronaltruism.Cooperationcanevolveevenintheabsenceofpositiveas- 17 sortmentwhenevercollectivecooperationyieldssynergisticb[90].Quellerinhis extensiontoinclusivetheoryclearlydemonstratedtherolesofpositiveassortment andnon-additivitytotheevolutionofcooperation[13,91]. InHamilton'soriginalformofkinselectiontheory, r isthecotofrelatedness betweendonorsandrecipients;however,thismeasureinQueller'sinclusivetheoryis generalizedtoameasureofassortmentbetweenindividualswithcooperativegenotypeand therecipient'scharacterwithwhichdonorsinteract[13,91].InQueller'sinclusive theory r isacovariancemeasureofassortment: r = Cov g;z 0 Cov ( g;z ) (2.1) where g istheselfgenotype, z istheselfphenotype(0fordefectionand1forcooperation), and z' istheaveragephenotypeofrecipient[13,91].Substitutingthisratiointothe Hamilton'sformofkinselectiontheory,wehave: Cov g;z 0 b>Cov ( g;z ) c (2.2) whichsimplysaysthatnaturalselectionfavorscooperationwhenassortmentbetweenthose withcooperativegenotypeandcharactersofrecipients,scaledbythebofaltruism onaveragebegreaterthantheassortmentbetweenthecooperativegenotypeanddonor's character,scaledbythecostofthealtruism[13,91].ThisextendedversionofHamilton's kinselectiontheoryisapplicabletointeractionsamongrelatives,non-relatives,andeven acrossspecies[91],aswellasaccommodatingthegenotype/phenotypethatresult fromconditionalbehavior(e.g.initeratedinteractions[91]).Itisworthmentioningthat inclusivetheoryinbothoriginalandextendedformisonlyapplicabletoproblems 18 withdirectionalselection,andwhenselectionisdisruptive,thisruleisnotapropermodelof socialbehavior[91].Byaddinganothercovariancetermtomeasuretheassortmentbetween altruisticgenotypeandthedegreetowhichcooperativebehaviorismutual,scaledbythe amountofdeviationfromadditivity,Quelleralsoincludednon-additivityintohisinclusive formulationofHamilton'sruleas: Cov g;z 0 b + Cov g;zz 0 d>Cov ( g;z ) c (2.3) where d> 0representsthesynergyfactor, d< 0representsdiminishingreturns, d =0 representsadditivity.Inequality(2.3)depictstwofundamentalwaystocompensateforan averagecarrier'slocale[91]: thelpfromothers,and/or tsynergisticrewardsformutualcooperation Inclusivetheoryinitsmanygeneralformshasbeenappliedtoaddressawide rangeofbiologicalproblemsandwidelyacceptedasageneralmodeltoexplaintheevolution ofsocialbehavior[92,93,94,95,96,97,98,99];however,allofthesegeneralizations overtheyearsdistortedtheinitialideaofinclusiveandcreatedanerraticlanguage forthistheory,whichhaveledtotmisunderstandingandmeaninglessargument [100].Thestrongestcriticismsconcernhowspgeneticvariationsaltertheoutcomeof selection,andhowinclusivetheorycanbepredictiveandprovidevitalperspectiveon evolutionaryprocess[89,101,102,103,104].Thesecriticismsclaimthatinclusive theoryisalimitedconceptandworksonlyforasmallsubsetofevolutionaryprocessesin naturesinceitassumespersonaltobeadditivetotheindividualactions;however, 19 Figure2.1:Resultofregressionanalysisofahypotheticalchangeinallelefrequency.Allen etal.usedthisexampletoshowregressionanalysiscouldbemisleading.Inthisanalysisthe startingingredientsarethegenetictypesrepresentedbycolors(blue:cooperator( g =1), grey:defector( g =0)),interactionbetweenpartnersisrepresentedbyarrows,andnumbers ofeventualasanindicatorisrepresentedbynumbersofeachindividual presentataparticulartime.Itispossibletocalculatethechangeinallelefrequencyfrom thesedata.Theyalinearmodelofnumberbasedoneachindividualsown genotypeandpartnersgenotypetothisdataset.Forthisprocess,theyobtained B;C> 0 and BR C> 0,whichindicatestheriseinfrequencyofthecooperatortypeisduetocostly cooperationbetweencloselyrelatedpartners.Withouthavingfurtherinformation,thereis nostatisticalorscienreasontoconcludethatthisinterpretationiscorrect[1]. weknowthatthisisatleastnottruefortheQueller'sextensiontoinclusivetnesstheory sinceheaddedthesynergyfactorasanon-additivetermtopersonal[1].Moreover, thepredictabilitypowerofinclusivetheoryhasbeenunderresearcherscriticismsince itincludesthelinearregressionbasestoavoidadditivityrestraint(Fig.(2.1)and(2.2)). Researchershaveshownthatthistheorynotusefuliftheobjectiveistoanalyzewhether mutationsthatmodifysocialbehaviorarefavoredoropposedbynaturalselection[1]. Insum,inclusivetheorywithitsgeneralizationformsisakindofcausalanalysis thatanalyzestheevolutionarycausesofsocialphenotypes[100].Initsoriginalform,three forcesareinvolvedtoevolutionarychanges:thecostasadirectcomponentthat 20 Figure2.2:Regressionisnotcapableofidentifyingthecausesoffrequencychange.Inthis exampleAllenetal.demonstratedhowregressionmethodscanbemisleadingincharacter- izingthereasonofchangeinallelefrequency.Theirthreehypotheticalexamplesdepicthow theregressionmethodleadstowronginterpretations.(a)Ahanger-on(indicatedbypurple) interactswithapartner.Theregressionrecipemisinterpretsthisbehavioras mutuallybcooperation( B> 0 ;C< 0).(b)Ajealousindividual(indicatedbyred) attacksanindividualofhighwhichreducestherecipientsfrom5to4,and theattackersfrom1to0.Theregressionrecipemisinterpretsthisattackascostly cooperation( B;C> 0).(c)Anurse(indicatedbyblue)helpsanindividualoflow whichasaresultincreasestherecipientsfrom0to0.5(thisgivesanextra50%chance ofhavingananddecreasesthenursesfrom1to0.5.Theregressionrecipe againmisinterpretsthisaidascostlyharmingorspite( B< 0 ;C> 0)[1]. 21 thealtruists,thebasanindirectcomponentthatsocialpartners bythefocalindividual'sphenotype,andthecotofrelatednessasanadjustment betweenthesetwocomponents[100].Misunderstandinginclusivenesstheoryistheresult ofdistortionofHamilton'srule[100].Hamilton'soriginalformulationintermsofdirect andindirectcausesprovidesausefulpartitionthatworksforsimplebiologicalprocesses. Ontheotherhand,themodernformsofthistheoryprovidemorecomprehensivecausal analysistothesocialbehaviorthatincludesmoredirectandindirectcomponentstothe Inthemoderninclusivetheorycostsandbarecontextdependent, andcotsofrelatednessintheirgeneralizedformtranslateallcomponentsinto commonunitsoftotalusingmultipleregressiontechniques[100].However,itisworth mentioningthatthelimitationsofthistheorywithitscausalanalysisarereasonablywell known.Thecausalanalysesyieldtoenhanceunderstandingratherthanalternativestomore complexanddetailedmathematicalanalysesofparticularproblems[100]. 2.3Evolutionofcooperationinstochasticgames Peoplewillmostlikelyacttlyintsituations[25].Thesituationcouldbe thestateoftheenvironment,thepartnersattitude,ortheavailabilityofresources,etc. Therefore,theactionofanindividualisarandomvariableconditionedonenvironmental parameters.Usingthisidea,wecanextendthestandardgametheorytoconditionalgames ormoregenerallytostochasticgames.Iterativegamesaremoreinterestingsinceineach interaction,playersgaininformationaboutthevisitedopponents.Thisinformationcan beexploitedforthenextroundsbyactingconditionallyonthisgainedinformation.In evolutionarygametheory,conditionalstrategieshaveshownagreatpotentialinsurviving 22 situations.Itisobviousthattohaveconditionalstrategy,individualsshouldhave informationaboutthestateofthegameandmemorytosavethisinformationtorecallfor futureactions.Conditionalstrategyofaplayercouldbearandomvariableofitsprevious actions,theopponent'spreviousactions,receivedpayoracombinationofthem.For example,\TitForTat"(TFT)isastrategyconditionedontheopponent'spreviousaction; thusitisonlyneedtohaveoneunitofmemory(fortwoplayergamesthisunitisone bit).Althoughthisstrategyisverysimpleanddefeatedallofitscomplicatedstrategiesin Axelrod'sfamoustournamentandscoredextremelywell,ithasan\Achillesheel"[4,5]. Inanoisyanderror-pronearenainwhichishardforplayerstoobservetheopponent's action,asingleerrorbetweentwoTFTplayersleadstoasequenceofmutualrecriminations, whichcanonlybebrokenbyanothermistake.Insuchenvironment,theperformanceof TFTreducesalotandtwoTFTplayersactastwoAllDefectors(AllD).Therefore,to solvethisproblemitisgoodtoaddforgivenessandgenerositytotheplayers.Thenew GenerousTFT(GTFT)ismoreresistanttothenoiseandhasabetterperformanceina noisyenvironmentcomparetostandardTFT.TosolveTFTprobleminnoisyenvironment, NowakandSigmundalsoproposedasimilarstrategythatperformswellinnoisyenvironment [15].ThisstrategyisWin-StayLose-Shift(WSLS)strategy.Inthisstrategy,playeracts inthenextroundthesameifhispayishigherthanagiventhresholdandswitchesto theotherstrategyotherwise[15].SimilartoTFT,thisstrategyneedsoneunitofmemory tosaveitsownpreviousaction.TheactionofplayerwithWSLSstrategyisconditioned directlyonitspreviousactionandindirectlyonitsopponent'saction,sincethepayof theplayerisafunctionofbothplayers'actions.Theperformanceofsimpleconditional strategiessuchasTFT,GTFTandWSLSencouragedbiologiststoexplaintheevolution ofaltruismby reactivestrategies orreciprocalinteractionsbasedonrepeatedencounters 23 anddecisionmakings.Sincenaturalinteractionscomparetointeractionsbetween computerprogramsisfullofuncertainty,thereactionsofindividualsaregenerallystochastic: theycooperatewithaprobabilityconditionedontheirownandopponents'previousactions [105].Anotherreasonthatmakestheinteractionofbiologicalorganismsstochasticisthe populationsize.Inpopulationdynamicsandstandardevolutionarygametheorywhenthe interactionbetweenindividualsismodeledbyreplicatorequation,thepopulationsizeis assumedtobeite;however,biologicalsystemsaresubjecttoallsortsofotherrandom andareinherentlystochastic.Forexample,noteveryencounterforaspecialtypeof individualhasthesameonss.Moreover,thebiologicaldynamicsarectedby manyotherrandomsourcessuchasvariabilityinmatingsuccess,foragingsuccess,infant mortality,etc.Thisstochasticityiswidelyacceptedbyscientists,usuallyitisignoredand notincludedinthebiologicalmodelssinceitisassumedthattheofthisstochasticity isnegligible[106].Whileignoringthisstochasticitymightbereasonableinsomesituations, itisshownthat,ingeneral,eveniftheofthisstochasticityarearbitrarilysmall,they mayqualitativelychangetheasymptoticbehaviorofthesystem[107];thus,theyshould beincludedinthemodel,andonewaytoconsiderthisstochasticityistousestochastic evolutionarygamesinwhichindividualsactcompletelyconditionallyontheirstateand theiropponents'previousactions. LloydStowellShapleyintroducedstochasticgamesin1953asadynamicalgamewith probabilistictransitionsplayedbyoneormoreplayers.Similartostandardgames,stochastic gamesaresequential.Inthiskindofgames,playersselectactionsandreceiveapaybased onthecurrentstateandthechosenactions.Thegamethenmovestoanewrandomstate, whosedistributiondependsonthepreviousstateandtheactionschosenbytheplayers,and thisprocedureisrepeatedforaornumberofstages.Similartothestandard 24 games,thetotalpaytoanindividualisusuallythediscountedsumofthestagepay orthelimitinferioroftheaveragesofthestagepay[108].Twoplayersstochastic gamesarewidelyusedinevolutionarygametheorytomodelpopulationsizedynamics [18,23,109,110,111,112,113]. Tomodelspeciesinteractioninpopulation,stochasticgamesareusuallycom- binedwithgraphicalmodels,calledstochasticevolutionarygraphicalgame.Thesegraphical modelshavebeenusedtoanalyzediscretesystemsthatoperateonstochastic(adversarial) environments[74].Inevolutionarygraphtheorypossiblerationsofasystemand itsenvironmentarerepresentedasvertices,andthetransitionscorrespondtointeractions betweenindividualsthatchangethestateofthesystemortheenvironmentarerepresented asedges.Arunofthegamecorrespondstoanpathinthisstochasticgraph.Similar tostandardgames,inmanycases,thereexistsastablestochasticstrategyoranequilibrium point;however,optimalstrategiesforbothplayersmaynotexist. 2.4Altruisminmultiplayergame The\tragedyofthecommon"isasocialdilemmainwhichmembersofagroupseekingthe maximumself-interestbyexploitingapublicgood,andbydoingsotheyoftenharmtheir ownandothermembers'long-terminterest[114,115].Asoftoday,tragedyofthecommon isnotonlywell-knowninsocialsciencesandpolitics[114]butalsoplaysanimportantrole tomodelandstudytheevolutionofsocialbehaviorinbiology[116,117,118,119].Oneway tostudyandmodelsuchsocialdilemmasinanimalbehaviorisevolutionarygametheory, whichdescribespairwiseorgroupwiseinteractionsofapopulationwithpayfor tstrategies,andamongfamousgamesinevolutionarygametheory\publicgoods 25 game"hasbeenmostcommonlyusedtodescribesocialdilemmassimilartothetragedyof thecommon. Inexperimentaleconomicsthepublicgoodsgameisaclassiclaboratorystandardfor studyingcollectiveactionproblems[120,121,122,123].Inthisgame,membersofthe groupdecidetoinvesttheamountoftheirpossessedtokensintoapotorcommongood. Then,thetotalcontributedtokensinthepotaremultipliedbyafactor(called\synergy factor"),whichisgreaterthanoneandlessthanthenumberofplayersinthegame.Atthe end,thispublicgoodpayisequallydistributedamongtheplayersinthegroup,nomatter whethertheyinvestedtheirtokenintothecommongoodornot.Ideally,thegroupexploits maximalinvestmentfromthecommongoodifeverybodycontributesthemaximumamount andtakeadvantageofthesynergy;however,thisstrategyisvulnerabletotheexistenceof defectorsor\freeriders",thosewhoreceivesharefromthecommongoodyetdonotinvest theirtokens.Itcanbeeasilyshownthatinthisdilemma,therationalNashequilibrium istheself-interestedstrategybecausenotcontributinganythingclearlydominatesallother playersregardlessoftheirstrategy. Interestingly,inrealitywhatisobservedfrequentlyisnottherationalNashequilibrium, andpeopletendtosharetheirtokensintothecommongood.Theactuallevelsofcontribution foundisvariable[124],andtheaveragecontributiontypicallyisafunctionofthesynergy factor[125].Severalsolutionshavebeenproposedtoaddressthisdilemma.Forexample Caprarosuggestedasolutiontothisdilemmabasedontheideathatmembersofthegroup canforecastifactingcooperativelypays,andthentheycooperateinaratethatdependson theirforecast(similartoquorumsensingmechanisminmicrobialcommunities).According tohismodel,increasingthesynergyfactorwillincreaselevelofcooperation[126].Hardin inhispapersuggestedthatthetragedyofthecommonscanonlybeavoidedifdefectors 26 orfreeridersreceivepunishmentfornotparticipatingintheinvestment[114],andsince thenpunishmenthasshownitspotentialinmaintainingcooperationinpublicgoodsgame [127,128,129,130,131,132,133].Ontheotherhand,punishingcooperatorsormoralists [133]donotperformverywellinwell-mixedpopulationssincetheylosethecompetition againstthenon-punishingcooperators,orsecondarydefectors,sincemoralistsacquirean additionalcostforpunishingdefectors,andasaresult,defectorscaninvadethepopulation ofnon-punishingcooperators[115].Therefore,thereshouldbeotherwaysthatstabilize cooperationsincepunishmentisacostlyprocess[8].Dreberetal.showedintheirexperiment thatalthoughtheoptionofcostlypunishmentincreasestherateofcooperation,punishment doesnotaltertheaveragepayofthegroup.Moreover,theydiscoveredastrongnegative correlationbetweentotalpayanduseofcostlypunishment.Thisobservation thatthosepeoplewhowinthecompetitionandreceivethehighesttotalpaytendnotto usethecostlymaladaptivepunishment[8].Therefore,theremightbeotherreasonsthat costlypunishmentstillexistsoverthecourseofevolution.Dengandchuintheirpaper proposedthatthenon-linearityofthestructuresofpublicgoodsmightbetheforcethat maintainscooperationwithinthewell-mixedpopulations[134].Wakanoetal.foundthat thespatialevolutionarydynamicsofecologicalpublicgoodsinsystems mightpromotecooperation[135]. Theresearchonsocialbehaviorisstillgoinganddtversionsofpublicgoodsgame playimportantrolesinthis 27 2.5Altruisminspatialgames Inevolutionaryspatialgames,playersplaythegamewiththeirneighborsinan arena.Evolutionaryspatialgamesconcerntheevolutionoftstrategiesoverspace [47].DespitehavingJohnvonNeumannasthesamefather,thetheoryofspatialgames andthetheoryofcellularautomata,atglance,seemtobetotallyunrelated;however, afterrecentdevelopmentinboththesetwouncorrelateddisciplineshavebeenmerged together[136].Initeratedevolutionaryspatialgames,theplayersaredistributedonatwo orhigherdimensionalarray,calledarenaofthegame.Then,ineveryroundeveryplayer playsthegameagainstitsimmediateneighbors.Basedonthepayofplayers,eachsiteis occupiedbytheoriginalowneroroneofitsneighbors.Severalstandardevolutionarygames havebeenextendedtospatialgames,andinmanyofthesecases,cooperationemergesin gamesinwhichisnotpossibletomaintaincooperationintheirstandardforms[2,25,27, 66,132,133,135].Moreover,itisseenthatthespatialgreatlychangetheresult ofthefrequency-dependentgamedynamics,andasaresult,onecanmanyfascinating mathematicalphenomenasuchasspatialchaos,fractalpatterns,andkaleidoscopelikeshapes [2,27,74].Forexample,NowakandMayshowedthatSpatialIteratedPrisoner'sDilemma (S-IPD)withthefollowingpaymatrix: E = 0 B @ CD C 10 Db 1 C A b> 1 ; 0 < ˝ 1(2.4) depictsspatialchaosif1 : 8 1 : 8, andasaresult,bigclustersofdefectorsexpandinthearenauntiltheyconquerthearena. 28 Ontheotherhand,cooperatorhasadvantagetothedefectorif b< 2,andbigclustersof cooperatorsexpandinthearena.Chaoticpatternsemergefor1 : 8 R>P>S Forexample,Axelrodusedthefollowingsetofvaluesinhistournament[4]: R =3 ;S = 1 ;T =5 ; and P =0 Weusethesamepayhere,andbasedonthesepaythepaymatrixforIPDgame is: E IPD = 0 B @ 30 51 1 C A (4.19) whichmakesdefectionanevolutionarystablestrategy.Thismeansthatafterdefectionis establishedinthepopulation,anyinvasionofcooperation,resultedbymutation,willbe unsuccessful.Byapplyingthenecessitycondition(4.18)tothisgame,wehave: 1+(4 5+0) p i 0 ) p i 1 whichmakesitimpossibletomakecooperationevolutionarystable. 107 4.2Stabilityanalysisofpointsinunconditional games:areview Oneotherequivalentwaytoselectabilityanalysisofstrategiestoinvestigatethefaithof cooperationinagivengameistoderivethedynamicsthatrulethevariationofstrategies overthecompetition.Then,bytheequilibriumpointsofthedynamics,whichare usuallynonlinearordinarytialequations,wecanchecktheirstabilityproperties.The evolutionarygamesofwell-mixedpopulationsgenerallyhavereplicatordynamics.Inthis section,replicatorequationisappliedtocheckthestabilityofpossibleequilibriumpointsin unconditionaltwoplayergames.Moreover,theequivalencyofreplicatorequationandthe \Priceequation"initssimplerformisgiven.Replicatorequationhasbeenusedtodescribe thevariationofstrategies'concentrationinagivenpopulation.Ifthepopulationisvery large,andifthegenerationsblendcontinuouslyintoeachother,wecanassumethatthe increaserate_ p i =p i isafunctionofitsevolutionarysuccess.Therefore,wehave: _ p i p i = w i w (4.20) Thisyieldstothereplicatorequationas: _ p i = p i ( w i w )(4.21) 108 Ifthepaymatrixofagameisanti-diagonalas: E = 0 B @ CD C 0 a Db 0 1 C A (4.22) thereplicatorequation(4.21)iswrittenas: _ p i = p i (1 p i ) ( b + a ) p i + a (4.23) Bydoingsimplematrixoperation,suchascolumn-wiseoperationsthatdonotchangethe gameproperties[36],thepaymatrix(4.2)canbewrittenasthefollowinganti-diagonal matrix: E = 0 B @ CD C 0 S P DT R 0 1 C A (4.24) Therefore,thereplicatorequationforthisgamewouldbe: _ p i = p i (1 p i ) ( S P ) ( S P + T R ) p i (4.25) InappendixCweshowthattheright-handsideofthisequationisthecovariancebetween genotypeandwhichistheselectionforceinthe\Priceequation"insteadystate. 109 Thus,wehave: _ p i = cov ( g;w )= p i (1 p i ) ( S P ) ( S P + T R ) p i (4.26) Thereplicatorequationofunconditionaltwoplayersgameshasthreeequilibriumpoints.To checkthestabilityofeachequilibrium,weshouldtakethederivativeofthisequationrespect to\ p i andcheckthesignofthisderivativeineachequilibrium.Wehave: d _ p i dp i =(1 p i ) ( S P ) ( S P + T R ) p i p i ( S P ) ( S P + T R ) p i ( S P + T R ) p i (1 p i ) Defector( p i =0)isastableequilibrium,if: P>S Cooperator( p i =1)isastableequilibrium,if: R>T Mixedstrategy p i = ( S P ) ( S P + T R ) isstableequilibrium,if: ( S P )( T R ) S P + T R > 0 110 Figure4.1:Phasediagramofreplicatorequationforiteratedprisonerdilemma. 4.2.1Iteratedprisonersdilemma Forthisgame,thereplicatorequation(4.25)is: _ p i = p i 1 p 2 i (4.27) Inthisgamewehavetwoequilibriumpoints,andamongthesetwopoints p i =0isevolu- tionarystable.Figure(4.1)showsthephasediagramofthisgame.Accordingtothis itisobviousthatdefectorisanevolutionarystablestrategy. 4.3Evolutionofcooperationinconditionalgames Instandarditeratedgames,playersplayunconditionally.Ifplayersdonothaveinformation abouteachother,orthereisnotanyenvironmentalrelatednessorclusteringinthepopula- tion,playersplayunconditionally,anditishardtoevolvecomplicatedcooperativebehavior outofthesestandardgamesinwhichusuallydefectionisconquerorofthecompetition.Many 111 researcherstriednewideastoaddressevolutionofcooperationandaltruismbyintroducing externaltermstotheseiteratedgames[7,17,19].Theseapproacheshaveatleastonething incommon:toevolvecooperationinthepopulation,thereshouldbeatleastaminimum levelofcommunicationinwhichinformationisexchangedbetween,whichasaresultwill increasetheleveloftrustbetweenindividuals.Thisinformationcanbeabouttheplayers' previousactionsorthestateofthegame.Inthissection,weanalyzetheperformanceof conditionalstrategiesintwoplayersconditionalgames.Todoso,weintroduceassortment term\ q ",astheprobabilityofcooperationwhenacooperatorfacesadefectororviceversa. Inotherwords,assortment q canbeseenasonestrategyserrorrateinrecognizingtheother type,oritcanbethefaithrateofonetypetotheothertype.Here,wealsoinvestigate theroleofcommunicationandinformationexchangebetweenplayerswiththeseconditional strategiesoverthecourseofevolution.Itisworthmentioningthatforanygivenstrategy q is acontinuousparameter.Thus,thecooperationrateofacooperatorwithanassortment term q =0 : 5whenfacingadefectorisandequalsto50percent. Similartotheprevioussection,weusethepaymatrixgiveninequation(4.2)forthese conditionalgames.Forsimplicity,weconsideragamebetweenaconditionalcooperative strategyandanunconditionaldefector.Inotherwords,nomatterwhattheopponents strategyis,defectorsalwaysdefect,andprobabilityofcooperationforconditionalcooperative strategy,whileinteractingwithacooperator,isoneandis q ,whileinteractingwitha defector.Later,wediscussfullystochasticgame.Wecansummarizetheplayers'strategies 112 (onesemi-conditionalandtheotherpurestrategy)inthefollowing\strategymatrix": Z = 0 B @ CD C 1 q D 00 1 C A (4.28) Theelementsofthismatrixrepresenttheprobabilityofcooperationinanytypeofinter- action.Accordingtothisstrategymatrix,wecancalculatetheprobabilityofeachtypeof interaction.Byusingtheelementsofstrategymatrix,wecancalculatetheprobabilityof anytypeofinteractioninapopulationinwhichtheconcentrationofconditionalcooperative strategyis p i .Wecansummarizetheseprobabilitiesisanewmatrix,called\GameMatrix" as: G = 0 B @ CD Cp 2 i p i (1 p i ) q Dp i (1 p i ) q (1 p i ) 2 +2 p i (1 p i )(1 q ) 1 C A (4.29) Consideringtheseprobabilitiesofinteractions,foragivenpopulation,wecantreat ofeachstrategyandeachoftheseinteractionsasarandomvariableandcalculatetheprob- abilityandvaluesofeachrandomvariableduringroundsofconditionalgame.Table(4.2) summarizestheprobabilitiesandvaluesoftheserandomvariablesintheseconditionalgames. Asweknow,theconditionsthatfavorcooperationintwoplayersgameare: cov ( g;w ) > 0 ( S P )+( T P ) cov g 0 ;z cov ( g;z ) +( R S T + P ) cov g 0 ;zz 0 cov ( g;z ) 0(4.30) 113 Table4.2:Listofrandomvariablesintheconditionaliteratedgame. G i Probability gg 0 zz 0 P z P z 0 CC p 2 i 1111 RR DD (1 p i ) 2 0000PP CD p i (1 p i ) q 1010 ST DD p i (1 p i )(1 q )1000PP DC (1 p i ) p i q 0101 TS DD (1 p i ) p i (1 q )0100PP Thesenecessityconditionsareequivalent.Thesecondconditionjusthastheinclusive formulationofthecondition.Theequivalencyoftheseconditionsisgiveninappendix C. Forthisconditionalgame,theofeachstrategy(conditionalcooperativestrategy andpuredefection)canbewrittenas: w C = p i p i R + q (1 p i ) S +(1 q )(1 p i ) P w D =(1 p i ) p i qT +( p i (1 q )+(1 p i )) P (4.31) Bypluggingtable(4.2)intothenecessityconditions(4.30),wehave: cov ( g;w )= p i ( R + P +( S + T 2 P ) q ) p 2 i ) + p i ( R P +(3 P 2 S T ) q ) p i +( S P ) q = 0 z }| { p i (1 p i ) ( R P ( S + T 2 P ) q ) p i +( S P ) q Therefore,thenecessityconditionthatfavorsconditionalcooperativestrategyis: ( R P ( S + T 2 P ) q ) p i +( S P ) q> 0(4.32) 114 Figure4.2:Isoclineanddomainofattractionforconditionalcooperationanddefection strategiesinIPD. 4.3.1Conditionalcooperationiniteratedprisonersdilemma BypluggingthepaymatrixofIPDintothenecessitycondition(4.32),conditionalcoop- erativestrategyisastablepointifandonlyif: (2 3 q ) p i >q (4.33) or: q< 2 p i 1+3 p i (4.34) Figure(4.2)showstheregioninwhichtheconditionalcooperativestrategyisstable.According tothisitisobviousthatif q 2 [0 : 5 ; 1],defectionisevolutionarystablestrategy,andit resistsagainstanyintroducedconcentrationofconditionalcooperativestrategywith q> 0 : 5. However,ifweletconditionalcooperativestrategieshavelowerassortmentvalues,defection 115 cannotbeevolutionarystablestrategyanymoresinceinthisgame,averylowconcentration ofconditionalcooperativestrategywithalowassortmenttermisenoughtoinvadeapopula- tionofpuredefectors.Infact,ifmutationintroducesaconditionalcooperativestrategywith assortment q =0,thisstrategywillconqueranypopulationofdefectors.Thisstrategyisthe evolutionarystablestrategyofthisgame.Ifassortment q islessthan0.5,bothconditional cooperativestrategyandpuredefectionstrategyareresistanttoinvasionoftheopposite strategy,introducedbymutation(verylowconcentration);thus,inthiscaseIPDbetween conditionalcooperativestrategyandpuredefectionissimilartocoordinationgameinwhich thewinnerofthegamewillbetheonewithhigherinitialconcentration.Insuchconditions, thedomainofattractionbetweenthesestrategiesisafunctionofassortment.Thelowerthe assortmenttermis,thebiggerthedomainofattractionofconditionalcooperativestrategy wouldbe. 4.4Evolutionofcooperationinconditionalgames:sta- bilityanalysis Here,weapplyreplicatorequationasanequivalentapproachtoselectabilityanalysisto checkthestabilityofpossibleequilibriumpointsintheseconditionalgames.Usingthis approachgivesusbetterunderstandingofassortment'sonthegamedynamicandthe stabilityofpoints.Consideringtheconditionalcooperativestrategyandpuredefection 116 strategy,wecanwritetheconditionalpaymatrixas: E = 0 B @ CD CRqS +(1 q ) P DqT +(1 q ) PP 1 C A (4.35) Bydoingsimplematrixoperation,wecanconvertthepaymatrix(4.35)toananti-diagonal matrixas: E = 0 B @ CD C 0( S P ) q D ( P R )+( T P ) q 0 1 C A (4.36) Therefore,thereplicatorequationforthisgamewouldbe: _ p i = p i (1 p i ) ( R P )+ q ( S +2 P T ) p i + q ( S P ) (4.37) whichisidenticaltothecovariancebetweengenotypeand(ortheselectionforcein the\Priceequation").Bytakingthederivativeofthisequationrespecttotheconcentration ofconditionalcooperativestrategyinthepopulation,wehave: d _ p i dp i =(1 p i ) ( R P )+ q ( S +2 P T ) p i + q ( S P ) p i ( R P )+ q ( S +2 P T ) p i + q ( S P ) +[( R P )+ q ( S +2 P T ) p i (1 p i )(4.38) 117 Defection( p i =0)isevolutionarystablestrategyifandonlyof: ( S P ) q< 0 Conditionalcooperativestrategy( p i =1)isevolutionarystableifandonlyif: ( T P ) q< ( R P ) Mixedstrategy p i = ( q ( S P )) ( P R )+( S 2 P + T ) q isstablepointifandonlyif: q ( S P )(( P R )+( T P ) q ) ( P R )+( S 2 P + T ) q > 0 4.4.1StabilityofconditionalcooperativestrategyinIPD Consideringtheconditionalandpurestrategiesofcooperatoranddefector,forthisgame thereplicatorequation(4.37)wouldbe: _ p i = p i (1 p i )((2 3 q ) p i q )(4.39) Inthisgame,wecanhavetwotothreeequilibriumpoints.Figure(4.3)showsthephase diagramofthisgamefort q .Accordingtothisbothconditionalcooperator andpuredefectorcanbeevolutionarystable.Defectorisevolutionarystablestrategyif onlyif q 0 : 5.Conditionalcooperativestrategycanbeastablepoint,ifandonlyif q< 0 : 5.Inthisgame,mixedstrategycannotbeevolutionarystable.Forsmallassortment terms( q< 0 : 5),theonlyevolutionarystablestrategyisconditionalcooperativestrategy withassortment q =0. 118 Figure4.3:Phasediagramofreplicatorequationforconditionaliteratedprisonerdilemma. blue: q =0orcompleteconditionalgame,red: q =0 : 2,green: q =0 : 4,lightblue: q =0 : 6, yellow: q =0 : 8,andblack: q =1 : 0orunconditionalgame. 4.5Communicationandinformationexchangeincon- ditionalgames Inthissection,wetrytoquantifytheamountofinformationthatconditionalcooperator gainsabouttheopponent'stypeduringthegame.Moreover,byaddingnoiseintothe communicationchannel,wetrytoanalyzetheofcommunicationnoiseonfateof thegame.Finally,weconsiderconditionalstrategyforbothcooperationanddefection. Conditionaldefectivestrategiesonaveragehavelowerpayincomparisontothepure defectorstrategyinoneinteraction;however,thislowerpaypaysbackwhentheygain morepayinlongrunbyconfusingvigilant/conditionalcooperators.Ourhypothesisis thatinsuchcasesbothstrategiescanco-exist. 119 4.5.1Communicationbetweenconditionalcooperatorandpure defectorinnoiselessenvironment Here,wehavetworandomvariables:actionofconditionalcooperatorandopponent'stype. Thestrategymatrixofconditionalcooperatorandpuredefectorcanbewrittenas: S C = 0 B @ CD Cp C j C p C j D Dp D j C p D j D 1 C A = 0 B @ CD C 1 q D 01 q 1 C A (4.40) where p C j C isprobabilityofcooperationforaconditionalcooperatorwhenitinteractswith anotherconditionalcooperator, p C j D isprobabilityofcooperationforaconditionalcoop- eratorwhenitinteractswithapuredefector,where p D j C isprobabilityofdefectionfora conditionalcooperatorwhenitinteractswithanotherconditionalcooperator,and p D j D is probabilityofdefectionforaconditionalcooperatorwhenitinteractswithapuredefector. S D = 0 B @ CD Cp C j C p C j D Dp D j C p D j D 1 C A = 0 B @ CD C 00 D 11 1 C A (4.41) where p C j C isprobabilityofcooperationforapuredefectorwhenitinteractswithacon- ditionalcooperator, p C j D isprobabilityofcooperationforapuredefectorwhenitinteracts withanotherpuredefector,where p D j C isprobabilityofdefectionforapuredefectorwhen itinteractswithaconditionalcooperator,and p D j D isprobabilityofdefectionforapure defectorwhenitinteractswithanotherpuredefector.Basedonthesestrategymatrixes,the 120 actionprobabilitiesofconditionalcooperatoris: P C C = p i + q (1 p i ) P D C =(1 q )(1 p i )(4.42) Consequently,byusingequation(4.42)theentropyofconditionalcooperator'sactionis: H ( A C )= p i + q (1 p i ) log p i + q (1 p i ) (1 q )(1 p i )log (1 q )(1 p i ) Similarly,theconditionalentropyofactiongivenopponent'stypeis: H A C O = X j 2f C;D g p j H A C O = j = (1 p i ) q log q +(1 q )log(1 q ) (4.43) Therefore,themutualinformationbetweenactionofconditionalcooperatorandtypeof opponentis: I ( A C ; O )= H ( A C ) H A C O = p i + q (1 p i ) log p i + q (1 p i ) (1 q )(1 p i )log (1 q )(1 p i ) +(1 p i ) q log q +(1 q )log(1 q ) 121 Figure4.4:ofassortmentonmutualinformationbetweenactionofconditionalco- operatorandopponenttype.X-axisisassortmentterm( q ),Y-axisistheconcentrationof conditionalcooperationinpopulation,andZ-axisismutualinformationbetweenactionof conditionalcooperatorandopponenttype. Figure(4.4)showsthismutualinformationfort q andconcentrationofconditional cooperator. 4.5.2Communicationbetweenconditionalcooperatorandpure defectorinnoisyenvironment Tomodeltheofenvironmentalnoiseontheactionofconditionalcooperator,we thefollowingerrormatrix: = 0 B @ CD C 1 D 1 1 C A (4.44) 122 Errormatrixshowshowconditionalcooperatorestimatesthefrequencyofeachtypeduring thegame.Byapplyingthiserrormatrixtotheactualconcentrationofplayers,theestimated concentrationsbyconditionalcooperatorwouldbe: 0 B @ ^ ˆ C ^ ˆ D 1 C A = 0 B @ 1 1 1 C A 0 B @ ˆ C 1 ˆ C 1 C A = 0 B @ + ˆ C 2 C 1 ˆ C +2 C 1 C A (4.45) Bypluggingestimatedconcentrations(4.45)intoequation(4.4),wecancalculatethemu- tualinformationbetweenactionofconditionalcooperatorandopponent'stypeinnoisy environments.Figure(4.5)showsthismutualinformationfort q andconcentration ofconditionalcooperatorinenvironmentswithtlevelsofnoise. 4.5.3Communicationbetweenvigilantcooperatoranddefectorin conditionalgames Inthissection,wewanttoquantifythemutualinformationbetweenactionandtypeof opponents,ifbothtypesarevigilant.Thestrategymatrixofconditional/vigilantcooperator anddefectorcanbewrittenas: S C = 0 B @ CD Cp C j C p C j D Dp D j C p D j D 1 C A = 0 B @ CD C 1 q C D 01 q C 1 C A (4.46) 123 Figure4.5:ofassortmentandenvironmentalnoiseonmutualinformationbetween actionofconditionalcooperatorandopponenttype.X-axisisassortmentterm( q ),Y-axisis theconcentrationofconditionalcooperationinpopulation,andZ-axisismutualinformation betweenactionofconditionalcooperatorandopponenttype:(a)onepercentnoise,(b)two percentnoise,(c)epercentnoise,(d)tenpercentnoise,(e)twentyepercentnoise,and (f)ypercentnoise. 124 S D = 0 B @ CD Cp C j C p C j D Dp D j C p D j D 1 C A = 0 B @ CD Cq D 0 D 1 q D 1 1 C A (4.47) Basedonthesestrategymatrices,theactionprobabilitiesofconditionalcooperatorand defectorare: P C C = p i + q C (1 p i ) P D C =(1 q C )(1 p i )(4.48) P C D = p i q D P D D =1 p i q D (4.49) Consequently,byusingequations(4.48)and(4.49),wecancalculatetheentropyofplayers' actionsas: H ( A C )= p i + q C (1 p i ) log p i + q C (1 p i ) (1 q C )(1 p i )log (1 q C )(1 p i ) H ( A D )= p i q D log p i q D p i (1 q D )log p i 1 q D (4.50) 125 Similarly,theconditionalentropyofactiongiventypeofopponentsforeachplayeris: H A C O = X j 2f C;D g p j H A C O = j = (1 p i ) q C log q C +(1 q C )log(1 q C ) (4.51) H A D O = X j 2f C;D g p j H A D O = j = p i q D log q D +(1 q D )log(1 q D ) (4.52) Therefore,themutualinformationbetweenactionofvigilantplayersandtypeofopponent is: I ( A C ; O )= H ( A C ) H A C O = p i + q C (1 p i ) log p i + q C (1 p i ) (1 q C )(1 p i )log (1 q C )(1 p i ) +(1 p i ) q C log q C +(1 q C )log(1 q C ) I ( A D ; O )= H ( A D ) H A D O = p i q D log p i q D p i (1 q D )log p i 1 q D + p i q D log q D +(1 q D )log(1 q D ) 126 Figure(4.6)showsthismutualinformationfort q andconcentrationofconditional cooperator.Similartoprevioussection,wecanapplytheerrormatrixtoestimatethe concentrationofplayers,andbypluggingtheseestimatedvaluesintoequations(4.53)and (4.53)wecancalculatethemutualinformationbetweenactionofvigilantplayersandtype ofopponent. 4.6ChapterSummary Inthischapter,evolutionarygametheorywasappliedtotwoplayersgamesforinves- tigatingtheconditionsinwhichcooperationemergesamongindividuals.Tostudy theevolutionofcooperationinunconditionalgames,weappliedtheQueller'sinclusive nesstheorywhichusesthe\Priceequation".Moreover,thepopulationdynamicsofthese gamesweremodeledbyreplicatorequation,andtheresultstheinclusive theory.Secondly,byintroducingconditionalcooperativestrategy,weinvestigatedtheemer- genceofcooperationovertheconditionalgames.Weextendedtheinclusivenesstheoryto conditionalgamesandderivedthenecessityconditionsformaintainingcooperationinsuch conditionalgames.Then,weinvestigatedthestabilityofthepossibleevolutionarystable strategiesbyusingreplicatorequation.Interestingly,ourobservationsshowemergenceof cooperationinsituationsinwhichisnotpossibletohavecooperationiftheplayersplay unconditionally.Finally,wequanthecommunicationrateandinformationexchange betweenindividualsduringtheseconditionalgamesbycalculatingthemutualinformation betweenactionsofvigilantplayersandtypeofopponents.Wealsoanalyzedhowassortment andenvironmentalnoisethiscommunication. 127 Figure4.6:ofassortmentonmutualinformationbetweenactionofvigilantplayers andopponenttype.X-axisisassortmentterm( q ),Y-axisistheconcentrationofconditional cooperationinpopulation,andZ-axisismutualinformationbetweenactionofconditional cooperatorandopponenttype:(a)mutualinformationbetweenactionofvigilantcooperator andopponenttype(b)mutualinformationbetweenactionofvigilantdefectorandopponent type 128 Chapter5 EvolutionofSuper-reciprocityin IteratedTrustDilemma(ITD) Itwillhaveblood;theysay,bloodwillhaveblood. -Shakespeare,Macbeth Oliver: Irememberyou! Grocer: AndIrememberyoutoo.Nowgetoutofmystore andstayout! Oliver: Oh,don'tbelikethat.Letbygonesbebygones.Let's helpeachother.Youhaveabusiness,andwehaveabusiness. We'llsendpeopletoyourstore,andyousendpeopletoour store.Whatdoyousay? Grocer: YoumindyourbusinessandI'llmindmybusiness. NowgetoutbeforeIthrowyouout! TitforTat,C.Rogers,1935 Sofar,wediscussedwell-establishedtheoriesthatexplainhowevolution,whichfavors short-termbcanmaintaincooperationforlong-termbInthischapter,we explicitlydiscusstheroleofcommunicationinestablishingtrustandover\blind" faith.Wealsotrytoquantifytheminimumamountofinformationthatisnecessaryto exchangebetweenplayersforbuildingthetrust.Trustisbasedlargelyonproof,evidence, orrealfactsandisthebasicprincipleofknowledge-basedjudgment.Ontheotherhand, faithis\thematterofhope".Itisabeliefthatdoesnotrequireproof,evidence,ortangible facts.Here,weshowthathowplayersbuildtrustandbringcommitmentduringtheiterated gamebycommunicationandexperiencingthegame. IntheclassicalIteratedPrisonersDilemma(IPD),directreciprocityasaconditional strategyontheopponentsactionisaformofcommunicationleadingtoaltruisticbehav- 129 ior.However,ifplayersusestochasticstrategies,asecondaryformofcooperationinwhich playersalternateinreceivingtheTemptation(T)andSucker(S)rewardscanemerge(super- reciprocity).Thisreciprocalbehaviorwillbecomeevolutionarydominantifthereward(T) isincreasedaboveacertainthreshold,butisinherentlymoreriskythanprimarycoopera- tionsinceitreliesontrustbetweenplayersintwoconsecutiveiterationsofthegame.Here, weinvestigatehowtenvironmentalconditionssuchasmutationrate,environmental noise,andrewardTatheevolutionofreciprocity.Super-reciprocalstrategiesrelyon thesynchronizationoftwogenesandarethusmuchmoresensitivetoenvironmentalchanges thattheaccuracyofplayerspredictionofopponents'futuremoves.Wethatin- creasingtheenvironmentalnoiseormutationrateisdeleterioustosuper-reciprocity,while increasingTstabilizesitsevolution.Conversely,inenvironmentsthatarehighlypredictable andwherethereisnopayadvantagetoengageinreciprocalcooperation,basiccooperation viareciprocalcommunicationremainsthestrategyofchoice. 5.1IteratedTrustDilemma Instandarditeratedprisonersdilemmaofequation(4.2),if R< S + T 2 ,IPDgetsreplaced byanewgamepreviouslyreferredtoastheIteratedLiftDilemma(ILD)[150,151].Here, wecallthisgameas\IteratedTrustDilemma(ITD)",sincetomaintainthedominant strategythereshouldbetrustandcommitmentbetweenplayers.Playersinthisgame havestochasticstrategies.Playerschoosetheirnextmovebasedonthegameplayedin thepreviousround(Cooperationvs.Cooperation(CC),Cooperationvs.Defection(CD), Defectionvs.Cooperation(DC),andDefectionvs.Defection(DD))andtheprobabilitythat issavedintheirgenomeaspartoftheirstrategy.Iftwoplayersinteractforthetime, 130 theircooperationprobabilitydependsonanothergene( P C )intheirgenome.Thisgene indicateswithwhatprobabilitytheycooperateforthetimetheymeetanopponent. Thus,eachstrategycontainsegenes: ( P C ;P 1 ;P 2 ;P 3 ;P 4 )(5.1) where P C istheprobabilityofcooperationfortheinteraction, P 1 istheprobability ofcooperationifinthepreviousinteractionbothplayerscooperated, P 2 istheprobability ofcooperationifinthepreviousinteractionIcooperatedandmyopponentdefected, P 3 is theprobabilityofcooperationifinthepreviousinteractionIdefectedandcheatedonmy opponentwhilemyopponentcooperated,and P 4 istheprobabilityofcooperationifinthe previousinteractionbothplayersdefectedandcheatedoneachother.Forexample,ifthe genomeorthestrategyofoneplayeris: (0 : 2 ; 0 : 6 ; 0 : 05 ; 0 : 75 ; 0 : 35) thechanceofcooperationforthisplayerintheinteractionis20percent.More- over,thisplayerwillcooperatewiththeprobabilities0.6,0.05,0.75,and0.35inthenext roundofgame,ifthegamebetweenitsopponentinthecurrentroundiscooperation- cooperation,cooperation-defection,defection-cooperation,anddefection-defectionrespec- tively.Itisworthmentioningthatthestrategyofplayersisduringtheirlifetime.For instance,if P 1 geneofaplayeris0.4,thechanceofcooperationforthisplayerisalways40 percent,ifinthepreviousthisplayeranditsopponentbothplayedcooperatively.InITD, unlikeIPD,playersbyswitchingfromdefectiontocooperationandexchangingTrewards 131 withSrewardsmaximizetheirmeanThisstrategy,calledsecondarycooperation, isasuper-reciprocalstrategyandhasaperiodicpatterninwhichafteraplayerreceives theTrewardbycheatingonitsopponent,itwillreceiveanSrewardinthenextround bycooperatingandbeingcheatedfromtheopponent.Toengageinsecondarycooperation, playersshouldhaveastrategyas: ( P C ;x; 0 : 0 ; 1 : 0 ;y ) ,where x and y canbeanyarbitrarycontinuesvaluein[0 ; 1].Inthenextsection,we mathematicallyprovethatsuchstrategyisoptimalandtheplayerofthistypeofstrategy willhavethemaximummeanss,if R< S + T 2 .Now,thequestionifthisstrategyis evolvableinanevolutionarygameornot,andhowdorandomprocessessuchasmutation orenvironmentalnoisetheevolutionofsecondarycooperation?Itisobviousthat thesecondarycooperationisinherentlymoreriskythantheprimarycooperationsinceit reliesontrustbetweenplayersintwoconsecutiveiterationsofthegame(comparedtothe primarycooperationwhichhastodealwithasinglemove).Moreover,primarycooperation lessbywithdrawingfromthereciprocationofcooperativemovesincomparisonto thissecondarycooperation. Initialresearchonthisgamewaslimitedtothesimplerecognitionofsecondarycoop- erationasanalternativetoprimarycooperationordefectionstrategies[4]andhowerror correctioncanleadtoshortphasesofsecondarycooperationwithincooperativeIPDregimes [152].Delahayeetal.forthetimeinvestigatedthepropertiesofsecondarycooperation bothwithintheclassicalIPDpayscheme(R=3,S=0,T=5andP=1),byintroducing asecondarycooperatingcapablestrategycalled\Reason",aswellaswithintheITDpay 132 scheme(R=3,S=0, T> 6,andP=1)[151].Thestrategiesusedinthisstudyhada memorydepthequaltoonewitharandommoveanddeterministicmovesthereafter. Delahayeetal.limitedtheirstrategiestouptothreestrategiesatatimeandtheyused well-mixedpopulationintheiranalysis[151]. Here,wetakeatapproachtoDelahayetal.andevolvestrategiescapableof playingsecondarycooperationtoinvestigatetheconditionsthatmakesecondarycoopera- tionastableevolutionaryattractorinadditiontoprimarycooperationanddefection.We hypothesizethatthestabilityofsecondarycooperationdependsbothontheplayer'sability topredictfuturemovesaswellasthepayadvantagethatsecondarycooperationhaveover otherstrategies.Wealsoinvestigatetheoftemptationreward,environmentalnoise, andmutationrateonstrategydominancewithinanevolvingpopulationtodeterminethe parametersettingsthattriggertransitionsbetweenstrategies.Finallyweinvestigatethe rateofcommunicatedinformationbetweenplayersintwoconsecutiveroundofgamesto inwhatinformationrate,controlledbyenvironmentalnoise,thesecondaryandprimary cooperationcollapsetodefection. 5.2Secondarycooperationastheoptimalstrategyin ITD Beforewediscusstheexperimentsandouragentbasedmodeling,hereIdiscusstheoptimality ofsecondarycooperationinITD.Todoso,Iassumeahomogenouspopulationofagents, andItheoptimalstrategyforagivengame. 133 5.2.1Optimalstrategyamongagroupofplayersthatcompletely engageinsecondarycooperationplay Tocompletelyengageinsecondarycooperation,playersshouldhavethefollowinggenome: ( P C ;P 1 ; 0 : 0 ; 1 : 0 ;P 4 )(5.2) Thusagroupofsecondarycooperatorsshouldhaveagenepoolconsistsofgenesas(5.2). Here,theobjectiveistouseoptimizationtechniquetotheoptimalvaluesfor P C , P 1 ,and P 4 genesthatmaximizeofthesecondarycooperators.Todoso,assumegenesofsuper cooperatorsbe( q;x; 0 ; 1 ;y ).Gene q ,asweknow,istheprobabilityofcooperationinthe move.Consideringthisgenome,theprobabilityofeachtypeofgame( CC;CD;DC;DD )in theinteractionis: ˇ 1 (0)= ˇ CC (0)= q 2 ; ˇ 2 (0)= ˇ CD (0)= q (1 q ) ; ˇ 3 (0)= ˇ DC (0)= q (1 q ) ; ˇ 4 (0)= ˇ DD (0)=(1 q ) 2 (5.3) Asweknow,thepaymatrixforthisgameis: E = 0 B @ CD CRS DTP 1 C A (5.4) 134 Therefore,themeanaftertheinteractionoverthepopulationis: w (0)= Rq 2 +( S + T ) q (1 q )+ P (1 q ) 2 (5.5) Bytakingthederivativeof w (0)respectto q ,wecantheoptimalvaluefortheprobability ofcooperationintheinteraction.Usingthisidea,theoptimalvalueforthisprobability is: P C = S + T +2 P 2( S + T R P ) (5.6) Sincethistermappearsonceinthemeanofplayersafterroundsofiteratedgames,it doesnotthemeanofplayers;thus,asweseelater,overthecourseofevolution thistermvanishes,andmostlyplayersprefertodefectduringtheinteraction.Now, let'sassumethefrequencyofeachtypeofinteractionisknownafter\ t "roundsofgames. Wecanrecursivelycalculatethefrequencyofeachtypeofinteractionforthenextroundas: ˇ 1 ( t +1)= x 2 ˇ 1 ( t )+ y 2 ˇ 4 ( t ) ˇ 2 ( t +1)= ˇ 2 ( t )+ x (1 x ) ˇ 1 ( t )+ y (1 y ) ˇ 4 ( t ) ˇ 3 ( t +1)= ˇ 3 ( t )+ x (1 x ) ˇ 1 ( t )+ y (1 y ) ˇ 4 ( t ) ˇ 4 ( t +1)=(1 x ) 2 ˇ 1 ( t )+(1 y ) 2 ˇ 4 ( t )(5.7) Bylookingatequation(5.7),weseethatthefrequencyofCDandDCgamesincreaseover generations,andthesegamesactasattractorsingamespacesinceovergenerationsthese gamesareseenmoreoften;therefore,nomatterwhatthevaluesof x and y are,eventually 135 wejustseeCDandDCgamesinthepopulation,andthemeanofthepopulation convergesto: lim t ! + 1 w ( t )= S + T 2 (5.8) Infact, x and y justdeterminethespeedofgettingtotheattractorofthisgame.Theonly casesthatmakeCCandDDgamesstablearetheextremevaluesfor x and y (0and1).If bothofthesegenesarezero,wehaveoscillationsfromCCtoDDgames. 5.2.2Optimalstrategyamongahomogenousgroupofplayersin bothIPDandITDregimes Now,let'sassumethatthepopulationishomogeneouslywithplayerswith( q;x;v;z;y ) gene,wheretheprobabilityofcooperationinthemoveis q .Thisscenarioisquite tofouragentbasedmodelinginwhichpopulationisheterogeneousandt strategiesarecompetingwitheachothertoconquerthepopulation.Theoptimalprobability ofinteractionisidenticaltowhatwehadinthepreviouscase(equation(5.6)).Similar tothepreviouscase,giventhefrequencyofeachtypeofinteractionafter\ t "roundsofgame, wecanrecursivelycalculatethefrequencyofeachtypeofinteractionforthenextroundas: ˇ 1 ( t +1)= x 2 ˇ 1 ( t )+ vzˇ 2 ( t )+ vzˇ 3 ( t )+ y 2 ˇ 4 ( t ) ˇ 2 ( t +1)= x (1 x ) ˇ 1 ( t )+ v (1 z ) ˇ 2 ( t )+ z (1 v ) ˇ 3 ( t )+ y (1 y ) ˇ 4 ( t ) ˇ 3 ( t +1)= x (1 x ) ˇ 1 ( t )+ z (1 v ) ˇ 2 ( t )+ v (1 z ) ˇ 3 ( t )+ y (1 y ) ˇ 4 ( t ) ˇ 4 ( t +1)=(1 x ) 2 ˇ 1 ( t )+(1 v )(1 z ) ˇ 2 ( t )+(1 v )(1 z ) ˇ 3 ( t )+(1 y ) 2 ˇ 4 ( t ) (5.9) 136 ThisdynamiccanbemodeledasaMarkowprocesswiththefollowingtransitionmatrix: M = 0 B B B B B B B B B @ x 2 vzvzy 2 x (1 x ) v (1 z ) z (1 v ) y (1 y ) x (1 x ) z (1 v ) v (1 z ) y (1 y ) (1 x ) 2 (1 v )(1 z )(1 v )(1 z )(1 y ) 2 1 C C C C C C C C C A (5.10) ThestablestateofthisMarkovprocessistherighteigenvectoroftransitionmatrixthat correspondstoeigenvalue =1,whichwouldbe: V 1 = 0 B B B B B B B B B @ y ( y (1 v z )+2 vz ) (1 x )(1+ x v vx z ( x 2 v +1)) y (1+ y x ) 1+ x v vx z ( x 2 v +1) y (1+ y x ) 1+ x v vx z ( x 2 v +1) 1 1 C C C C C C C C C A (5.11) Sincethiseigenvectorshowsthedistributionofactionsafternumberofiterated games,itshouldsumuptoone;therefore,tohavethepropereigenvectorweneedtonor- malizethisvector: X ( 1 )= 0 B B B B B B B B B @ v 1 v 2 v 3 v 4 1 C C C C C C C C C A = 0 B B B B B B B B B B @ y ( y (1 v z )+2 vz ) 1 v +2 y z +2 vz + vx 2 vy 2 +2 xy 2 2 x 2 y + x 2 z y 2 z x 2 y 2 2 vxz +2 vzy y (1 x )(1+ y x ) 1 v +2 y z +2 vz + vx 2 vy 2 +2 xy 2 2 x 2 y + x 2 z y 2 z x 2 y 2 2 vxz +2 vzy y (1 x )(1+ y x ) 1 v +2 y z +2 vz + vx 2 vy 2 +2 xy 2 2 x 2 y + x 2 z y 2 z x 2 y 2 2 vxz +2 vzy (1 x )(1+ x v z vx xz +2 vz ) 1 v +2 y z +2 vz + vx 2 vy 2 +2 xy 2 2 x 2 y + x 2 z y 2 z x 2 y 2 2 vxz +2 vzy 1 C C C C C C C C C C A (5.12) 137 Themeanssofthepopulationafternumberofiteratedgamesis: w ( 1 )= Rv 1 + Sv 2 + Tv 3 + Pv 4 (5.13) Totheoptimalvalueofgenomethatgivesthemaximummeanweshouldhave: @ w @x = @ w @v = @ w @z = @ w @y =0 0 x 1 ; 0 v 1 ; 0 z 1 ; 0 y 1(5.14) 138 whichgivesus: 8 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > < > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > : 1+( v + z 2 y 1) x 2 +(2 x v z 1) y 2 +2(1 x + y ) vz +2 y v z : [ y (2 x y 2)( S + T )+2(( v + z 1) x + vz ) P ] 2 vx + y 2 2 xy + xz x vz [ y ( y (1 v z )+2 vz ) R + y (1 x )(1+ y x )( S + T )+(1 x )(1+ x v z vx xz +2 vz ) P =0 1+( v + z 2 y 1) x 2 +(2 x v z 1) y 2 +2(1 x + y ) vz +2 y v z : [(1+2 vz vy zy ) R +(1 x )(1+2 y x )( S + T )] 2 1 vy +2 xy x 2 yz y + vz [ y ( y (1 v z )+2 vz ) R + y (1 x )(1+ y x )( S + T )+(1 x )(1+ x v z vx xz +2 vz ) P =0 1+( v + z 2 y 1) x 2 +(2 x v z 1) y 2 +2(1 x + y ) vz +2 y v z : [ y (2 z y ) R +(1 x )(2 z x 1) P ] + 1 2 z x 2 + y 2 +2 xz 2 yz [ y ( y (1 v z )+2 vz ) R + y (1 x )(1+ y x )( S + T )+(1 x )(1+ x v z vx xz +2 vz ) P =0 1+( v + z 2 y 1) x 2 +(2 x v z 1) y 2 +2(1 x + y ) vz +2 y v z : [ y (2 v y ) R +(1 x )(2 v x 1) P ] + 1 2 v x 2 + y 2 +2 vx 2 vy [ y ( y (1 v z )+2 vz ) R + y (1 x )(1+ y x )( S + T )+(1 x )(1+ x v z vx xz +2 vz ) P =0 (5.15) Sincedingaclosedformanalyticalsolutionforthisnonlinearoptimizationproblemwith constraints(5.14)isexhaustive,weusedcomputationalmethodstodtheoptimalsolution foragivenpaymatrix.Todoso,weapplied fmincon(.) packagein MatLab software tothisnonlinearoptimizationproblem.Table(5.1)showstheresultsfortpay 139 matrices. Table5.1:Optimalgenomeofageneralplayeriniteratedtrustgame.Eachrowcontains themeanandvarianceofoptimalvaluesovertwentyruns. Temptationreward(T) P 1 P 2 P 3 P 4 2 1 : 0 0 : 00 : 79 0 : 10 : 79 0 : 080 : 65 0 : 12 4 1 : 0 0 : 00 : 88 0 : 040 : 88 0 : 040 : 90 0 : 03 5 0 : 88 0 : 090 : 82 0 : 070 : 72 0 : 130 : 93 0 : 01 6 0 : 70 0 : 160 : 64 0 : 160 : 72 0 : 170 : 96 0 : 001 7 0 : 60 0 : 230 : 22 0 : 080 : 99 0 : 00 : 38 0 : 23 8 0 : 5 0 : 250 : 06 0 : 021 : 0 0 : 00 : 5 0 : 25 9 0 : 56 0 : 250 : 02 0 : 0061 : 0 0 : 00 : 44 0 : 25 10 0 : 46 0 : 250 : 02 0 : 0161 : 0 0 : 00 : 54 0 : 25 11 0 : 54 0 : 250 : 01 0 : 0061 : 0 0 : 00 : 46 0 : 25 12 0 : 48 0 : 250 : 01 0 : 0041 : 0 0 : 00 : 52 0 : 25 13 0 : 54 0 : 250 : 02 0 : 011 : 0 0 : 00 : 46 0 : 25 14 0 : 48 0 : 250 : 005 0 : 01 : 0 0 : 00 : 52 0 : 25 15 0 : 52 0 : 250 : 0 0 : 01 : 0 0 : 00 : 48 0 : 25 Accordingtotable(5.1),itisobviousthattheoptimalstrategiesare: P 1 =(0 : 0 ; 1 : 0 ; 0 : 0 ; 1 : 0),masterslavestrategy P 2 =(0 : 0 ; 0 : 0 ; 1 : 0 ; 1 : 0),supercooperationstrategy(5.16) Sincepopulationishomogenous,thereisnocompetitionbetweenanytypes;thus,master slaveandsupercooperationbothyieldtomaximummeanandtheyarebothoptimal strategies;however,ifpopulationisnothomogenous,masterslavestrategywouldbevery poorstrategy,anditwillbeextinctovergenerations. 140 5.3Quantifyingcommunicationandinformationex- changebycalculatingmutualinformationbetween randomvariablesinconsecutiveiterationsofITD game Here,wewanttoquantifythecommunicationrateandtheamountofexchangedinformation betweenplayersovertheconsecutiveinteractionsinroundsofITD.Inoursimulations,welet astrategyplaysagainstitself,theevolvedstrategyinnoiselessenvironment,andarandom strategy;however,hereallthederivationsarewrittenforgeneralstrategies. Afterderivingtheprobabilitydistributionsofeachtypeofinteractionandplaysofplayers (cooperation(C)ordefection(D)),wecancalculatethemutualinformationbetweenthese randomvariablesatconsecutiveroundsofgame.Thismutualinformationisourmeasure toquantifytheexchangedinformationbetweenplayersduringiteratedgames.Moreover, byaddingnoisetothecommunicationchannel,wequantifytheofnoiseonexchanged informationandconsequentlyontheevolutionofcooperation,hopingtothebreakpoints ofthecooperativestrategiesinIPDandITDregimes.Sincetheprobabilityofcooperation overtheinteractiondoesnotplayanimportantroleonaverageofplayers,we ignorethisgenefortheentireanalysisofthissection.Thus,thestrategiesofplayershas fourgenesas: f S S = P S 1 ;P S 2 ;P S 3 ;P S 4 f S O = P O 1 ;P O 2 ;P O 3 ;P O 4 (5.17) 141 Inequation(5.17),theopponent'sstrategyiswrittenaccordingtoopponent'sviewpoint. Sincewewriteeverythingincentralplayer'sviewpoint,wehavetomodifyopponent'sstrat- egyas: f S O = P O 1 ;P O 3 ;P O 2 ;P O 4 (5.18) Wecanthefollowingrandomvariables: A S ( t ) 2f C;D g A O ( t ) 2f C;D g G ( t ) 2f CC;CD;DC;DD g = f g 1 ;g 2 ;g 3 ;g 4 g (5.19) where A S , A O ,and G standforcentralplayersaction,opponent'saction,andgameplayed respectivelyattime\ t ".Theprobabilityofeacheventis: f P A S = h P S C ( t ) ;P S D ( t ) i 0 f P A O = h P O C ( t ) ;P O D ( t ) i 0 f P t G =[ ˇ 1 ( t ) ;ˇ 2 ( t ) ;ˇ 1 ( t ) ;ˇ 4 ( t )]=[ ˇ 1 ( t ) ;ˇ 2 ( t ) ;ˇ 3 ( t ) ;ˇ 4 ( t )] 0 (5.20) 5.3.1Mutualinformationbetween G ( t )andotherrandomvari- ablesat\ t +1" Herewewanttocalculatethemutualinformationbetweengameplayedatround\ t "and allotherrandomvariablesat\ t +1".Inthissituation,playersknowwhatgamehasbeen playedatround\ t "andbasedonthisavailableinformationtheywanttochoosetheiraction. 142 Inotherwords,playersremembertheirownandopponent'spreviousactions.Thisisthe maximuminformationoneplayercanhaveaboutthepreviousroundofgameandbased onthisavailableinformationitiseasierfortheplayertodecidewhattodo.Wewantto calculatethemutualinformationbetween G ( t )and A S ( t +1), A O ( t +1),and G ( t +1). 5.3.1.1Mutualinformationbetween G ( t )and A S ( t +1) First,wehavetocalculatethetransitionmatrices.Wehave: T GS = 0 B @ CCCDDCDD CP S 1 P S 2 P S 3 P S 4 D 1 P S 1 1 P S 2 1 P S 3 1 P S 4 1 C A (5.21) Therefore,wehave: H A S ( t +1) = P S C ( t +1)log 2 P S C ( t +1) P S D ( t +1)log 2 P S D ( t +1) (5.22) where: P S C ( t +1)= P S 1 ˇ 1 ( t )+ P S 2 ˇ 2 ( t )+ P S 3 ˇ 4 ( t )+ P S 4 ˇ 4 ( t ) = f G S f P t G P S D ( t +1)= 1 P S 1 ˇ 1 ( t )+ 1 P S 2 ˇ 2 ( t )+ 1 P S 3 ˇ 3 ( t )+ 1 P S 4 ˇ 4 ( t ) = 1 f G S f P t G (5.23) 143 Thus,wehave: H A S ( t +1) = f G S f P t G log 2 f G S f P t G 1 f G S f P t G log 2 1 f G S f P t G = 2 X i =1 4 X j =1 T GS ( i;j ) f P t G ( j )log 2 4 X j =1 T GS ( i;j ) f P t G ( j ) (5.24) Theconditionalentropybetween G ( t )and A S ( t +1)isgivenby: H A S ( t +1) G ( t ) = 4 X i =1 ˇ i H A S ( t +1) G ( t )= g i = 4 X i =1 2 X j =1 f P t G ( i ) T GS ( j;i )log 2 T GS ( j;i )(5.25) Therefore,themutualinformationbetween G ( t )and A S ( t +1)is: I A S ( t +1); G ( t ) = H A S ( t +1) H A S ( t +1) G ( t ) = 2 X i =1 4 X j =1 T GS ( i;j ) f P t G ( j )log 2 4 X j =1 T GS ( i;j ) f P t G ( j ) + 4 X i =1 2 X j =1 f P t G ( i ) T GS ( j;i )log 2 T GS ( j;i ) (5.26) 5.3.1.2Mutualinformationbetween G ( t )and A O ( t +1) Tothemutualinformationbetween G ( t )and A O ( t +1)weshouldfollowthesameidea. Wehave: T GO = 0 B @ CCCDDCDD CP O 1 P O 3 P O 2 P O 4 D 1 P O 1 1 P O 3 1 P O 2 1 P O 4 1 C A (5.27) 144 Therefore,wehave: H A O ( t +1) = g G O f P t G log 2 g G O f P t G 1 g G O f P t G log 2 1 g G O f P t G = 2 X i =1 4 X j =1 T GO ( i;j ) f P t G ( j )log 2 4 X j =1 T GO ( i;j ) f P t G ( j ) (5.28) Theconditionalentropybetween G ( t )and A O ( t +1)similartoprevioussubsectionisgiven by: H A O ( t +1) G ( t ) = 4 X i =1 ˇ i H A O ( t +1) G ( t )= g i = 4 X i =1 2 X j =1 f P t G ( i ) T GO ( j;i )log 2 T GO ( j;i )(5.29) Therefore,themutualinformationbetween G ( t )and A O ( t +1)is: I A O ( t +1); G ( t ) = H A O ( t +1) H A O ( t +1) j G ( t ) = 2 X i =1 4 X j =1 T GO ( i;j ) f P t G ( j )log 2 4 X j =1 T GO ( i;j ) f P t G ( j ) + 4 X i =1 2 X j =1 f P t G ( i ) T GO ( j;i )log 2 T GO ( j;i ) (5.30) Itisworthmentioningthatherewederivethemutualinformationbetweenround t and t +1basedonselfplayer'sviewpoint.Inotherwords, ˇ 2 ( : )istheactionthatselfplayer cooperatesandopponentdefects.Towritetheequationsforopponent,weusethesame randomvariables.Forexample,ifopponentiscentralplayeranditcooperates,andself playerastheopponentinthisviewpointdefects,weuse ˇ 3 ( : )toderivetheequations. 145 5.3.1.3Mutualinformationbetween G ( t )and G ( t +1) Thetransitionmatrixbetween G ( t )and G ( t +1)is: T GG = 0 B B B B B B B B @ CCCDDCDD CCP S 1 P O 1 P S 2 P O 3 P S 3 P O 2 P S 4 P O 4 CDP S 1 1 P O 1 P S 2 1 P O 3 P S 3 1 P O 2 P S 4 1 P O 4 DC 1 P S 1 P O 1 1 P S 2 P O 3 1 P S 3 P O 2 1 P S 4 P O 4 DD 1 P S 1 1 P O 1 1 P S 2 1 P O 3 1 P S 3 1 P O 2 1 P S 4 1 P O 4 1 C C C C C C C C A (5.31) Therefore,wehave: H G ( t +1) = 4 X i =1 4 X j =1 T GG ( i;j ) f P t G ( j )log 2 4 X j =1 T GG ( i;j ) f P t G ( j ) (5.32) Theconditionalentropybetween G ( t )and G ( t +1)is: H G ( t +1) G ( t ) = 4 X i =1 4 X j =1 f P t G ( i ) T GG ( j;i )log 2 T GG ( j;i )(5.33) Thus,themutualinformationbetween G ( t )and G ( t +1)is: I G ( t +1); G ( t ) = H G ( t +1) H G ( t +1) G ( t ) = 4 X i =1 4 X j =1 T GG ( i;j ) f P t G ( j )log 2 4 X j =1 T GG ( i;j ) f P t G ( j ) + 4 X i =1 4 X j =1 f P t G ( i ) T GG ( j;i )log 2 T GG ( j;i ) (5.34) 146 5.3.2Mutualinformationbetween A S ( t )andotherrandomvari- ablesat\ t +1" Tothemutualinformationbetween A S ( t )and A S ( t +1), A O ( t +1),and G ( t +1), wehavetoderivethetransitionmatrices.Itisobviousthattheserandomvariables arefunctionsof A O ( t )aswell;therefore, A O ( t )canbeseenasthehiddenstateofMarkov processbetweentheserandomvariableswhichneedstobeestimated.Therearetwoways toestimate A O ( t ):onewayistorecursivelycalculate A O ( t )from t =1 ;::: andtheother waywouldbetheestimationof G ( t )insteadystatefromtherighteigenvectoroftransition matrixcorrespondingtoeigenvalueequalstooneandusingthissteadystatetoestimate A O ( t ). 5.3.2.1Mutualinformationbetween A S ( t )and A S ( t +1) Thetransitionmatrixbetween A S ( t )and A S ( t +1)isgivenby: T SS = 0 B @ CD CP S 1 P A C O ( t ) A C S ( t ) + P S 2 P A D O ( t ) A C S ( t ) P S 3 P A C O ( t ) A D S ( t ) + P S 4 P A D O ( t ) A D S ( t ) D 1 P S 1 P A C O ( t ) A C S ( t ) + 1 P S 2 P A D O ( t ) A C S ( t ) 1 P S 3 P A C O ( t ) A D S ( t ) + 1 P S 4 P A D O ( t ) A D S ( t ) 1 C A (5.35) or: T SS = 0 B @ CD C P S 1 ˇ 1 + P S 2 ˇ 2 ˇ 1 + ˇ 2 P S 3 ˇ 3 + P S 4 ˇ 4 ˇ 3 + ˇ 4 D 1 P S 1 ˇ 1 + 1 P S 2 ˇ 2 ˇ 1 + ˇ 2 1 P S 3 ˇ 3 + 1 P S 4 ˇ 4 ˇ 3 + ˇ 4 1 C A (5.36) 147 Therefore,wehave: H A S ( t +1) = f G S f P t G log 2 f G S f P t G 1 f G S f P t G log 2 1 f G S f P t G ( t ) = 2 X i =1 2 X j =1 T SS ( i;j ) f P A S ( j )log 2 2 X j =1 T SS ( i;j ) f P A S ( j ) (5.37) Theconditionalentropybetween A S ( t )and A S ( t +1)isgivenby: H A S ( t +1) A S ( t ) = X i 2f C;D g n P A S ( t )= i o H A S ( t +1) A S ( t )= i = ˇ 1 + ˇ 2 z }| { n P A S ( t )= C o 2 X i =1 T SS ( i; 1)log 2 T SS ( i; 1) ˇ 3 + ˇ 4 z }| { n P A S ( t )= D o 2 X i =1 T SS ( i; 2)log 2 T SS ( i; 2) = 2 X i =1 2 X j =1 ( ˇ 2 i 1 + ˇ 2 i ) T SS ( j;i )log 2 T SS ( j;i ) (5.38) Therefore,themutualinformationbetween A S ( t )and A S ( t +1)is: I ( A S ( t +1); A S ( t ))= H A S ( t +1) H A S ( t +1) j A S ( t ) = 2 X i =1 2 X j =1 T SS ( i;j ) f P A S ( j )log 2 2 X j =1 T SS ( i;j ) f P A S ( j ) + 2 X i =1 2 X j =1 ( ˇ 2 i 1 + ˇ 2 i ) T SS ( j;i )log 2 T SS ( j;i ) (5.39) 148 5.3.2.2Mutualinformationbetween A S ( t )and A O ( t +1) Tothemutualinformationbetween A O ( t +1)and A S ( t ),weshouldfollowthesame idea.Wehave: T SO = 0 B @ CD CP O 1 P A C O ( t ) A C S ( t ) + P O 3 P A D O ( t ) A C S ( t ) P O 2 P A C O ( t ) A D S ( t ) + P O 4 P A D O ( t ) A D S ( t ) D 1 P O 1 P A C O ( t ) A C S ( t ) + 1 P O 3 P A D O ( t ) A C S ( t ) 1 P O 2 P A C O ( t ) A D S ( t ) + 1 P O 4 P A D O ( t ) A D S ( t ) 1 C A (5.40) or: T SO = 0 B @ CD C P O 1 ˇ 1 + P O 3 ˇ 2 ˇ 1 + ˇ 2 P O 2 ˇ 3 + P O 4 ˇ 4 ˇ 3 + ˇ 4 D 1 P O 1 ˇ 1 + 1 P O 3 ˇ 2 ˇ 1 + ˇ 2 1 P O 2 ˇ 3 + 1 P O 4 ˇ 4 ˇ 3 + ˇ 4 1 C A (5.41) Therefore,wehave: H A O ( t +1) = g G O f P t G log 2 g G O f P t G 1 g G O f P t G log 2 1 g G O f P t G = 2 X i =1 2 X j =1 T SO ( i;j ) f P A S ( j )log 2 2 X j =1 T SO ( i;j ) f P A S ( j ) (5.42) Theconditionalentropybetween A S ( t )and A O ( t +1)isgivenby: H A O ( t +1) A S ( t ) = X i 2f C;D g n P A S ( t )= i o H A O ( t +1) A S ( t )= i = ˇ 1 + ˇ 2 z }| { n P A S ( t )= C o 2 X i =1 T SO ( i; 1)log 2 T SO ( i; 1) ˇ 3 + ˇ 4 z }| { n P A S ( t )= D o 2 X i =1 T SO ( i; 2)log 2 T SO ( i; 2) = 2 X i =1 2 X j =1 ( ˇ 2 i 1 + ˇ 2 i ) T SO ( j;i )log 2 T SO ( j;i ) (5.43) 149 Thus,themutualinformationbetween A S ( t )and A O ( t +1)is: I A O ( t +1); A S ( t ) = H A O ( t +1) H A O ( t +1) A S ( t ) = 2 X i =1 2 X j =1 T SO ( i;j ) f P A S ( j )log 2 2 X j =1 T SO ( i;j ) f P A S ( j ) + 2 X i =1 2 X j =1 ( ˇ 2 i 1 + ˇ 2 i ) T SO ( j;i )log 2 T SO ( j;i ) (5.44) 5.3.2.3Mutualinformationbetween A S ( t )and G ( t +1) Thetransitionmatrixbetween A S ( t )and G ( t +1)is: T SG = 0 B B B B B B B B B B B B B B B B B B B B B B B B B B B B @ P S 1 P O 1 P A C O ( t ) A C S ( t ) + P S 2 P O 3 P A D O ( t ) A C S ( t ) P S 3 P O 2 P A C O ( t ) A D S ( t ) + P S 4 P O 4 P A D O ( t ) A D S ( t ) P S 1 1 P O 1 P A C O ( t ) A C S ( t ) + P S 2 1 P O 3 P A D O ( t ) A C S ( t ) P S 3 1 P O 2 P A C O ( t ) A D S ( t ) + P S 4 1 P O 4 P A D O ( t ) A D S ( t ) 1 P S 1 P O 1 P A C O ( t ) A C S ( t ) + 1 P S 2 P O 3 P A D O ( t ) A C S ( t ) 1 P S 3 P O 2 P A C O ( t ) A D S ( t ) + 1 P S 4 P O 4 P A D O ( t ) A D S ( t ) 1 P S 1 1 P O 1 P A C O ( t ) A C S ( t ) + 1 P S 2 1 P O 3 P A D O ( t ) A C S ( t ) 1 P S 3 1 P O 2 P A C O ( t ) A D S ( t ) + 1 P S 4 1 P O 4 P A D O ( t ) A D S ( t ) 1 C C C C C C C C C C C C C C C C C C C C C C C C C C C C A (5.45) or: T SG = 0 B B B B B B B B @ CD CC P S 1 P O 1 ˇ 1 + P S 2 P O 1 ˇ 2 ˇ 1 + ˇ 2 P S 3 P O 2 ˇ 3 + P S 4 P O 4 ˇ 4 ˇ 3 + ˇ 4 CD P S 1 ( 1 P O 1 ) ˇ 1 + P S 2 ( 1 P O 3 ) ˇ 2 ˇ 1 + ˇ 2 P S 3 P O 2 ˇ 3 + P S 4 ( 1 P O 4 ) ˇ 4 ˇ 3 + ˇ 4 DC ( 1 P S 1 ) P O 1 ˇ 1 + ( 1 P S 2 ) P O 3 ˇ 2 ˇ 1 + ˇ 2 ( 1 P S 3 ) P O 2 ˇ 3 + ( 1 P S 4 ) P O 4 ˇ 4 ˇ 3 + ˇ 4 DD ( 1 P S 1 )( 1 P O 1 ) ˇ 1 + ( 1 P S 2 )( 1 P O 3 ) ˇ 2 ˇ 1 + ˇ 2 ( 1 P S 3 )( 1 P O 2 ) ˇ 3 + ( 1 P S 4 )( 1 P O 4 ) ˇ 4 ˇ 3 + ˇ 4 1 C C C C C C C C A (5.46) Wehave: H G ( t +1) = 4 X i =1 2 X j =1 T SG ( i;j ) f P A S ( j )log 2 2 X j =1 T SG ( i;j ) f P A S ( j ) (5.47) 150 Theconditionalentropybetween A S ( t )and G ( t +1)isgivenby: H G ( t +1) A S ( t ) = X i 2f C;D g n P A S ( t )= i o H G ( t +1) A S ( t )= i = 2 X i =1 4 X j =1 ( ˇ 2 i 1 + ˇ 2 i ) T SG ( j;i )log 2 T SG ( j;i ) (5.48) Thus,themutualinformationbetween A S ( t )and G ( t +1)is: I G ( t +1); A S ( t ) = H G ( t +1) H G ( t +1) A S ( t ) = 4 X i =1 2 X j =1 T SG ( i;j ) f P A S ( j )log 2 2 X j =1 T SG ( i;j ) f P A S ( j ) + 2 X i =1 4 X j =1 ( ˇ 2 i 1 + ˇ 2 i ) T SG ( j;i )log 2 T SG ( j;i ) 5.3.3Mutualinformationbetween A O ( t )andotherrandomvari- ablesat\ t +1" Tothemutualinformationbetween A O ( t )andotherrandomvariablesat\ t +1",we havetofollowthesameideaandderivethetransitionmatrices. 5.3.3.1Mutualinformationbetween A O ( t )and A S ( t +1) Thetransitionmatrixbetween A O ( t )and A S ( t +1)isgivenby: T OS = 0 B @ CD CP S 1 P A C S ( t ) A C O ( t ) + P S 3 P A D S ( t ) A C O ( t ) P S 2 P A C S ( t ) A D O ( t ) + P S 4 P A D S ( t ) A D O ( t ) D 1 P S 1 P A C S ( t ) A C O ( t ) + 1 P S 3 P A D S ( t ) A C O ( t ) 1 P S 2 P A C S ( t ) A D O ( t ) + 1 P S 4 P A D S ( t ) A D O ( t ) 1 C A (5.49) 151 or: T OS = 0 B @ CD C P S 1 ˇ 1 + P S 3 ˇ 3 ˇ 1 + ˇ 3 P S 2 ˇ 2 + P S 4 ˇ 4 ˇ 2 + ˇ 4 D 1 P S 1 ˇ 1 + 1 P S 3 ˇ 3 ˇ 1 + ˇ 3 1 P S 2 ˇ 2 + 1 P S 4 ˇ 4 ˇ 2 + ˇ 4 1 C A (5.50) Therefore,wehave: H A S ( t +1) = f G S f P t G log 2 f G S f P t G 1 f G S f P t G log 2 1 f G S f P t G = 2 X i =1 2 X j =1 T OS ( i;j ) f P A O ( j )log 2 2 X j =1 T OS ( i;j ) f P A O ( j ) (5.51) Theconditionalentropybetween A O ( t )and A S ( t +1)isgivenby: H A S ( t +1) A O ( t ) = X i 2f C;D g n P A O ( t )= i o H A S ( t +1) A O ( t )= i = ˇ 1 + ˇ 3 z }| { n P A O ( t )= C o 2 X i =1 T OS ( i; 1)log 2 T OS ( i; 1) ˇ 2 + ˇ 4 z }| { n P A O ( t )= D o 2 X i =1 T OS ( i; 2)log 2 T OS ( i; 2) = 2 X i =1 2 X j =1 ( ˇ i + ˇ i +2 ) T OS ( j;i )log 2 T OS ( j;i ) (5.52) 152 Therefore,themutualinformationbetween A O ( t )and A S ( t +1)is: I A S ( t +1); A O ( t ) = H A S ( t +1) H A S ( t +1) A O ( t ) = 2 X i =1 2 X j =1 T OS ( i;j ) f P A O ( j )log 2 2 X j =1 T OS ( i;j ) f P A O ( j ) + 2 X i =1 2 X j =1 ( ˇ i + ˇ i +2 ) T OS ( j;i )log 2 T OS ( j;i ) (5.53) 5.3.3.2Mutualinformationbetween A O ( t )and A O ( t +1) Wehave: T OO = 0 B @ CD CP O 1 P A C S ( t ) A C O ( t ) + P O 2 P A D S ( t ) A C O ( t ) P O 3 P A C S ( t ) A D O ( t ) + P O 4 P A D S ( t ) A D O ( t ) D 1 P O 1 P A C S ( t ) A C O ( t ) + 1 P O 2 P A D S ( t ) A C O ( t ) 1 P O 3 P A C S ( t ) A D O ( t ) + 1 P O 4 P A D S ( t ) A D O ( t ) 1 C A (5.54) or: T OO = 0 B @ CD C P O 1 ˇ 1 + P O 2 ˇ 3 ˇ 1 + ˇ 3 P O 3 ˇ 2 + P O 4 ˇ 4 ˇ 2 + ˇ 4 D 1 P O 1 ˇ 1 + 1 P O 2 ˇ 3 ˇ 1 + ˇ 3 1 P O 3 ˇ 2 + 1 P O 4 ˇ 4 ˇ 2 + ˇ 4 1 C A (5.55) Therefore,wehave: H A O ( t +1) = g G O f P t G log 2 g G O f P t G 1 g G O f P t G log 2 1 g G O f P t G = 2 X i =1 2 X j =1 T OO ( i;j ) f P A O ( j )log 2 2 X j =1 T OO ( i;j ) f P A O ( j ) (5.56) 153 Theconditionalentropybetween A O ( t )and A O ( t +1)isgivenby: H A O ( t +1) A O ( t ) = X i 2f C;D g n P A O ( t )= i o H A O ( t +1) A O ( t )= i = ˇ 1 + ˇ 3 z }| { n P A O ( t )= C o 2 X i =1 T OO ( i; 1)log 2 T OO ( i; 1) ˇ 2 + ˇ 4 z }| { n P A O ( t )= D o 2 X i =1 T OO ( i; 2)log 2 T OO ( i; 2) = 2 X i =1 2 X j =1 ( ˇ i + ˇ i +2 ) T OO ( j;i )log 2 T OO ( j;i ) (5.57) Therefore,themutualinformationbetween A O ( t )and A O ( t +1)is: I A O ( t +1); A O ( t ) = H A O ( t +1) H A O ( t +1) A O ( t ) = 2 X i =1 2 X j =1 T OO ( i;j ) f P A O ( j )log 2 2 X j =1 T OO ( i;j ) f P A O ( j ) + 2 X i =1 2 X j =1 ( ˇ i + ˇ i +2 ) T OO ( j;i )log 2 T OO ( j;i ) (5.58) 5.3.3.3Mutualinformationbetween A O ( t )and G ( t +1) Thetransitionmatrixbetween A O ( t )and G ( t +1)is: T OG = 0 B B B B B B B B B B B B B B B B B B B B B B B B B B B B @ P S 1 P O 1 P A C S ( t ) A C O ( t ) + P S 3 P O 2 P A D S ( t ) A C O ( t ) P S 2 P O 3 P A C S ( t ) A D O ( t ) + P S 4 P O 4 P A D S ( t ) A D O ( t ) P S 1 1 P O 1 P A C S ( t ) A C O ( t ) + P S 3 1 P O 2 P A D S ( t ) A C O ( t ) P S 2 1 P O 3 P A C S ( t ) A D O ( t ) + P S 4 1 P O 4 P A D S ( t ) A D O ( t ) 1 P S 1 P O 1 P A C O ( t ) A C S ( t ) + 1 P S 3 P O 2 P A D S ( t ) A C O ( t ) 1 P S 2 P O 3 P A C S ( t ) A D O ( t ) + 1 P S 4 P O 4 P A D S ( t ) A D O ( t ) 1 P S 1 1 P O 1 P A C S ( t ) A C O ( t ) + 1 P S 3 1 P O 2 P A D S ( t ) A C O ( t ) 1 P S 2 1 P O 3 P A C S ( t ) A D O ( t ) + 1 P S 4 1 P O 4 P A D S ( t ) A D O ( t ) 1 C C C C C C C C C C C C C C C C C C C C C C C C C C C C A (5.59) 154 or: T OG = 0 B B B B B B B B @ CD CC P S 1 P O 1 ˇ 1 + P S 3 P O 2 ˇ 3 ˇ 1 + ˇ 3 P S 2 P O 3 ˇ 2 + P S 4 P O 4 ˇ 4 ˇ 2 + ˇ 4 CD P S 1 ( 1 P O 1 ) ˇ 1 + P S 3 ( 1 P O 2 ) ˇ 3 ˇ 1 + ˇ 3 P S 2 ( 1 P O 3 ) ˇ 2 + P S 4 ( 1 P O 4 ) ˇ 4 ˇ 2 + ˇ 4 DC ( 1 P S 1 ) P O 1 ˇ 1 + ( 1 P S 3 ) P O 2 ˇ 3 ˇ 1 + ˇ 3 ( 1 P S 2 ) P O 3 ˇ 2 + ( 1 P S 4 ) P O 4 ˇ 4 ˇ 2 + ˇ 4 DD ( 1 P S 1 )( 1 P O 1 ) ˇ 1 + ( 1 P S 3 )( 1 P O 2 ) ˇ 3 ˇ 1 + ˇ 3 ( 1 P S 2 ) P O 3 ˇ 2 + ( 1 P S 4 )( 1 P O 4 ) ˇ 4 ˇ 2 + ˇ 4 1 C C C C C C C C A (5.60) Wehave: H G ( t +1) = 4 X i =1 2 X j =1 T OG ( i;j ) f P A O ( j )log 2 2 X j =1 T OG ( i;j ) f P A O ( j ) (5.61) Theconditionalentropybetween A O ( t )and G ( t +1)isgivenby: H G ( t +1) A O ( t ) = X i 2f C;D g n P A O ( t )= i o H G ( t +1) A O ( t )= i = 2 X i =1 4 X j =1 ( ˇ i + ˇ i +2 ) T OG ( j;i )log 2 T OG ( j;i ) (5.62) Thus,themutualinformationbetween A O ( t )and G ( t +1)is: I G ( t +1); A O ( t ) = H G ( t +1) H G ( t +1) A O ( t ) = 4 X i =1 2 X j =1 T OG ( i;j ) f P A O ( j )log 2 2 X j =1 T OG ( i;j ) f P A O ( j ) + 2 X i =1 4 X j =1 ( ˇ i + ˇ i +2 ) T OG ( j;i )log 2 T OG ( j;i ) (5.63) Inthissection,wecalculatethemutualinformationbetweenrandomvariablesattime\ t " and\ t +1"innoisyenvironments.Todoso,weassumeplayershavefuzzy/feeblemind. Thus,byintroducingerrorterm\ ",orchannelerror,wecanwriteanerrormatrixforwhat 155 aplayerremembersfromhispreviousaction,hisopponent'saction,andpreviousgame.This noisetheplayersnextmovebycreatinganerrorterminestimating/rememberingthe previousactions.Therefore,theplayersnextaction,andconsequentlytheirmean willbebytheerrorcausedbyenvironmentalnoise.Figure(5.1)showstheerrorof channelsforthethreerandomvariablesattime\ t ".Considering(5.1),wecand thefollowingerrormatrices: Err S = 0 B @ 1 1 1 1 1 1 1 C A (5.64) Err O = 0 B @ 1 2 2 2 1 2 1 C A (5.65) Err G = 0 B B B B B B B B B @ (1 1 )(1 2 )(1 1 ) 2 1 (1 2 ) 1 2 (1 1 ) 2 (1 1 )(1 2 ) 1 2 1 (1 2 ) 1 (1 2 ) 1 2 (1 1 )(1 2 )(1 1 ) 2 1 2 1 (1 2 )(1 1 ) 2 (1 1 )(1 2 ) 1 C C C C C C C C C A (5.66) 5.3.4Mutualinformationbetween G ( t )andotherrandomvari- ablesat\ t +1" First,wederivethemutualinformationbetween G ( t )and A S ( t +1).Similartonoiseless environment,wecanwritethetransitionmatricesandcalculatethemutualinformation between G ( t )and A S ( t +1).Thetransitionmatrixbetween G ( t )and A S ( t +1)isgivenat 156 Figure5.1:Theschematicofchannelerror.Thiserrorhappenswhenplayershavefuzzymind abouttheirownandopponent'spreviousaction(a)thebinarysymmetricchannelerrorof player'sselfaction.(b)thebinarysymmetricchannelerrorofopponent'sselfaction.(c) thechannelerrorofchosenactionsbyplayeranditsopponent(gameplayed)atprevious iteration. 157 equation(5.21).Therefore,wehave: A S ( t +1)= T GS ^ f P t G = T GS Err G f P t G (5.67) H A S ( t +1) = f G S ^ f P t G log 2 f G S ^ f P t G 1 f G S ^ f P t G log 2 1 f G S ^ f P t G = 2 X i =1 4 X j =1 T GS ( i;j ) Err G f P t G log 2 2 X j =1 T GS ( i;j ) Err G f P t G (5.68) Theonlycebetweenthenoiselessenvironmentandnoisyenvironmentisusing ^ f P t G insteadof f P t G ,whichcanbesubstitutedby Err G f P t G inequation(5.24).Similartoaction entropy,wecansubstitute f P t G with Err G f P t G inequations(5.25)and(5.26)tocalculatethe mutualinformationbetween G ( t )and A S ( t +1)innoisyenvironments. Tothemutualinformationbetween G ( t )and A O ( t +1)innoisyenvironment,we shouldfollowthesameidea.Thetransitionmatrixbetween G ( t )and A O ( t +1)isgivenat equation(5.27).Wehave: A O ( t +1)= T GO ^ f P t G = T GO Err G f P t G (5.69) H A O ( t +1) = g G O ^ f P t G log 2 g G O ^ f P t G 1 g G O ^ f P t G log 2 1 g G O ^ f P t G = 2 X i =1 4 X j =1 T GO ( i;j ) Err G f P t G log 2 2 X j =1 T GO ( i;j ) Err G f P t G (5.70) Similartopreviouscase,wecansubstitute f P t G with Err G f P t G inequations(5.29)and(5.30) andcalculatethemutualinformationbetween G ( t )and A O ( t +1)innoisyenvironments. Tothemutualinformationbetween G ( t )and G ( t +1)wehavetowritethetran- 158 sitionmatrixinnoisyenvironmentswhichisgivenin(5.31).Thegameentropyinnoisy environmentis: H G ( t +1) = 4 X i =1 4 X j =1 T GG ( i;j ) ^ f P t G log 2 4 X j =1 T GG ( i;j ) ^ f P t G = 4 X i =1 4 X j =1 T GG ( i;j ) Err G f P t G log 2 4 X j =1 T GG ( i;j ) Err G f P t G (5.71) Theconditionalentropybetween G ( t )and G ( t +1)is: H G ( t +1) G ( t ) = 4 X i =1 4 X j =1 ^ f P t G ( i ) T GG ( j;i )log 2 T GG ( j;i ) = 4 X i =1 4 X j =1 Err G f P t G ( i ) T GG ( j;i )log 2 T GG ( j;i )(5.72) Therefore,themutualinformationbetween G ( t )and G ( t +1)innoisyenvironmentis: I G ( t +1); G ( t ) = H G ( t +1) H G ( t +1) G ( t ) = 4 X i =1 4 X j =1 T GG ( i;j ) Err G f P t G log 2 4 X j =1 T GG ( i;j ) Err G f P t G + 4 X i =1 4 X j =1 Err G f P t G ( i ) T GG ( j;i )log 2 T GG ( j;i ) (5.73) 5.3.5Mutualinformationbetween A S ( t )andotherrandomvari- ablesat\ t +1"innoisyenvironments Similartonoiselessenvironments,tothemutualinformationbetween A S ( t )and A S ( t +1), A O ( t +1),and G ( t +1)wehavetoderivethetransitionmatrices.Asweknow A O ( t )shows itselfasahiddenrandomvariableinthetransitionmatrices;andsincenowweareinnoisy 159 environments,wehavetoconsidertheofnoiseinestimationofthishiddenrandom variable. 5.3.5.1Mutualinformationbetween A S ( t )and A S ( t +1) Thetransitionmatrixbetween A S ( t )and A S ( t +1)isgivenby: ^ T SS = 0 B @ CD CP S 1 ^ P A C O ( t ) A C S ( t ) + P S 2 ^ P A D O ( t ) A C S ( t ) P S 3 ^ P A C O ( t ) A D S ( t ) + P S 4 ^ P A D O ( t ) A D S ( t ) D 1 P S 1 ^ P A C O ( t ) A C S ( t ) + 1 P S 2 ^ P A D O ( t ) A C S ( t ) 1 P S 3 ^ P A C O ( t ) A D S ( t ) + 1 P S 4 ^ P A D O ( t ) A D S ( t ) 1 C A (5.74) where: ^ P A C O ( t ) A C S ( t ) = ^ ˇ 1 ^ ˇ 1 +^ ˇ 2 = P 4 i =1 Err G (1 ;i ) f P t G ( i ) P 2 i =1 P 4 j =1 Err G ( i;j ) f P t G ( j ) ^ P A D O ( t ) A C S ( t ) = ^ ˇ 2 ^ ˇ 1 +^ ˇ 2 = P 4 i =1 Err G (2 ;i ) f P t G ( i ) P 2 i =1 P 4 j =1 Err G ( i;j ) f P t G ( j ) ^ P A C O ( t ) A D S ( t ) = ^ ˇ 3 ^ ˇ 3 +^ ˇ 4 = P 4 i =1 Err G (3 ;i ) f P t G ( i ) P 4 i =3 P 4 j =1 Err G ( i;j ) f P t G ( j ) ^ P A D O ( t ) A D S ( t ) = ^ ˇ 4 ^ ˇ 3 +^ ˇ 4 = P 4 i =1 Err G (4 ;i ) f P t G ( i ) P 4 i =3 P 4 j =1 Err G ( i;j ) f P t G ( j ) (5.75) Therefore,wehave: ^ T SS = 0 B B B B B @ CD C P 2 i =1 P 4 j =1 g G S ( i ) Err G ( i;j ) g P t G ( j ) P 2 i =1 P 4 j =1 Err G ( i;j ) g P t G ( j ) P 4 i =3 P 4 j =1 g G S ( i ) Err G ( i;j ) g P t G ( j ) P 4 i =3 P 4 j =1 Err G ( i;j ) g P t G ( j ) D 1 P 2 i =1 P 4 j =1 g G S ( i ) Err G ( i;j ) g P t G ( j ) P 2 i =1 P 4 j =1 Err G ( i;j ) g P t G ( j ) 1 P 4 i =3 P 4 j =1 g G S ( i ) Err G ( i;j ) g P t G ( j ) P 4 i =3 P 4 j =1 Err G ( i;j ) g P t G ( j ) 1 C C C C C A (5.76) 160 Thus,theentropyofselfactionis: H A S ( t +1) = f G S ^ f P t G log 2 f G S ^ f P t G 1 f G S ^ f P t G log 2 1 f G S ^ f P t G = 2 X i =1 2 X j =1 ^ T SS ( i;j ) ^ f P A S ( j )log 2 2 X j =1 ^ T SS ( i;j ) ^ f P A S ( j ) (5.77) Theconditionalentropybetween A S ( t )and A S ( t +1)isgivenby: H A S ( t +1) A S ( t ) = X i 2f C;D g ˆ P ^ A S ( t )= i ˙ H ^ A S ( t +1) ^ A S ( t )= i = ^ ˇ 1 +^ ˇ 2 z }| { ˆ P ^ A S ( t )= C ˙ 2 X i =1 T SS ( i; 1)log 2 T SS ( i; 1) ^ ˇ 3 +^ ˇ 4 z }| { ˆ P ^ A S ( t )= D ˙ 2 X i =1 T SS ( i; 2)log 2 T SS ( i; 2) = 2 X i =1 2 X j =1 (^ ˇ 2 i 1 +^ ˇ 2 i ) ^ T SS ( j;i )log 2 ^ T SS ( j;i ) = 2 X i =1 2 X j =1 2 i X k =2 i 1 4 X l =1 Err G ( k;l ) f P t G ( l ) ^ T SS ( j;i )log 2 ^ T SS ( j;i ) (5.78) Therefore,themutualinformationbetween A S ( t )and A S ( t +1)is: I ( A S ( t +1); A S ( t ))= H A S ( t +1) H A S ( t +1) A S ( t ) = 2 X i =1 2 X j =1 ^ T SS ( i;j ) ^ g P A S ( j )log 2 ^ T SS ( i;j ) ^ g P A S ( j ) + 2 X i =1 2 X j =1 i +2 X k = i 4 X l =1 Err G ( k;l ) f P t G ( l ) ^ T SS ( j;i )log 2 ^ T SS ( j;i ) (5.79) 161 5.3.5.2Mutualinformationbetween A S ( t )and A O ( t +1) Tothemutualinformationbetween A O ( t +1)and A S ( t ),weshouldfollowthesame idea.Wehave: ^ T SO = 0 B @ CD CP O 1 ^ P A C O ( t ) A C S ( t ) + P O 3 ^ P A D O ( t ) A C S ( t ) P O 2 ^ P A C O ( t ) A D S ( t ) + P O 4 ^ P A D O ( t ) A D S ( t ) D 1 P O 1 ^ P A C O ( t ) A C S ( t ) + 1 P O 3 ^ P A D O ( t ) A C S ( t ) 1 P O 2 ^ P A C O ( t ) A D S ( t ) + 1 P O 4 ^ P A D O ( t ) A D S ( t ) 1 C A (5.80) or: ^ T SO = 0 B B B B B @ CD C P 2 i =1 P 4 j =1 g G O ( i ) Err G ( i;j ) g P t G ( i ) P 2 i =1 P 4 j =1 Err G ( i;j ) g P t G ( i ) P 4 i =3 P 4 j =1 g G O ( i ) Err G ( i;j ) g P t G ( i ) P 4 i =3 P 4 j =1 Err G ( i;j ) g P t G ( i ) D 1 P 2 i =1 P 4 j =1 g G O ( i ) Err G ( i;j ) g P t G ( i ) P 2 i =1 P 4 j =1 Err G ( i;j ) g P t G ( i ) 1 P 4 i =3 P 4 j =1 g G O ( i ) Err G ( i;j ) g P t G ( i ) P 4 i =3 P 4 j =1 Err G ( i;j ) g P t G ( i ) 1 C C C C C A (5.81) Thus,theentropyofopponent'sactionis: H A S ( t +1) = g G O ^ f P t G log 2 g G O ^ f P t G 1 g G O ^ f P t G log 2 1 g G O ^ f P t G = 2 X i =1 2 X j =1 ^ T SO ( i;j ) ^ f P A S ( j )log 2 ^ T SO ( i;j ) ^ f P A S ( j ) (5.82) 162 Theconditionalentropybetween A S ( t )and A O ( t +1)isgivenby: H A O ( t +1) A S ( t ) = X i 2f C;D g ˆ P ^ A S ( t )= i ˙ H ^ A O ( t +1) ^ A S ( t )= i = ^ ˇ 1 +^ ˇ 2 z }| { ˆ P ^ A S ( t )= C ˙ 2 X i =1 T SO ( i; 1)log 2 T SO ( i; 1) ^ ˇ 3 +^ ˇ 4 z }| { ˆ P ^ A S ( t )= D ˙ 2 X i =1 T SO ( i; 2)log 2 T SO ( i; 2) = 2 X i =1 2 X j =1 (^ ˇ 2 i 1 +^ ˇ 2 i ) ^ T SO ( j;i )log 2 ^ T SO ( j;i ) = 2 X i =1 2 X j =1 i +2 X k = i 4 X l =1 Err G ( k;l ) f P t G ( l ) ^ T SO ( j;i )log 2 ^ T SO ( j;i ) (5.83) Therefore,themutualinformationbetween A S ( t )and A O ( t +1)is: I ( A O ( t +1); A S ( t ))= H A O ( t +1) H A O ( t +1) A S ( t ) = 2 X i =1 2 X j =1 ^ T SO ( i;j ) ^ g P A S ( j )log 2 ^ T SO ( i;j ) ^ g P A S ( j ) + 2 X i =1 2 X j =1 i +2 X k = i 4 X l =1 Err G ( k;l ) f P t G ( l ) ^ T SO ( j;i )log 2 ^ T SO ( j;i ) (5.84) 163 5.3.5.3Mutualinformationbetween A S ( t )and G ( t +1) Thetransitionmatrixbetween A S ( t )and G ( t +1)is: ^ T SG = 0 B B B B B B B B B B B B B B B B B B B B B B B B B B B B @ P S 1 P O 1 ^ P A C O ( t ) A C S ( t ) + P S 2 P O 3 ^ P A D O ( t ) A C S ( t ) P S 3 P O 2 ^ P A C O ( t ) A D S ( t ) + P S 4 P O 4 ^ P A D O ( t ) A D S ( t ) P S 1 1 P O 1 ^ P A C O ( t ) A C S ( t ) + P S 2 1 P O 3 ^ P A D O ( t ) A C S ( t ) P S 3 1 P O 2 ^ P A C O ( t ) A D S ( t ) + P S 4 1 P O 4 ^ P A D O ( t ) A D S ( t ) 1 P S 1 P O 1 ^ P A C O ( t ) A C S ( t ) + 1 P S 2 P O 3 ^ P A D O ( t ) A C S ( t ) 1 P S 3 P O 2 ^ P A C O ( t ) A D S ( t ) + 1 P S 4 P O 4 ^ P A D O ( t ) A D S ( t ) 1 P S 1 1 P O 1 ^ P A C O ( t ) A C S ( t ) + 1 P S 2 1 P O 3 ^ P A D O ( t ) A C S ( t ) 1 P S 3 1 P O 2 ^ P A C O ( t ) A D S ( t ) + 1 P S 4 1 P O 4 P A D O ( t ) A D S ( t ) 1 C C C C C C C C C C C C C C C C C C C C C C C C C C C C A (5.85) or: ^ T SG = 0 B B B B B B B B @ CD CC P S 1 P O 1 ^ ˇ 1 + P S 2 P O 3 ^ ˇ 2 ^ ˇ 1 +^ ˇ 2 P S 3 P O 2 ^ ˇ 3 + P S 4 P O 4 ^ ˇ 4 ^ ˇ 3 +^ ˇ 4 CD P S 1 ( 1 P O 1 ) ^ ˇ 1 + P S 2 ( 1 P O 3 ) ^ ˇ 2 ^ ˇ 1 +^ ˇ 2 P S 3 ( 1 P O 2 ) ^ ˇ 3 + P S 4 ( 1 P O 4 ) ^ ˇ 4 ^ ˇ 3 +^ ˇ 4 DC ( 1 P S 1 ) P O 1 ^ ˇ 1 + ( 1 P S 2 ) P O 3 ^ ˇ 2 ^ ˇ 1 +^ ˇ 2 ( 1 P S 3 ) P O 2 ^ ˇ 3 + ( 1 P S 4 ) P O 4 ^ ˇ 4 ^ ˇ 3 +^ ˇ 4 DD ( 1 P S 1 )( 1 P O 1 ) ^ ˇ 1 + ( 1 P S 2 )( 1 P O 3 ) ^ ˇ 2 ^ ˇ 1 +^ ˇ 2 ( 1 P S 3 )( 1 P O 2 ) ^ ˇ 3 + ( 1 P S 4 )( 1 P O 4 ) ^ ˇ 4 ^ ˇ 3 +^ ˇ 4 1 C C C C C C C C A (5.86) Wecanalsowritethistransitionmatrixas: ^ T SG = 0 B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B @ P S 1 P O 1 P 4 i =1 Err G (1 ;i ) g P t G ( i )+ P S 2 P O 3 P 4 i =1 Err G (2 ;i ) g P t G P 2 i =1 P 4 j =1 Err G ( i;j ) g P t G ( j ) P S 3 P O 2 P 4 i =1 Err G (3 ;i ) g P t G ( i )+ P S 4 P O 4 P 4 i =1 Err G (4 ;i ) g P t G ( i ) P 4 i =3 P 4 j =1 Err G ( i;j ) g P t G ( j ) P S 1 1 P O 1 P 4 i =1 Err G (1 ;i ) g P t G ( i )+ P S 2 1 P O 3 P 4 i =1 Err G (2 ;i ) g P t G ( i ) P 2 i =1 P 4 j =1 Err G ( i;j ) g P t G ( j ) P S 3 1 P O 2 P 4 i =1 Err G (3 ;i ) g P t G ( i )+ P S 4 1 P O 4 P 4 i =1 Err G (4 ;i ) g P t G ( i ) P 4 i =3 P 4 j =1 Err G ( i;j ) g P t G ( j ) 1 P S 1 P O 1 P 4 i =1 Err G (1 ;i ) g P t G ( i )+ 1 P S 2 P O 3 P 4 i =1 Err G (2 ;i ) g P t G ( i ) P 2 i =1 P 4 j =1 Err G ( i;j ) g P t G ( j ) 1 P S 3 P O 2 P 4 i =1 Err G (3 ;i ) g P t G ( i )+ 1 P S 4 P O 4 P 4 i =1 Err G (4 ;i ) g P t G ( i ) P 4 i =3 P 4 j =1 Err G ( i;j ) g P t G ( j ) 1 P S 1 1 P O 1 P 4 i =1 Err G (1 ;i ) g P t G ( i )+ 1 P S 2 1 P O 3 P 4 i =1 Err G (2 ;i ) g P t G ( i ) P 2 i =1 P 4 j =1 Err G ( i;j ) g P t G ( j ) 1 P S 3 1 P O 2 P 4 i =1 Err G (3 ;i ) g P t G ( i )+ 1 P S 4 1 P O 4 P 4 i =1 Err G (4 ;i ) g P t G ( i ) P 4 i =3 P 4 j =1 Err G ( i;j ) g P t G ( j ) 1 C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C A (5.87) 164 Wehave: H G ( t +1) = 4 X i =1 2 X j =1 ^ T SG ( i ; j ) ^ A S ( j )log 2 4 X j =1 ^ T SG ( i ; j ) ^ A S ( j ) (5.88) where: ^ A S = 0 B @ P 2 i =1 P 4 j =1 Err G ( i;j ) f P t G ( j ) P 4 i =3 P 4 j =1 Err G ( i;j ) f P t G ( j ) 1 C A (5.89) Theconditionalentropybetween A S ( t )and G ( t +1)isgivenby: H G ( t +1) A S ( t ) = X i 2f C;D g ˆ P ^ A S ( t )= i ˙ H G ( t +1) ^ A S ( t )= i = X i 2f C;D g 4 X j =1 ˆ P ^ A S ( t )= i ˙ ^ T SG ( j ; i )log 2 ^ T SG ( j ; i ) (5.90) Thus,themutualinformationbetween A S ( t )and G ( t +1)is: I G ( t +1); A S ( t ) = H G ( t +1) H G ( t +1) A S ( t ) = 4 X i =1 2 X j =1 ^ T SG ( i ; j ) ^ A S ( j )log 2 4 X j =1 ^ T SG ( i ; j ) ^ A S ( j ) + X i 2f C;D g 4 X j =1 ˆ P ^ A S ( t )= i ˙ ^ T SG ( j ; i )log 2 ^ T SG ( j ; i ) (5.91) 165 5.3.6Mutualinformationbetween A O ( t )andotherrandomvari- ablesat\ t +1" Tothemutualinformationbetween A O ( t )andotherrandomvariablesat\ t +1",we havetofollowthesameideaandderivethetransitionmatrices. 5.3.6.1Mutualinformationbetween A O ( t )and A S ( t +1) Thetransitionmatrixbetween A O ( t )and A S ( t +1)isgivenby: ^ T OS = 0 B @ CD CP S 1 ^ P A C S ( t ) A C O ( t ) + P S 3 ^ P A D S ( t ) A C O ( t ) P S 2 ^ P A C S ( t ) A D O ( t ) + P S 4 ^ P A D S ( t ) A D O ( t ) D 1 P S 1 ^ P A C S ( t ) A C O ( t ) + 1 P S 3 ^ P A D S ( t ) A C O ( t ) 1 P S 2 ^ P A C S ( t ) A D O ( t ) + 1 P S 4 ^ P A D S ( t ) A D O ( t ) 1 C A (5.92) where: ^ P A C S ( t ) A C O ( t ) = ^ ˇ 1 ^ ˇ 1 +^ ˇ 3 = P 4 i =1 Err G (1 ;i ) f P t G ( i ) P 2 i =1 P 4 j =1 Err G (2 i 1 ;j ) f P t G ( j ) ^ P A D S ( t ) A C O ( t ) = ^ ˇ 3 ^ ˇ 1 +^ ˇ 3 = P 4 i =1 Err G (3 ;i ) f P t G ( i ) P 2 i =1 P 4 j =1 Err G (2 i 1 ;j ) f P t G ( j ) ^ P A C S ( t ) A D O ( t ) = ^ ˇ 2 ^ ˇ 2 +^ ˇ 4 = P 4 i =1 Err G (2 ;i ) f P t G ( i ) P 2 i =1 P 4 j =1 Err G (2 i;j ) f P t G ( j ) ^ P A D S ( t ) A D O ( t ) = ^ ˇ 4 ^ ˇ 2 +^ ˇ 4 = P 4 i =1 Err G (4 ;i ) f P t G ( i ) P 2 i =1 P 4 j =1 Err G (2 i;j ) f P t G ( j ) (5.93) Therefore,wehave: ^ T OS = 0 B B B @ CD C P 2 i =1 P 4 j =1 g G S (2 i 1) Err G (2 i 1 ;j ) g P t G ( j ) P 2 i =1 P 4 j =1 Err G (2 i 1 ;j ) g P t G ( j ) P 2 i =1 P 4 j =1 g G S (2 i ) Err G (2 i;j ) g P t G ( j ) P 2 i =1 P 4 j =1 Err G (2 i;j ) g P t G ( j ) D 1 P 2 i =1 P 4 j =1 g G S (2 i 1) Err G (2 i 1 ;j ) g P t G ( j ) P 2 i =1 P 4 j =1 Err G (2 i 1 ;j ) g P t G ( j ) 1 P 2 i =1 P 4 j =1 g G S (2 i ) Err G (2 i;j ) g P t G ( j ) P 2 i =1 P 4 j =1 Err G (2 i;j ) g P t G ( j ) 1 C C C A (5.94) 166 Thus,theentropyofselfactionis: H A S ( t +1) = f G S ^ f P t G log 2 f G S ^ f P t G 1 f G S ^ f P t G log 2 1 f G S ^ f P t G = 2 X i =1 2 X j =1 ^ T OS ( i;j ) ^ f P A O ( j )log 2 2 X j =1 ^ T OS ( i;j ) ^ f P A O ( j ) (5.95) where: ^ A O = 0 B @ P 2 i =1 P 4 j =1 Err G (2 i 1 ;j ) f P t G ( j ) P 2 i =1 P 4 j =1 Err G (2 i;j ) f P t G ( j ) 1 C A (5.96) Theconditionalentropybetween A O ( t )and A S ( t +1)isgivenby: H A S ( t +1) A O ( t ) = X i 2f C;D g n P ^ A O ( t )= i o H ^ A S ( t +1) ^ A O ( t )= i = ^ ˇ 1 +^ ˇ 3 z }| { n P ^ A O ( t )= C o 2 X i =1 T OS ( i; 1)log 2 T OS ( i; 1) ^ ˇ 2 +^ ˇ 4 z }| { n P ^ A O ( t )= D o 2 X i =1 T OS ( i; 2)log 2 T OS ( i; 2) = 2 X i =1 2 X j =1 (^ ˇ i +^ ˇ i +2 ) ^ T OS ( j;i )log 2 ^ T OS ( j;i ) = 2 X i =1 2 X j =1 X k 2f i;i +2 g 4 X l =1 Err G ( k;l ) f P t G ( l ) ^ T OS ( j;i )log 2 ^ T OS ( j;i ) (5.97) Therefore,themutualinformationbetween A O ( t )and A S ( t +1)is: I ( A S ( t +1); A O ( t ))= H A S ( t +1) H A S ( t +1) A O ( t ) = 2 X i =1 2 X j =1 ^ T OS ( i;j ) ^ g P A O ( j )log 2 2 X j =1 ^ T OS ( i;j ) ^ g P A O ( j ) + 2 X i =1 2 X j =1 X k 2f i;i +2 g 4 X l =1 Err G ( k;l ) f P t G ( l ) ^ T OS ( j;i )log 2 ^ T OS ( j;i ) (5.98) 167 5.3.6.2Mutualinformationbetween A O ( t )and A O ( t +1) Tothemutualinformationbetween A O ( t +1)and A O ( t ),weshouldfollowthesame idea.Wehave: ^ T OO = 0 B @ CD CP O 1 ^ P A C S ( t ) A C O ( t ) + P O 2 ^ P A D S ( t ) A C O ( t ) P O 3 ^ P A C S ( t ) A D O ( t ) + P O 4 ^ P A D S ( t ) A D O ( t ) D 1 P O 1 ^ P A C S ( t ) A C O ( t ) + 1 P O 2 ^ P A D S ( t ) A C O ( t ) 1 P O 3 ^ P A C S ( t ) A D O ( t ) + 1 P O 4 ^ P A D S ( t ) A D O ( t ) 1 C A (5.99) or: ^ T OO = 0 B B B @ CD C P 2 i =1 P 4 j =1 g G O (2 i 1) Err G (2 i 1 ;j ) g P t G ( j ) P 2 i =1 P 4 j =1 Err G (2 i 1 ;j ) g P t G ( j ) P 2 i =1 P 4 j =1 g G O (2 i ) Err G (2 i;j ) g P t G ( j ) P 2 i =1 P 4 j =1 Err G (2 i;j ) g P t G ( j ) D 1 P 2 i =1 P 4 j =1 g G O (2 i 1) Err G (2 i 1 ;j ) g P t G ( j ) P 2 i =1 P 4 j =1 Err G (2 i 1 ;j ) g P t G ( j ) 1 P 2 i =1 P 4 j =1 g G O (2 i ) Err G (2 i;j ) g P t G ( j ) P 2 i =1 P 4 j =1 Err G (2 i;j ) g P t G ( j ) 1 C C C A (5.100) Thus,theentropyofopponent'sactionis: H A S ( t +1) = g G O ^ f P t G log 2 g G O ^ f P t G 1 g G O ^ f P t G log 2 1 g G O ^ f P t G = 2 X i =1 2 X j =1 ^ T OO ( i;j ) ^ f P A O ( j )log 2 2 X j =1 ^ T OO ( i;j ) ^ f P A O ( j ) (5.101) Theconditionalentropybetween A O ( t )and A o ( t +1)isgivenby: H A O ( t +1) A O ( t ) = X i 2f C;D g n P ^ A O ( t )= i o H ^ A O ( t +1) ^ A O ( t )= i = ^ ˇ 1 +^ ˇ 3 z }| { n P ^ A O ( t )= C o 2 X i =1 T OO ( i; 1)log 2 T OO ( i; 1) ^ ˇ 2 +^ ˇ 4 z }| { n P ^ A O ( t )= D o 2 X i =1 T OO ( i; 2)log 2 T OO ( i; 2) = 2 X i =1 2 X j =1 (^ ˇ i +^ ˇ i +2 ) ^ T OO ( j;i )log 2 ^ T OO ( j;i ) = 2 X i =1 2 X j =1 X k 2f i;i +2 g 4 X l =1 Err G ( k;l ) f P t G ( l ) ^ T OO ( j;i )log 2 ^ T OO ( j;i ) (5.102) 168 Therefore,themutualinformationbetween A O ( t )and A O ( t +1)is: I ( A O ( t +1); A O ( t ))= H A O ( t +1) H A O ( t +1) A O ( t ) = 2 X i =1 2 X j =1 ^ T OO ( i;j ) ^ g P A O ( j )log 2 2 X j =1 ^ T OO ( i;j ) ^ g P A O ( j ) + 2 X i =1 2 X j =1 X k 2f i;i +2 g 4 X l =1 Err G ( k;l ) f P t G ( l ) ^ T OO ( j;i )log 2 ^ T OO ( j;i ) (5.103) 5.3.6.3Mutualinformationbetween A O ( t )and G ( t +1) Thetransitionmatrixbetween A O ( t )and G ( t +1)is: ^ T OG = 0 B B B B B B B B B B B B B B B B B B B B B B B B B B B B @ P S 1 P O 1 ^ P A C S ( t ) A C O ( t ) + P S 3 P O 2 ^ P A D S ( t ) A C O ( t ) P S 2 P O 3 ^ P A C S ( t ) A D O ( t ) + P S 4 P O 4 ^ P A D S ( t ) A D O ( t ) P S 1 1 P O 1 ^ P A C S ( t ) A C O ( t ) + P S 3 1 P O 2 ^ P A D S ( t ) A C O ( t ) P S 2 1 P O 3 ^ P A C S ( t ) A D O ( t ) + P S 4 1 P O 4 ^ P A D S ( t ) A D O ( t ) 1 P S 1 P O 1 ^ P A C S ( t ) A C O ( t ) + 1 P S 3 P O 2 ^ P A D S ( t ) A C O ( t ) 1 P S 2 P O 3 ^ P A C S ( t ) A D O ( t ) + 1 P S 4 P O 4 ^ P A D S ( t ) A D O ( t ) 1 P S 1 1 P O 1 ^ P A C S ( t ) A C O ( t ) + 1 P S 3 1 P O 2 ^ P A D S ( t ) A C O ( t ) 1 P S 2 1 P O 3 ^ P A C S ( t ) A D O ( t ) + 1 P S 4 1 P O 4 ^ P A S S D ( t ) A D O ( t ) 1 C C C C C C C C C C C C C C C C C C C C C C C C C C C C A (5.104) or: ^ T OG = 0 B B B B B B B B @ CD CC P S 1 P O 1 ^ ˇ 1 + P S 3 P O 2 ^ ˇ 3 ^ ˇ 1 +^ ˇ 3 P S 2 P O 3 ^ ˇ 2 + P S 4 P O 4 ^ ˇ 4 ^ ˇ 2 +^ ˇ 4 CD P S 1 ( 1 P O 1 ) ^ ˇ 1 + P S 3 ( 1 P O 2 ) ^ ˇ 3 ^ ˇ 1 +^ ˇ 3 P S 2 ( 1 P O 3 ) ^ ˇ 2 + P S 4 ( 1 P O 4 ) ^ ˇ 4 ^ ˇ 2 +^ ˇ 4 DC ( 1 P S 1 ) P O 1 ^ ˇ 1 + ( 1 P S 3 ) P O 2 ^ ˇ 3 ^ ˇ 1 +^ ˇ 3 ( 1 P S 2 ) P O 3 ^ ˇ 2 + ( 1 P S 4 ) P O 4 ^ ˇ 4 ^ ˇ 2 +^ ˇ 4 DD ( 1 P S 1 )( 1 P O 1 ) ^ ˇ 1 + ( 1 P S 3 )( 1 P O 2 ) ^ ˇ 3 ^ ˇ 1 +^ ˇ 3 ( 1 P S 2 )( 1 P O 3 ) (^ ˇ 2 )+ ( 1 P S 4 )( 1 P O 4 ) ^ ˇ 4 ^ ˇ 2 +^ ˇ 4 1 C C C C C C C C A (5.105) 169 Wecanalsowritethistransitionmatrixas: ^ T OG = 0 B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B @ P S 1 P O 1 P 4 i =1 Err G (1 ;i ) g P t G ( i )+ P S 3 P O 2 P 4 i =1 Err G (3 ;i ) g P t G P 2 i =1 P 4 j =1 Err G (2 i 1 ;j ) g P t G ( j ) P S 2 P O 3 P 4 i =1 Err G (2 ;i ) g P t G ( i )+ P S 4 P O 4 P 4 i =1 Err G (4 ;i ) g P t G ( i ) P 2 i =1 P 4 j =1 Err G (2 i;j ) g P t G ( j ) P S 1 1 P O 1 P 4 i =1 Err G (1 ;i ) g P t G ( i )+ P S 3 1 P O 2 P 4 i =1 Err G (3 ;i ) g P t G ( i ) P 2 i =1 P 4 j =1 Err G (2 i 1 ;j ) g P t G ( j ) P S 2 1 P O 3 P 4 i =1 Err G (2 ;i ) g P t G ( i )+ P S 4 1 P O 4 P 4 i =1 Err G (4 ;i ) g P t G ( i ) P 2 i =1 P 4 j =1 Err G (2 i;j ) g P t G ( j ) 1 P S 1 P O 1 P 4 i =1 Err G (1 ;i ) g P t G ( i )+ 1 P S 3 P O 2 P 4 i =1 Err G (3 ;i ) g P t G ( i ) P 2 i =1 P 4 j =1 Err G (2 i 1 ;j ) g P t G ( j ) 1 P S 2 P O 3 P 4 i =1 Err G (2 ;i ) g P t G ( i )+ 1 P S 4 P O 4 P 4 i =1 Err G (4 ;i ) g P t G ( i ) P 2 i =1 P 4 j =1 Err G (2 i;j ) g P t G ( j ) 1 P S 1 1 P O 1 P 4 i =1 Err G (1 ;i ) g P t G ( i )+ 1 P S 3 1 P O 2 P 4 i =1 Err G (3 ;i ) g P t G ( i ) P 2 i =1 P 4 j =1 Err G (2 i 1 ;j ) g P t G ( j ) 1 P S 2 1 P O 3 P 4 i =1 Err G (2 ;i ) g P t G ( i )+ 1 P S 4 1 P O 4 P 4 i =1 Err G (4 ;i ) g P t G ( i ) P 2 i =1 P 4 j =1 Err G (2 i;j ) g P t G ( j ) 1 C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C A (5.106) Wehave: H G ( t +1) = 4 X i =1 2 X j =1 ^ T OG ( i ; j ) ^ A O ( j )log 2 4 X j =1 ^ T OG ( i ; j ) ^ A O ( j ) (5.107) Theconditionalentropybetween A O ( t )and G ( t +1)isgivenby: H G ( t +1) A O ( t ) = X i 2f C;D g ˆ P ^ A O ( t )= i ˙ H G ( t +1) ^ A O ( t )= i = X i 2f C;D g 4 X j =1 ˆ P ^ A O ( t )= i ˙ ^ T OG ( j ; i )log 2 ^ T OG ( j ; i ) (5.108) 170 Thus,themutualinformationbetween A O ( t )and G ( t +1)is: I G ( t +1); A O ( t ) = H G ( t +1) H G ( t +1) A O ( t ) = 4 X i =1 2 X j =1 ^ T OG ( i ; j ) ^ A O ( j )log 2 4 X j =1 ^ T OG ( i ; j ) ^ A O ( j ) + X i 2f C;D g 4 X j =1 ˆ P ^ A O ( t )= i ˙ ^ T OG ( j ; i )log 2 ^ T OG ( j ; i ) (5.109) Inthenextsection,wedesignanagentbasedmodelingandsimulatedevolutionarygameto checkthevalidityofourhypothesesonnecessityofcommunication,informationexchange, andtrustraisingforevolutionofsuper-reciprocalbehavior. 5.4ExperimentalSetup Computationalexperimentsweregenerallyperformedasawellmixedpopulation,giventhe spmutationrate(varies)andareplacementrate=0 : 01.However,1,000,000updates wereperformed,andthepaytablewasadaptedbychangingTaccordingly.Ateach update,everyplayeronthetoruslike32 32grid(withwrappingboundaryconditions) playseachofitsimmediateneighborsexactlyonce.Uponbirth,eachplayerbeginsby consultingits P C geneforeachopponent,andoneofthefourconditionalgenesthereafter, dependingonitsownplayandtheopponentsresponse.Playersareselectedforremoval randomlywithaprobabilitygivenbythereplacementrate r ,givingrisetooverlapping generations(asynchronousupdating)[ ? ,153,110].Aslongastheplayeranditsopponent arenotreplaced,theycontinuetoconsulttheirconditionalgenestomakedecisions,sothe replacementratedeterminestheaveragelengthofplayhistorybetweentwoplayers(ifa playerspartnerisreplaced,thepartnerisgreetedbyconsultingtheunconditionalgene).To 171 implementwell-mixedpopulationsusingourgridstructure,weonlychangedtheidentity ofthepoolusedforreplacingindividualsmarkedfordeath,thuskeepingtherestofthe dynamicsconsistent.Thepoolisgivenbyall1,023remainingstrategiesinthepopulation (inaMoranprocess,itisnotusualfortheindividualtobemarkedfordeathtobeincluded inthecandidatesforreplication).Thechosenplayerforreplacementisselectedrandomly amongall1,024playersinthepopulation,irrespectiveofpopulationstructureor Afterreplication,agenotypeismutatedwithaprobability m ,whichisthemeannumber ofmutationspergeneperindividual,implementedasaPoissonprocess.Formostofthe resultsinthisagentbasedmodeling,thegenesprobabilitiesarecoarse-grainedto15bits, whichmeansthattheprobabilitiesarechosenfromamong2 15 =32 ; 768possiblevalues, representingthenumberofpossibleallelesatthatlocus.Thisresolutionthecritical mutationrates.Sincethemutationprobabilitiesarethoughttorepresentthedecisionof entirepathwaysofperhapshundredsofgenes,theyshouldnotbecomparedtoper-nucleotide mutationratesRatherthancollectingpopulationaveragesofplays,weinsteadstudythe evolutionofstrategiesbyfollowingthelineofdescent(LoD)ofplayergenotypesforeach replicaterun.TheLoDisobtainedbychoosingarandomplayerattheendoftherunand followingitsdirectancestorsbackwardstothegenotype[137].Afterreconstruction theLoD,themeanprobabilitiestocooperateordefect( P CC ;P CD ;P DC ;P DD )aswellas thefractionofpastmovesencountered( ˇ CC ;ˇ CD ;ˇ DC ;ˇ DD )wereobtainedbytakingthe averagefromsimulationupdate900,000to950,000toexcludeanynoiseinthepopulation thatwouldbedtfromtheLoD.Thisremovesmostorallofthetransientandalso thevarianceduetopickingrandomgenotypesastheoriginatorsoftheLoD.Sincethe LoDsplitsatthemostrecentcommonancestor(MRCA)ofthepopulationattheend oftherun,theLoDpasttheMRCAisnotnecessarilyrepresentativeoftheevolutionary 172 dynamics.Thus,discardingthelast50,000updatestruncatestheLoDtogenotypesbefore theMRCAforalmostallruns.UsingtheMRCAgenotypeinsteadoftheconsensusgenotype asrepresentativeofthepointdoesnotchangetheresults. Additionally,environmentalnoisewasintroducedbyrandomlychangingwhatanagent rememberedaboutit'slastmoveandtheopponentsmove.Dependingonthenoiseproba- bility,whattheagentremembered(CorD)wasdrawnfromauniformrandomdistribution. Mutualinformationwasobtainedbyeithercomputationallysimulatingactualagentsplay thegame1000timesandmaking200moves.Ofthese200movesthe160werediscarded toeliminatefromarandommove.Mutualinformationwascalculatedfromthese plays.AlternativelymutualinformationwasdirectlycalculatedbyusingaMarkovprocess thatestimatedthecurrentandlastmoveprobabilitiesfortwointeractingagents.These estimatesweredirectlytranslatedintomutualinformation.Bothmethodshadequivalent results.ForallmainweusedtheMarkovprocessestimates.Computationalexper- imentsrequiredextensiveparametersweepsandmanyreplicates.Accurateassessmentof orderparametersandmutualinformationrequiresahugeamountofreplicates.Tokeep thecomputationalloadinareasonablerange(acoupleofCPUyears)weusedasmoothing function( MatLab smoothn[154])insteadofincreasingthenumberofreplicates,which wouldhavehadthesame 5.5Results Inthissection,theofmutationrate,temptationreward,andenvironmentalnoiseon evolutionofsecondarycooperationinouragentbasedmodelingofiteratedtrustdilemma isinvestigated.Here,thepopulationiswell-mixed,andwechangeeachofthementioned 173 factorstoseehowgamedynamicchanges.Werepeateachscenariotwentytimes.Later,we discusstheofthesefactorstogetheronevolutionofcooperationandsupercooperation inbothIPDandITD. 5.5.1ofTemptationPay Figure(5.2)showsstrategyofplayersoverlineofdescent.Eachsubplotrepresentsone genethatcorrespondsto CC , CD , DC ,and DD previousroundofgame.Eachcolor showstheevolutionofstrategiesforonetemptationpay( T =4 ;::: 9).Figure(5.3) Figure5.2:Evolvedstrategiesoverlineofdescent.LoDGenomestatistics( P X )averaged over20experiments(1,000,000generationseach)atlowmutationrate =0 : 007, r (1%)andttemptationrewards T 4 ;::: 9.Eachcolorrepresentsonetemptation reward(blue:4,red:5,green:6,black:7,cyan:8,andmagenta:9):(a) P 1 overline ofdescent,(b) P 2 overlineofdescent,(c) P 3 overlineofdescent,and(d) P 4 overlineof descent. showsthestatisticsofinteractionovergenerations.Similartothepreviouseach 174 subplotrepresentsonetypeofplayedgame( CC , CD , DC ,and DD ).Eachcolorshowsthe statisticsofplayedgamesforonetemptationpay( T =4 ;::: 9).Secondarycooperation Figure5.3:Gamestatisticsoverlineofdescent.LoDGenomestatistics( ˇ X )averagedover 20experiments(1,000,000generationseach)atlowmutationrate =0 : 007, r (1%)andttemptationrewards T 4 ;::: 9.Eachcolorrepresentsonetemptation reward(blue:4,red:5,green:6,black:7,cyan:8,andmagenta:9):(a) ˇ 1 overline ofdescent,(b) ˇ 2 overlineofdescent,(c) ˇ 3 overlineofdescent,and(d) ˇ 4 overlineof descent. canexistinstandardIPD(lowtemptationpay);however,thisstrategyusuallyhasa shortlifetimesinceitisassociatedwithaninitialphaseofstrategyequilibrationthata populationundergoes.Ontheotherhand,asweincreasethetemptationpaydominant strategyswitchesfromprimarycooperationtosecondarycooperation.Asweexpected,both genevaluesareveryclose0and1inITD(especiallyforhighertemptationrewards(e.g.8or 9))sincesecondarycooperationismuchmoreerrorreluctantthantheprimarycooperation. AnybreakintheexchangeofCDandDCmoveswillresultinaharshpaylosstoboth playersandpossiblylengthyperiodsoferrorcorrection. 175 5.5.2ofmutationrate Figure(5.4)and(5.5)showstatisticsofstrategiesoverlineofdescentfortemptationreward 5and10respectively.Eachsubplotrepresentsonegene( P 1 , P 2 , P 3 ,and P 4 ).Eachcolor showstheevolutionofstrategiesforonemutationrate( =0 : 007 ;::: 0 : 020).Figure(5.6) Figure5.4:Evolvedstrategiesoverlineofdescent.LoDGenomestatistics( P X )averaged over20experiments(1,000,000generationseach)attmutationrate =0 : 007 ::: 0 : 02, r (1%)andtemptationrewards T =5.Eachcolorrepresentsonemutationrate (blue:0.007,red:0.008,green:0.009,black:0.01,cyan:0.012,magenta:0.016,andyellow: 0.02):(a) P 1 overlineofdescent,(b) P 2 overlineofdescent,(c) P 3 overlineofdescent,and (d) P 4 overlineofdescent. and(5.7)showstatisticsofinteractionsbetweenagentsoverthecourseofevolution.Similar tothepreviouseachsubplotrepresentsonetypeofplayedgame( CC , CD , DC ,and DD ).Eachcolorshowsthestatisticsofplayedgamesforonemutationrate.Inthesewell- mixedpopulationsandfor T =10,itisobservedthatsecondarycooperationonlyappearsat lowmutationratesi.e. 1%.Forlowervaluesof T ,i.e.,6 T< 8,wecouldnotany 176 Figure5.5:Evolvedstrategiesoverlineofdescent.LoDGenomestatistics( P X )averaged over20experiments(1,000,000generationseach)attmutationrate =0 : 007 ::: 0 : 02, r (1%)andtemptationrewards T =10.Eachcolorrepresentsonemutationrate (blue:0.007,red:0.008,green:0.009,black:0.01,cyan:0.012,magenta:0.016,andyellow: 0.02):(a) P 1 overlineofdescent,(b) P 2 overlineofdescent,(c) P 3 overlineofdescent,and (d) P 4 overlineofdescent. 177 Figure5.6:Gamestatisticsoverlineofdescent.LoDGenomestatistics( ˇ X )averagedover 20experiments(1,000,000generationseach)attmutationrate =0 : 007 ::: 0 : 02, r (1%)andtemptationrewards T =5.Eachcolorrepresentsonemutationrate (blue:0.007,red:0.008,green:0.009,black:0.01,cyan:0.012,magenta:0.016,andyellow: 0.02):(a) P 1 overlineofdescent,(b) P 2 overlineofdescent,(c) P 3 overlineofdescent,and (d) P 4 overlineofdescent. 178 Figure5.7:Gamestatisticsoverlineofdescent.LoDGenomestatistics( ˇ X )averagedover 20experiments(1,000,000generationseach)attmutationrate =0 : 007 ::: 0 : 02, r (1%)andtemptationrewards T =10.Eachcolorrepresentsonemutationrate (blue:0.007,red:0.008,green:0.009,black:0.01,cyan:0.012,magenta:0.016,andyellow: 0.02):(a) P 1 overlineofdescent,(b) P 2 overlineofdescent,(c) P 3 overlineofdescent,and (d) P 4 overlineofdescent. 179 secondarycooperativebehavioreveninthislowmutationrate.Unlikeprimarycooperation, secondarycooperationisanimmenselycostlybehaviortoevolvesimplybecauseitrelieson thecoordinationoftwogenesratherthanone.Secondarycooperationisverysensitiveto mutationandundergoesacriticaltransitionat ˇ 1 : 5%for T =10todefection(Fig.5.7c andd). 5.5.3ofenvironmentalnoise Figure(5.8)and(5.9)showstatisticsofstrategiesoverlineofdescentinnoisyenvironments fortemptationreward5and10respectively.Here,weassumebothselfandopponentnoise areidentical.Eachsubplotrepresentsonegene( P 1 , P 2 , P 3 ,and P 4 ).Eachcolorshowsthe evolutionofstrategiesforonenoiselevel( =0 : 001 ;::: 0 : 007).Figure(5.10)and(5.11) showstatisticsofinteractionsbetweenagentsovercourseofevolutioninnoisyenvironments. Similartothepreviouses,eachsubplotrepresentsonetypeofplayedgame( CC , CD , DC ,and DD ).Eachcolorshowsthestatisticsofplayedgamesforoneenvironmental noiselevel.Accordingto(5.10)and(5.11),secondarycooperationismorerobustto theenvironmentalnoise.Onereasonisthehightemptationreward( T =10).Forlower temptationpaysthesensitivityofsecondarycooperationtoenvironmentalnoiseismore obvious.Weaddressthisissuelaterbyintroducingtheorderparameter. 5.6Discussion Stochasticstrategieswhichengageinsecondarycooperationareevolvableiniteratedtrust dilemmagame(increasingtemtationrewardtoavaluegreaterthan\2 R S ").Sincethese strategiesrelyonthecoordinationoftwogenes,comparetoprimarycooperationthese 180 Figure5.8:Evolvedstrategiesoverlineofdescentinnoisyenvironments.LoDGenome statistics( P X )averagedover20experiments(1,000,000generationseach)attnoise rates =0 : 001 ::: 0 : 007,lowmutationrate (0.7%), r (1%)andtemptation reward T =5(IPD).Eachcolorrepresentsoneenvironmentalnoiselevel(blue:0.001,red: 0.002,green:0.003,black:0.004,cyan:0.005,magenta:0.006,andyellow:0.007):(a) P 1 overlineofdescent,(b) P 2 overlineofdescent,(c) P 3 overlineofdescent,and(d) P 4 over lineofdescent. 181 Figure5.9:Evolvedstrategiesoverlineofdescent.LoDGenomestatistics( P X )averaged over20experiments(1,000,000generationseach)attnoiserates =0 : 001 ::: 0 : 007, lowmutationrate (0.7%), r (1%)andtemptationreward T =10(ITD). Eachcolorrepresentsoneenvironmentalnoiselevel(blue:0.001,red:0.002,green:0.003, black:0.004,cyan:0.005,magenta:0.006,andyellow:0.007):(a) P 1 overlineofdescent, (b) P 2 overlineofdescent,(c) P 3 overlineofdescent,and(d) P 4 overlineofdescent. 182 Figure5.10:Gamestatisticsoverlineofdescent.LoDGenomestatistics( ˇ X )averagedover 20experiments(1,000,000generationseach)attnoiserates =0 : 001 ::: 0 : 007, lowmutationrate (0.7%), r (1%)andtemptationreward T =5(IPD).Each colorrepresentsoneenvironmentalnoiselevel(blue:0.001,red:0.002,green:0.003,black: 0.004,cyan:0.005,magenta:0.006,andyellow:0.007):(a) ˇ 1 overlineofdescent,(b) ˇ 2 overlineofdescent,(c) ˇ 3 overlineofdescent,and(d) ˇ 4 overlineofdescent. 183 Figure5.11:Gamestatisticsoverlineofdescent.LoDGenomestatistics( ˇ Y )averagedover 20experiments(1,000,000generationseach)attnoiserates =0 : 001 ::: 0 : 007, lowmutationrate (0.7%), r (1%)andxedtemptationreward T =10(ITD).Each colorrepresentsoneenvironmentalnoiselevel(blue:0.001,red:0.002,green:0.003,black: 0.004,cyan:0.005,magenta:0.006,andyellow:0.007):(a) ˇ 1 overlineofdescent,(b) ˇ 2 overlineofdescent,(c) ˇ 3 overlineofdescent,and(d) ˇ 4 overlineofdescent. 184 strategiesaremorecostly,andtheyaremoresensitivetoenvironmentaldisturbancessuch asmutationrateandenvironmentalnoisethatctthecertaintywithwhichagentspredict thefuturestatesoftheenvironment.Primarycooperationisastrategythatismostly relyononegene( P 1 ),itisexpectedthatthisstrategybemorestablethanthesecondary cooperationundermutationalload.Temptationreward2 R S (withoursetup, T =6) isthecriticalturningpointfromprimarycooperationtosecondarycooperation,andfor temptationrewardsgreaterthanthiscriticalvalueprimarycooperationisneverfavored oversecondarycooperation;however,for T =2 R S primaryandsecondarycooperationas thetwodominantattractorsofthisgamewillalternate.Sincesecondarycooperationismore sensitivetomutationrateforsmalltemptationrewardscomparetoprimarycooperation,we expectthatfortemptationpayslightlygreaterthancriticalvalue\2 R S "secondary cooperationwouldnothavetpayadvantagetoovertakeamorerobuststrategy suchasprimarycooperation.Nonetheless,byincreasingmutationrateorenvironmental noisebothofthesestrategiesswitchtodefection.ThisphenomenonhappensearlierinITD forsmallertemptationrewards.Usingtheaverageplayfrequenciesineachround,wecan twoorderparameters m 1 and m 2 forIPDandITDrespectivelytodeterminethe criticalareaofthisphasetransition: m 1 = h ˇ 1 ih ˇ 4 i h ˇ 1 i + h ˇ 4 i (5.110) m 2 = h ˇ 2 i + h ˇ 3 ih ˇ 1 ih ˇ 4 i (5.111) 185 Theseparameterscrosszeroatacriticalmutationrateandenvironmentalnoise,indicating thedominantstrategiesinIPDandITD.Secondarycooperationisstableatlowmutation rateandlowenvironmentalnoise,andasweincreasetheseparametersitswitchestodefec- tion.Ahightemptationrewardprovideslesscosttosecondarycooperationandconsequently theuncertaintyoftheenvironment. Figures(5.12-5.15)and(5.16-5.19)demonstratetheofmutationrateandtemp- tationrewardonevolutionofstrategiesandstatisticsofinteractionsbetweenagentsover courseofevolutioninnoiselessenvironmentsforIPDandITD.Bothgamesdepicthow strategiesswitchtodefectionwhenweincreasethemutationrate.Moreover,these showhowsecondarycooperationbecomesmoreresistanttomutationrateasweincreasethe temptationreward. Figure5.12:ofmutationrateandtemptationpayonstrategiesinnoiselessenvi- ronmentsaveragedoverlast50,000generationsof20experimentsattmutationrates =0 : 007 ::: 0 : 5, r (1%)andttemptationrewards T =5 ::: 10.X-axisindi- catestemptationrewardandY-axisindicatesmutationrate,andZ-axisindicates P 1 over lineofdescent. Figure(5.20)and(5.21)presenttheorderparameteroneandtwoandcriticalturning 186 Figure5.13:ofmutationrateandtemptationpayonstrategiesinnoiselessenvi- ronmentsaveragedoverlast50,000generationsof20experimentsattmutationrates =0 : 007 ::: 0 : 5, r (1%)andttemptationrewards T =5 ::: 10.X-axisindi- catestemptationrewardandY-axisindicatesmutationrate,andZ-axisindicates P 2 over lineofdescent. Figure5.14:ofmutationrateandtemptationpayonstrategiesinnoiselessenvi- ronmentsaveragedoverlast50,000generationsof20experimentsattmutationrates =0 : 007 ::: 0 : 5, r (1%)andttemptationrewards T =5 ::: 10.X-axisindi- catestemptationrewardandY-axisindicatesmutationrate,andZ-axisindicates P 3 over lineofdescent. 187 Figure5.15:ofmutationrateandtemptationpayonstrategiesinnoiselessenvi- ronmentsaveragedoverlast50,000generationsof20experimentsattmutationrates =0 : 007 ::: 0 : 5, r (1%)andttemptationrewards T =5 ::: 10.X-axisindi- catestemptationrewardandY-axisindicatesmutationrate,andZ-axisindicates P 4 over lineofdescent. Figure5.16:ofmutationrateandtemptationpayonstatisticsofgamesinnoiseless environmentsaveragedoverlast50,000generationsof20experimentsattmutation rates =0 : 007 ::: 0 : 5, r (1%)andttemptationrewards T =5 ::: 10.X-axis indicatestemptationrewardandY-axisindicatesmutationrate,andZ-axisindicates ˇ 1 over lineofdescent. 188 Figure5.17:ofmutationrateandtemptationpayonstatisticsofgamesinnoiseless environmentsaveragedoverlast50,000generationsof20experimentsattmutation rates =0 : 007 ::: 0 : 5, r (1%)andttemptationrewards T =5 ::: 10.X-axis indicatestemptationrewardandY-axisindicatesmutationrate,andZ-axisindicates ˇ 2 over lineofdescent. Figure5.18:ofmutationrateandtemptationpayonstatisticsofgamesinnoiseless environmentsaveragedoverlast50,000generationsof20experimentsattmutation rates =0 : 007 ::: 0 : 5, r (1%)andttemptationrewards T =5 ::: 10.X-axis indicatestemptationrewardandY-axisindicatesmutationrate,andZ-axisindicates ˇ 3 over lineofdescent. 189 Figure5.19:ofmutationrateandtemptationpayonstatisticsofgamesinnoiseless environmentsaveragedoverlast50,000generationsof20experimentsattmutation rates =0 : 007 ::: 0 : 5, r (1%)andttemptationrewards T =5 ::: 10.X-axis indicatestemptationrewardandY-axisindicatesmutationrate,andZ-axisindicates ˇ 4 over lineofdescent. pointsofprimaryandsecondarycooperationtodefectionforIPDandITDrespectively. Thesedepicthowincreasingmutationratereducesthecooperativebehaviorinthe populationandhowbyincreasingtemptationrewardsecondarycooperationbecomesmore resistanttothemutationrateanduncertaintyintheenvironment. Figure(5.22)and(5.23)showtheofenvironmentalnoiseontheprimaryand secondatycooperationforIPDandITDrespectively.Similartopreviousthese exhibithowincreasingenvironmentalnoise( Self and Other )reducesthecooperative behaviorinthepopulationandhowbyincreasingtemptationrewardsecondarycooperation becomesmoreresistanttotheenvironmentalnoise. Fig.(5.22)and(5.23)showthatalthoughsecondarycooperationismoresensitiveto mutationrateanditishardtoestablishthisstrategyinpopulationwithhighmutationrate, itismorerobusttoenvironmentalnoisecomparetotheprimarycooperation.Thereason 190 Figure5.20:Orderparametronewhichindictaescriticalswitchingcurvefromprimary cooperationtodefectionforttemptationrewardandmutationrateinnoiselessen- vironmentsaveragedoverlast50,000generationsof20experimentsattmutation rates =0 : 007 ::: 0 : 5, r (1%)andttemptationrewards T =5 ::: 10.X-axis indicatestemptationrewardandY-axisindicatesmutationrate. Figure5.21:Orderparametrtwowhichindictaescriticalswitchingcurvefromsecondary cooperationtodefectionforttemptationrewardandmutationrateinnoiselessen- vironmentsaveragedoverlast50,000generationsof20experimentsattmutation rates =0 : 007 ::: 0 : 5, r (1%)andttemptationrewards T =5 ::: 10.X-axis indicatestemptationrewardandY-axisindicatesmutationrate. 191 Figure5.22:OrderparametersforprimaryandsecondarycooperationduringIPDandITD gamesinnoisyenvironments.Hereorderparametersindicatehowenvironmentalnoisecauses thephasetransitionfromcooperationtodefectioninIPDandITD.Orderparametrswere averagedoverlast50,000generationsof20experimentsatdtenvironmentalnoise levels =0 : 0 ::: 0 : 5, r (1%)andttemptationrewards T =4 ::: 10.X-axis indicatesenvironmentalnoiseandY-axisindicatedtemptationreward:(a) m 1 forIPD,(b) m 2 forITD. 192 Figure5.23:ofselfandothernoiseonprimaryandsecondarycooperationdurinfIPD andITD.Hereorderparametersindicatehowenvironmentalnoisecausesthephasetransition fromcooperationtodefectioninIPDandITD.Orderparametrswereaveragedoverlast 50,000generationsof20experimentsattenvironmentalnoiselevels =0 : 0 ::: 0 : 5, r (1%)and T =5forIPDand T =10forITD.X-axisindicatesenvironmentalnoise andY-axisindicatedtemptationreward:(a) m 1 forIPD,(b) m 2 forITD. 193 forbeingmorerobustcanbeinthedependencyofsecondarycooperationtotwogenes. Toestablishthisstrategy,weneedtoevolvetwogenes( P 2 and P 3 ),whichwillbehard ifthemutationrateishigh.However,assoonasthesegenesareevolved,environmental noisehashardertimetobreakthecoordinationofthesetwogenescomparetotheprimary cooperation,whichreliesonjust P 1 gene.Moreover,similartotheofmutationrate onsecondarycooperation,weseethatasweincreasethetemptationrewardtherobustness ofsecondarycooperationincreases. Figures(5.24-5.25)and(5.26-5.27)showtheofenvironmentalnoiseonmutualin- formationbetweenrandomvariablesattwoconsecutiveroundsofIPDandITDrespectively. Similartoprevioustheseuresexhibithowincreasingenvironmentalnoise( Self and Other )reducesthecommunicationbetweentwoplayersandconsequentlycausesswitch- ingfromcooperativebehaviortodefection.Interestingly,weseethatthephasetransition fromcooperationtodefectionoccursataconstantrateofmutualinformation.Figure (5.28)and(5.29)showthevariationofmutualinformationbetweengameplayedandaction chosenfromselfplayerinthenextroundofgame( I GS )vs.thevariationoforderparameter. Ourresultsshowthattheminimumrateofinformationformaintainngprimaryorsecondary cooperationinnoisyenvironmentisaround0 : 18bitsofinformationforbothgames.To seehow0.18bitsofinformationisenoughtomaintainprimaryandsecondarycooperation inIPDandITD,weapplytheinformationentropyapproximationas: I ( x )=1 H ( x ) Thismeansthat0.18bitsofinformationisequivalentto0.82entropyinthesystem.In otherwords,iftheentropyinthesystemis0.82,thediscriminationrateforsuchsystemis 194 Figure5.24:ofenvironmentalnoiseonmutualinformationbetweenrandomvariables duringtwoconsecutiveroundsofIPD.Thephasetransitionfromcooperationtodefection (determinedbyorderparameterone)anddropinmutualinformationhavethesamepattern. Mutualinformationbetweenrandomvariableswerecalculatedanalyticallyandestimated fromagentbasedsimulationattenvironmentalnoiselevels =0 : 0 ::: 0 : 5, r (1%)andtemptationrewards T =5.X-axisindicatesenvironmentalnoiseandY-axis indicatestemptationreward,andZ-axisindicatesthemutualinformationbetweenrandom variables:(a) I GG ,(b) I GS . 195 Figure5.25:ofenvironmentalnoiseonmutualinformationbetweenrandomvariables duringtwoconsecutiveroundsofIPD.Thephasetransitionfromcooperationtodefection (determinedbyorderparameterone)anddropinmutualinformationhavethesamepattern. Mutualinformationbetweenrandomvariableswerecalculatedanalyticallyandestimated fromagentbasedsimulationattenvironmentalnoiselevels =0 : 0 ::: 0 : 5, r (1%)andtemptationrewards T =5.X-axisindicatesenvironmentalnoiseandY-axis indicatestemptationreward,andZ-axisindicatesthemutualinformationbetweenrandom variables:(a) I SG ,(b) I SS . 196 Figure5.26:ofenvironmentalnoiseonmutualinformationbetweenrandomvariables duringtwoconsecutiveroundsofITD.Thephasetransitionfromcooperationtodefection (determinedbyorderparametertwo)anddropinmutualinformationhavethesamepattern. Mutualinformationbetweenrandomvariableswerecalculatedanalyticallyandestimated fromagentbasedsimulationattenvironmentalnoiselevels =0 : 0 ::: 0 : 5, r (1%)andtemptationrewards T =7.X-axisindicatesenvironmentalnoiseandY-axis indicatestemptationreward,andZ-axisindicatesthemutualinformationbetweenrandom variables:(a) I GG ,(b) I GS . 197 Figure5.27:ofenvironmentalnoiseonmutualinformationbetweenrandomvariables duringtwoconsecutiveroundsofITD.Thephasetransitionfromcooperationtodefection (determinedbyorderparametertwo)anddropinmutualinformationhavethesamepattern. Mutualinformationbetweenrandomvariableswerecalculatedanalyticallyandestimated fromagentbasedsimulationattenvironmentalnoiselevels =0 : 0 ::: 0 : 5, r (1%)andtemptationrewards T =7.X-axisindicatesenvironmentalnoiseandY-axis indicatestemptationreward,andZ-axisindicatesthemutualinformationbetweenrandom variables:(a) I SG ,(b) I SS . 198 Figure5.28:Mutualinformationbetweengameplayedandactionselectedbyplayerin thenextroundandorderparameteroneduringroundsofIPDinnoisyenvironmentswith tnoiselevel.Bluelineistheorderparameteroneandtheredlineis I GS .Theblack dashedlineindicatesthephasetransitionandcriticalmutualinformationduringthephase transition. Figure5.29:Mutualinformationbetweengameplayedandactionselectedbyplayerin thenextroundandorderparameteroneduringroundsofITDinnoisyenvironmentswith tnoiselevel.Bluelineistheorderparameteroneandtheredlineis I GS .Theblack dashedlineindicatesthephasetransitionandcriticalmutualinformationduringthephase transition. 199 around0.7445(thisisshownin(5.30)),whichisahighdiscriminationrate.Insum, Figure5.30:Entropy H ( x ).The0.82entropyisequivalentto0.7445discriminationrate. increasingthemutationrateandenvironmentalnoisetohighervaluesleadstoatransition intodefection.ThisphasetransitionforITDoccursatamuchlowergenomicmutationrate thanwhatisobseredinIPD.Thiscouldbeowedtothefragilityofsecondarycooperation whenitcomestomutation,asisimpartedtoitbyitsdependenceonmoregenes.Thepay advantageofsecondarycooperationneedstobetlyhigh( T 8)toallowforthis strategytobecomethedominantstrategy(Fig.(5.31)).Ahighmutationalloadreducesthe stabilityofstrategieswhicharedependenttomorethanonegenes(i.e.secondarycooper- ation).Ontheotherhand,innoisyenvironmentssecondarycooperationismorerobustto environmentalnoiseincomparisontoprimarycooperation,sincethisstrategyreliesonthe coordinationoftwogenesofplayers,andduringroundsofITDtheseplayerscomunicate duringtheirinteraction.Thiscommunicationisveryrobusttoenvironmentalnoiseand comparetoprimarycooperationitcanexistsinverynoisyenvironments.Forenvironments thatarehighlypredictable(i.e.lowmutationrate),andwherethereisnopayadvantage toengageinsecondarycooperation,primarycooperationremainsthedominantstrategy. 200 Figure5.31:Qualitativephasetransitiondiagranasafunctionof andtemptationreward. Blacksolidlinesrepresentthephasetransitioncuresinnoiselessenvironmentandreddashed linesrepresentthephasetransitioncuresinnoisyenvironments. 5.7ChapterSummary Inthischapter,weshowedthatstochasticstrategiesengaginginsecondarycooperationcan beevolvedbyincreasingthetemptationrewards.Suchstrategiesrelyonthesynchronization oftwogenesandarethusmuchmoresensitivetoenvironmentalchangesthatthecer- taintyofplayersinpredictingfutureopponents'moves.Here,weinvestigatedthetransitions fromsecondarycooperationtodefectionatcriticalmutationrate,environmentalnoise,and temptationrewards.Abiologicalorsocialanaloguetosecondarycooperationhasnotyet beenformallyinvestigated.Itisthoughnothardtoimaginesuchabehaviorexistinginthe economicmarkets.Inparticular,duringtheemergenceofstockpriceleadingto thecreationofstockmarketbubbles,theunderlyingagreementbetweeninvestorsconstantly sellingandre-buyingthesamestockisreminiscentofsecondarycooperation.Theinvloved 201 partiesagreetotaketurnsinreceivingcostbybuyingacertainstock,withthepromisethat theywillbeabletosellthatstockatanincreasedvalueinthenearfuture.Thisgamein realityinvolvesmorethantwopartiesbutthetypeofobservedcooperationisverysimilar tosecondarycooperation. 202 Chapter6 GameTheoreticModelforSocial BehaviorinAnimals Theonlyencebetweenataxmanandataxidermistis thatthetaxidermistleavestheskin. -MarkTwain [Onfortaxreturns]Thisistooforamathe- matician.Ittakesaphilosopher. -AlbertEinstein -\Everybody"hastopaytaxes!Evenbusinessmen,thatrob andstealandcheatfrompeopleeveryday,even\they"have topay\taxes"! -LenniePike It'saMad,Mad,Mad,MadWorld,S.Kramer,1963 Theevolutionofaltruismhasbeenafascinatingconundrumforevolutionarybiologists, andscientistshavebeentryingtodiscoverthemechanismsthatmaintaincooperationsince thesebehaviorsarevulnerabletoexistenceofcheaters[155,156,157,158].One exampleofthisdilemmaisthe\tragedyofthecommons",whereindividualscaneither contributetoapublicgoodorabstain[114,116,158].Sincethepublicgoodisdistributed toeveryoneregardlessofaninvestmentornot,thoseindividualswhoabstainwillalways farebetter,unlessthepublicgoodsmultiplier(thesynergy)isunrealisticallyhigh[159]. Evolutionarygametheoryasaquantitativeframeworkforanalyzingtherationalbehavior ofspecieshasbeenusedtoaddresshowpublicgoodsisproducedamongindividuals. Severalevolutionarygametheoreticmechanismshavebeenproposedtohelpusunderstand thecollectivebehaviorofanimals[155,156,160,158,157,159,161].Amongtheproposed 203 mechanism,punishment(especiallyaltruisticpunishment[127])canexplainbetterhow cooperationpersistsamongindividuals[162,127,163,164,165,166,167]. Whileitispossibleinprincipletoenforcecooperationviacostlypunishmentasasignaling mechanism[133],itisshownthateventhoughpunishmentshiftsthecooperationregimeto emergeatlowersynergythresholds,individualsevolvetobecomepurecooperatorswhodo notpunish.Inotherwords,punishmentlosesitscommunicationpotentialinpopulations ofallcooperators.Thismeansthatthe threat ofpunishmentaloneisttodrive cooperation[115],whileactualpunishmentisrare.Onthecontrary,innaturalsystems weoftenobservevigorouspunishment[168].Anotherbetweenthepublicgoods gameand(some)naturalsystemsisthatinthepublicgoodsgamethepayisalinear functionoftheamountpaidin,whileinnaturalsystemsoftena threshold determineshow muchindividualsneedtocontributeinordertoobtainapaythatdoesnotdepend onthenumberofcontributorswhoultimatelytriggerit(morewolveshuntinginapackdoes notincreasethesizeofthemoose:youeithergetthemooseoryoudon't)[169,157]. Toremedythesediscrepanciesbetweenthepublicgoodsgameandthegame-theoretic dilemmaobservedforexampleincollectivehuntingstrategies,inthischapterweintroduce the\collectivehuntinggame"asanextensiontopublicgoodsgamewhereapayis distributedequallyonceenoughindividualscooperate(numberofcontributors threshold). Thisapproachissimilartotheproposedmodelsin[169,157];however,hereinsteadof alinearessfunctionweuseaHeavisidestepfunctiontomakethismodelapplicableto grouphuntingpatternsofhyenas.Thisalterationchangesthegamedramatically.Here,if theinvestmentinthepublicgoodsexceedsagiventhreshold,everyone(contributorsaswell ascheaters)receivesanequalshare r n ,where r and n arethepayandthenumber ofplayersinthegamerespectively,whileiftheinvestmentdoesnotreachthethreshold, 204 theinvestmentislost.Wethatpunishment,asanecommunicationmechanism, emergesasastablestrategyinthisgame,whichinreturnfacilitatescooperation.This suggeststhatthe\collectivehuntinggame"isabettermodelfornaturalgamblesthatare \all-or-nothing",andinwhichthetriggeredpaydoesnotdependonalinearfashionin thenumberofinvestors. 6.1Populationdynamicofpublicgoodsgame Similartoiteratedtwoplayersgame,thepopulationdynamicofsuchpopulationfollowsthe replicatorequation.Here,punishmentcanbeseenastheassortmentterm\ q "inconditional iteratedgamesdiscussedinchapter4.Inthisgame,therearetwotypesofplayers:coopera- torsorinvestorsanddefectorsorfreeriders.Moreover,theseplayerspunishfreeriderswith probabilities\ pc "and\ pd ".Cooperatorspunishfreeriderswiththeprobabilityof\ pc ", anddefectorspunishfreeriders(theirowntype)withtheprobabilityof\ pd ".Punishment asanaggressivebehaviorisacostlyaction.Playersacceptacosttopunishthefreeriders andreducetheirpayasaresultofpunishment.Themeanpayofeachtypeis: ˇ C = r k +1 ( kˆ c +1) pc (1 ˆ c ) ˇ D = rkˆ c k +1 pc ˆ c pd (1 ˆ c ) pd (1 ˆ c )(6.1) where r issynergyfactor, isthecosttothepunishment, istheofpunishmentto thefreeriders,and k isthenumberofneighborsaroundthecentralplayerina k +1game. Here, r and k aregameparametersandconstantforagivengame.However, and arethe strategyparameters.Thus,playerhasitsown pc (or pd iftheplayerisdefector), ,and 205 ,whichareconstantduringtheplayer'slifetime.Theseparameterscanbetfrom playertoplayer,andtheyinheritfromancestorstodescendant;therefore,theseparameters areevolvableduring.Having ˆ c , pc ,and pd isenoughtocalculatetheconcentrationof eachtypeofplayerswithitsknown and .Wehave: w = ˆ c ˇ C + ˆ d ˇ D = ˆ c ˇ C +(1 ˆ c ) ˇ D = ˇ D + ˆ c ( ˇ C ˇ D )(6.2) ˇ C w = ˇ C ˇ D ˆ c ( ˇ C ˇ D ) =(1 ˆ c )( ˇ C ˇ D )(6.3) ˇ D w = ˇ D ˇ D + ˆ c ( ˇ C ˇ D ) = ˆ c ( ˇ C ˇ D )(6.4) Thus,thereplicatorequationforpunishingcooperatoranddefectorwouldbe: _ ˆ c = ˆ c ( ˇ C w ) = ˆ c (1 ˆ c )( ˇ C ˇ D ) _ ˆ d = ˆ d ( ˇ D w )= _ ˆ c = ˆ c (1 ˆ c )( ˇ C ˇ D )(6.5) 206 ˇ C ˇ D inequation(6.5)canbewrittenas: ˇ C ˇ D = r k +1 1 pc pd (1 ˆ c )+ ˆ c pc pd + pd (6.6) Therefore,thereplicatorequation(6.5)canbewrittenas: _ ˆ c = ˆ c (1 ˆ c ) r k +1 1 pc pd (1 ˆ c )+ ˆ c pc pd + pd _ ˆ d = ˆ c (1 ˆ c ) r k +1 1 pc pd (1 ˆ c )+ ˆ c pc pd + pd (6.7) Ifdefectorspunishequallyascooperators( pc = pd ),thereplicatorequation(6.7)is to: _ ˆ c = ˆ c (1 ˆ c ) r k +1 1+ pc _ ˆ d = ˆ c (1 ˆ c ) r k +1 +1 pc (6.8) Replicatorequation(6.8)showsthatintheabsenceofpunishment( pc =0)ifsynergy factor r isgreaterthanthethegroupsize k +1,cooperationisevolutionarystableandit isworthtocooperate.Figure(6.1)showsthisphasetransitionforstandardpublicgoods game.Here,mixedstrategydoesnotexist,andwehavetwopointsas: ˆ c =0,defection ˆ c =1,cooperation(6.9) 207 Figure6.1:Phasetransitioninstandardpublicgoodsgameasafunctionofsynergyfactor. Bytakingthederivativeofequation(6.8)respectto ˆ c ,wecandeterminethestabilityof eachpoint.Wehave: d _ ˆ c dˆ c =(1 ˆ c ) r k +1 1+ pc ˆ c r k +1 1+ pc (6.10) Accordingtoequation(6.10),cooperationisevolutionarystableifandonlyif: r 1 pc ( k +1)(6.11) Defectionisevolutionarystableifandonlyif: r< 1 pc ( k +1)(6.12) 208 If pc 6 = pd ,wehavethreepointsincludingthemixedstrategy: ˆ c =0,defection ˆ c =1,cooperation ˆ c = 1 ( + ) pc pd 1+ pc pd r k +1 pd ,mixed(6.13) Again,bytakingthederivativeofequation(6.7)respectto ˆ c ,wecandeterminethestability ofeachpoint.Wehave: d _ ˆ c dˆ c =(1 ˆ c ) r k +1 1 pc pd (1 ˆ c )+ ˆ c pc pd + pd ˆ c r k +1 1 pc pd (1 ˆ c )+ ˆ c pc pd + pd +( + ) pc pd ˆ c (1 ˆ c ) (6.14) Accordingtoequation(6.14),cooperationisevolutionarystableifandonlyif: r 1 pc ( k +1)(6.15) Defectionisevolutionarystableifandonlyif: r< 1+ pc pd pd ( k +1)(6.16) Mixedstrategyisevolutionarystableifandonlyif: pc < pd (6.17) 209 whichmeansthatdefectorsshouldpunishtheirowntypemorefrequentincomparisonto cooperators. Ifdefectorsaresmartenoughtoavoidpunishingtheirowntype( pd =0),thereplicator equationforpunishingcooperatorandpuredefectoris: _ ˆ c = ˆ c ( ˇ C w ) = ˆ c (1 ˆ c ) r k +1 1+ pc ( ˆ c ( + ) ) _ ˆ d = ˆ d ( ˇ D w )= _ ˆ c = ˆ c (1 ˆ c ) r k +1 +1 pc ( ˆ c ( + ) ) (6.18) Since pd =0,mixedstrategypointisunstable.Cooperationisevolutionarystableif andonlyif: r 1 pc ( k +1)(6.19) Defectionisevolutionarystableifandonlyif: r< 1+ pc ( k +1)(6.20) 6.1.1Paymatrixrepresentationofthepublicgoodsgame Inthissectionthepaymatrixrepresentationofpublicgoodsgameisgiven.Sinceinthis chapterwemostlyfocusonthematrixrepresentationofthe\Collectivehuntinggame"as ourmodeltoanalyzehyena'shuntingpatterns,itisgoodtohavethematrixrepresentation ofstandardpublicgoodsgameinthissection.Todoso,wehavetocalculatethe 210 payofeachstrategyagainstsolidpopulationofothertypes.Formoredetails,pleaserefer to[169,157,115].First,weignorepunishment,andwecalculatethepaymatrixforgames inwhichthereisnopunishment,andinthisgamedefectorsorfreeridersdonotreceiveany negativecostorsignalfromthecooperativeinvestors.Thepaytoacooperatoragainsta solidpopulationofcooperatorsis: P c c = r 1 Thepaytoacooperatoragainstasolidpopulationofdefectorsis: P c d = r k +1 1 Afterrepeatingthesameprocessfordefector,thepaymatrixforpublicgoodsgameis: E = 0 B @ CD Cr 1 r k +1 1 D rk k +1 0 1 C A (6.21) Columnwiseoperationsonthispaymatrixgivesthefollowinganti-diagonalmatrix: E = 0 B @ CD C 0 r k +1 1 D 1 r k +1 0 1 C A (6.22) 211 Thispaymatrixgivesthefollowingreplicatorequation,whichisobviouslyequivalentto equation(6.7),whenthereisnopunishment. _ ˆ c = ˆ c (1 ˆ c ) r k +1 1 _ ˆ d = ˆ d (1 ˆ d ) r k +1 +1 (6.23) Byintroducingpunishmenttothegame,cooperatorsanddefectorsactasmoralistsand immoralists[133]withprobabilityofpunishment pc and pd .Moralistsarecooperators thatpunishandreceiveanadditionalcostforpunishment[115,170,171,172].Ontheother hands,immoralistsarefoolishdefectorsthatpunishtheirowntype.Byusingpay(6.1), wecanwritethepaymatrixofthestandardpublicgoodsgameofpunishingcooperators anddefectorsas: E = 0 B @ CD Cr 1 r k +1 1 pc D rk k +1 pc pd ( + ) 1 C A (6.24) Columnwiseoperationonthispaymatrixgivesthefollowinganti-diagonalmatrix: E = 0 B @ CD C 0 r k +1 + pd ( + ) pc 1 D 1 r k +1 pc 0 1 C A (6.25) Byanalyzingthisanti-diagonalpaymatrix,weseethatthestabilityconditionsforpunish- ingcooperatoranddefectorareidenticalto(6.15)and(6.16).Byignoringinsaneimmoralists 212 inthisgame,thepaymatrixforthisnewgamewouldbe: E = 0 B @ CD Cr 1 r k +1 1 pc D rk k +1 pc 0 1 C A (6.26) Columnwiseoperationonthispaymatrixgivesthefollowinganti-diagonalmatrix: E = 0 B @ CD C 0 r k +1 pc 1 D 1 r k +1 pc 0 1 C A (6.27) Byanalyzingthisanti-diagonalpaymatrix,weseethatthestabilityconditionsforpun- ishingcooperatorandpuredefectorareidenticalto(6.19)and(6.20). 6.1.2Punishmentasasignalingmechanismbetweencooperators anddefectorsinPGG Here,wetrytoquantifytheamountofinformationthatpunishingcooperatorgainsabout thepopulationbypunishingfreeridersoverroundsofstandardpublicgoodsgame.The mainpurposeofthissectionistoshowthatpunishmentontopofitsdirectonthe ofplayers,similartoassortmentinconditionalgames,isacommunicationmechanism thathelpscooperatorsgaininformationfromtheenvironment.Thismechanismconveyin- formationtothegamebydiscriminatingbetweendefectorsandcooperators.Inotherwords, punishmentisaconditionalstrategyontheopponent'stype.Withoutpunishmentitis impossibleforplayerswithunconditionalstrategiestoexchangeinformationanddistinguish 213 betweenfreeridersandinvestorsinthegroup.Similartoassortment,aperfectpunishment strategyprotectscooperatorsagainstanyinvasionoffreeriders.Themorectivecoop- eratorspunishthefreeriders,themoreinformationtheygainaboutthepopulationand theiropponents'type,andconsequentlythemoreimmunetheywillbeagainstdefectors. Byaddingerrorintothiscommunicationchannel,wetrytoanalyzetheofnoiseon punishmentandconsequentlyonfateofthepublicgoodsgame.Similartoprevioussection, weconsidertwopunishingscenarios:bothcooperatorsanddefectorspunishfreeriders, andsecond,onlycooperatorspunishfreeriders.Althoughthepunishingdefector(immoral- ist)reducesthepaytothedefectorinoneinteraction,itmightgainmorepayinlong runbyfoolingvigilant/punishingcooperators. 6.1.2.1Communicationbetweenpunishingcooperatorandpuredefectorover roundsofpublicgoodsgameinnoiselessenvironment Sincedefectorsdonotpunishtheirowntypeandtheyplayunconditionally,wejustcon- siderthebehaviororstrategyofcooperatorsagainsttheiropponents.Here,actionisnotto cooperateortodefect.Inthissection,weareinterestedhowpunishmentprovidesacommu- nicationchannelduringroundsofpublicgoodsgame;thus,here,strategyistopunish(P) ortoforgive(F).Itisobviousthatcooperatorsdonotpunishcooperators.Theyjustpunish defectorswithprobability pc ,andthisbringdiscriminationintothegame.Thestrategy matrixofpunishingcooperatorandpuredefectorcanbewrittenas: S C = 0 B @ CD Pp P j C p P j D Fp F j C p F j D 1 C A = 0 B @ CD P 0 pc F 11 pc 1 C A (6.28) 214 S D = 0 B @ CD Pp P j C p P j D Fp F j C p F j D 1 C A = 0 B @ PCD P 00 F 11 1 C A (6.29) where p P j X istheprobabilityofpunishmentiftheopponent'stypeis\ X ",and p F j X isthe probabilityofforgivenessiftheopponent'stypeis\ X ".Basedonthesestrategymatrices, theprobabilityofpunishment(P)andforgiveness(F)forpunishingcooperatoris: P F C = ˆ c + 1 pc (1 ˆ c ) P P C = pc (1 ˆ c )(6.30) Consequently,byusingequation(6.30)theentropyintheactionofpunishingcooperator ( A C )is: H ( A C )= ˆ c + 1 pc (1 ˆ c ) log ˆ c + 1 pc (1 ˆ c ) pc (1 ˆ c )log pc (1 ˆ c ) (6.31) Similarly,theconditionalentropyofpunishingcooperator'sactiongiventheopponent'stype is: H A C O = X j 2f C;D g ˆ j H A C O = j = (1 ˆ c ) pc log pc + 1 pc log 1 pc (6.32) 215 Figure6.2:ofpunishmentprobabilityonmutualinformationbetweenactionofpun- ishingcooperatorandopponent'stype.X-axisispunishmentprobability( pc ),Y-axisis theconcentrationofpunishingcooperationinpopulation,andZ-axisismutualinformation betweenactionofpunishingcooperatorandopponenttype. Therefore,themutualinformationbetweentheactionofpunishingcooperatorandoppo- nent'stypeis: I ( A C ; O )= H ( A C ) H A C O = ˆ c + 1 pc (1 ˆ c ) log ˆ c + 1 pc (1 ˆ c ) pc (1 ˆ c )log pc (1 ˆ c ) +(1 ˆ c ) pc log pc + 1 pc log 1 pc (6.33) Figure(6.2)showsthismutualinformationfort pc andtconcentrationof punishingcooperator.Accordingto(6.2),itisobviousthatthemutualinformation betweenpunishingprobabilityofcooperatorandavailableinformationforpunishingcooper- 216 ator,whichisthemutualinformationbetweenpunishingparameter pc andconcentration ofplayers,increasesasweincreasethepunishingprobability pc .Themutualinformationis maximumwhenpopulationisdividedequallytocooperatorsanddefectors.Ifplayersplay unconditionally( pc =0),thereisnoinformationtogain,andasaresult,ifsynergyfactor \ r "islow,cooperatorisvulnerabletoexistenceofdefectorsinthepopulation. 6.1.2.2Communicationbetweenpunishingcooperatorandpuredefectorover roundsofpublicgoodsgameinnoisyenvironment Tomodeltheofenvironmentalnoiseontheactionofpunishingcooperator,we thefollowingerrormatrix: = 0 B @ CD C 1 D 1 1 C A (6.34) Errormatrixshowshowpunishingcooperatorestimatesthefrequencyofinteractingwith eachtypeoverroundsofPGG.Byapplyingthiserrormatrixtotheactualconcentrationof players,theestimatedfrequencieswouldbe: 0 B @ ^ ˆ c ^ ˆ d 1 C A = 0 B @ 1 1 1 C A 0 B @ ˆ c 1 ˆ c 1 C A = 0 B @ + ˆ c 2 c 1 ˆ c +2 c 1 C A (6.35) 217 Bypluggingestimatedconcentrations(6.35)intoequation(6.2),wecancalculatethemu- tualinformationbetweenactionofpunishingcooperatorandopponent'stype.Figure(6.3) showsthismutualinformationfort p andconcentrationofpunishingcooperatorin environmentswithtlevelofnoise. 6.1.2.3Communicationbetweenpunishingcooperatoranddefectorinpublic goodsgame Inthissection,weareinterestedhowplayerswithpunishingattitudecommunicateand exchangeinformationoverroundsofpublicgoodsgame.Inthissection,bothcooperators anddefectorspunishfreeriders.Punishingdefectorsbydefectorsseemstobeafoolish strategy;however,ifthetofpunishmentfordefectorsisnotthathigh,defectorscanfool vigilantcooperatorsbypunishingtheirowntype(especiallyinnoisyenvironmentsinwhich playersmakemistakeinrecognizingthetypeoftheiropponent)andacceptasmallpenalty tothehopeofreceivinglongtermbfromcooperators.Similartoprevioussection, herestrategyistopunish(P)ortoforgive(F).Itisobviousthatcooperatorsdonotreceive punishmentfrombothtype.Ontheotherhand,defectorsarepenalizedwiththeprobability pc and pd bycooperatorsanddefectorsrespectively,andthisbringdiscriminationintothe game.Thestrategymatrixofpunishingcooperatoranddefectorcanbewrittenas: S C = 0 B @ CD Pp P j C p P j D Fp F j C p F j D 1 C A = 0 B @ CD P 0 pc F 11 pc 1 C A (6.36) 218 Figure6.3:ofassortmentandenvironmentalnoiseonmutualinformationbetween actionofconditionalcooperatorandopponenttype.X-axisispunishmentprobability( p c ), Y-axisistheconcentrationofpunishingcooperationinpopulation,andZ-axisismutual informationbetweenactionofpunishingcooperatorandopponent'stype:(a)onepercent noise,(b)twopercentnoise,(c)epercentnoise,(d)tenpercentnoise,(e)twentye percentnoise,and(f)ypercentnoise. 219 S D = 0 B @ CD Pp P j C p P j D Fp F j C p F j D 1 C A = 0 B @ CD P 0 pd F 11 pd 1 C A (6.37) Basedonthesestrategymatrices,theprobabilityofpunishment(P)andforgiveness(F)for punishingcooperatoris: P F C = ˆ c + 1 pc (1 ˆ c ) P P C = pc (1 ˆ c )(6.38) P F D = ˆ c + 1 pd (1 ˆ c ) P P D = pd (1 ˆ c )(6.39) Consequently,byusingequations(6.38)and(6.39),wecancalculatetheentropyofplayers' actionsas: H ( A C )= ˆ c + 1 pc (1 ˆ c ) log ˆ c + 1 pc (1 ˆ c ) pc (1 ˆ c )log pc (1 ˆ c ) (6.40) H ( A D )= ˆ c + 1 pd (1 ˆ c ) log ˆ c + 1 pd (1 ˆ c ) pd (1 ˆ c )log pd (1 ˆ c ) (6.41) 220 Similarly,theconditionalentropyofactiongiventypeofopponentsforeachplayeris: H A C O = X j 2f C;D g ˆ j H A C O = j = (1 ˆ c ) pc log pc + 1 pc log 1 pc (6.42) H A D O = X j 2f C;D g ˆ j H A D O = j = (1 ˆ c ) pd log pd + 1 pd log 1 pd (6.43) Therefore,themutualinformationbetweenactionofpunishingplayersandopponent'stype is: I ( A C ; O )= H ( A C ) H A C O = ˆ c + 1 pc (1 ˆ c ) log ˆ c + 1 pc (1 ˆ c ) pc (1 ˆ c )log pc (1 ˆ c ) +(1 ˆ c ) pc log pc + 1 pc log 1 pc (6.44) I ( A D ; O )= H ( A D ) H A D O = ˆ c + 1 pd (1 ˆ c ) log ˆ c + 1 pd (1 ˆ c ) pd (1 ˆ c )log pd (1 ˆ c ) +(1 ˆ c ) pd log pd + 1 pd log 1 pd (6.45) 221 Figure(6.4)showsthismutualinformationfort pc , pd ,andconcentrationofpun- ishingcooperator.Similartoprevioussection,wecanapplytheerrormatrixtoestimate theconcentrationofplayers,andbypluggingtheseestimatedvaluesintoequations(6.44) and(6.45),wecancalculatethemutualinformationbetweenactionofvigilant/punishing playersandtypeofopponent. 6.2Collectivehuntinggame Oneexampleofubiquitousyetvulnerablecooperativebehavioramongcheateris grouphuntingofmammalssuchasspottedhyenasintheirdespoticsocieties,wherethe highestrankingindividualsobtainthelargestshareofthekill[173].Thehuntingpatterns ofspottedhyenasisveryinteresting.Thesemammalshuntmostlocallyabundantungulate species,andtheirhuntinggroupsizevarieswithpreytype.Approximately,onlyone-third ofhuntingattemptsresultinpreycapture.Thesuccessrateisafunctionofthehyenas' age,experience,andthegroupsize[174].Theymostlyprefertodosolitaryhunting; however,theyshowsomeopportunisticbehaviorinattendingeasyhunts.Thekillsizeis morethanwhathyenaneeds,andthisisbtotheclan.Aftertheykilltheprey(1-3 hyenaperwildebeest,upto11forzebra),thekillisdistributedtotheentireclan(20-50 animals),withthehighestranksobtainingthelargestshare[174].Now,thequestioniswhat makessuch\unjust"societiesstable?Weknowthatfemalehyenasruleinhyenasociety. Theyareabouttenpercentlargerthanmalesandalsoshowaggressivebehaviorfrequently towardlowerrankfemalesandmales.Therefore,thisaggressivebehaviorandpunishment attitudemightbethesignalingforcewhichrulesasacommunicationmechanismtomaintain cooperationamonghyenas.Toaddressthisquestion,weintroducecollectivehuntinggame 222 Figure6.4:ofpunishmentprobabilityonmutualinformationbetweenactionofpun- ishingplayersandopponenttype.X-axisispunishmentprobability( ),Y-axisisthecon- centrationofpunishingcooperatorinpopulation,andZ-axisismutualinformationbetween actionofpunishingplayersandopponent'stype:(a)mutualinformationbetweenactionof vigilant/punishingcooperatorandopponent'stype(b)mutualinformationbetweenaction ofvigilant/punishingdefectorandopponent'stype 223 Figure6.5:UsingHeavisidestepfunctiontomodelallornothingsituationsingrouphunting behaviorofanimals. asanewgame-theoreticformulationbyextendingthestandardpublicgoodsgametomodel thesocialbehaviorofhyenasintheir\unjust"societyandstudyhowadespoticgroup structurecanberobust,evolutionarystable,andenforcedbypunishment.Instandard publicgoodsgameonecooperatorisenoughfordefectortoreceiveapaygreaterthan zero(6.1);however,inmammalswithgrouphuntingbehaviorthisisnottrue.Ifthegroup sizeisnotgreaterthanathreshold,whichisafunctionofthepreysize,thehuntwillbe unsuccessfulandnobodyreceivesanything;thereforethepaytothestrategiesshouldbe moforhuntingbehavior.TheeasiestwaytomodifythisbehavioristouseaHeaviside stepfunctionasitisshownin(6.5).Thisapproachissimilartotheproposedmodels in[169,157];however,Pachecoetal.appliedalinearfunctionintheirmodel. Here,weinsteadapplyaHeavisidestepfunctiontohaveabettermodelofgrouphunting behaviorofhyenas.HeavisidestepfunctionisthesimplestmotoPGG.Sincewe don'tapplythecalculusofvariationtothisgame,wedon'tneedtoworryaboutthebreak pointinHeavisidefunction.Ifwewanttoapplyoptimizationtechniquestothisproblem, itisbettertousesmoothandtiablenonlinearfunctions(e.g.logisticortangent 224 hyperbolicfunctions)forcollectivehuntinggame.Usingthestepfunctionyieldstothe followingpaysfortstrategiesintheclan: ˇ C = r k +1 ( kˆ c +1 ) 1 pc (1 ˆ c ) ˇ D = r k +1 ( kˆ c ) pc ˆ c pd (1 ˆ c ) pd (1 ˆ c )(6.46) ByComparingequations(6.1)and(6.46),weseethattheimportantparameterinthecol- lectivehuntinggameisthethresholdofthecontributedhunters.Basedonthisparameter wecanhavecompletelytpropertiesthatwecouldnotinthestandardpublic goodsgame. 6.2.1Collectivehuntinggamewithnopunishment Similartothestandardgame,weignorepunishmentandweanalyzecollectivehunting gamesbetweenpurecooperatorsanddefectors.Thepaytothecooperatorsanddefectors forthisgameis: ˇ C = r k +1 kˆ c +1 1 ˇ D = r k +1 kˆ c (6.47) Therefore,thepaymatrixforthisgameis: E = 0 B @ CD C r k +1 ( k +1 ) 1 r k +1 (1 ) 1 D r k +1 ( k )0 1 C A (6.48) 225 6.2.1.1Solitaryhunting( =1 ) Insolitaryhunting,thepaymatrixis: E =1 = 0 B @ CD C r k +1 1 r k +1 1 D r k +1 0 1 C A (6.49) whichisequivalentto: E =1 = 0 B @ CD C 0 r k +1 1 D 10 1 C A (6.50) Accordingtopaymatrix(6.50),whenkill-size r issmall( r ( Dk +1)( k +1) k ( D 1) ˇ Dk +1 D 1 forbiggroupsize Ifdespoticparameter D islessthanonewhichisadvantageoustothepoorplayers,poor cooperatorcanwinthegameagainstarichdefectorifandonlyif: r> ( D + k )( k +1) k (1 D ) ˇ D + k 1 D forbiggroupsize Thisshowsthatitispossibletohavecooperationasevolutionarystablestrategyamongpoor orrichdefectors;however,havingpoororrichstrategiesdoesnotyieldtocoexistenceof defectorsandcooperators.Next,weshowthathowpunishersusepunishmentasasignaling mechanismtomaintaincooperationintheclan.Thesepunishersacceptapunishingcostto reducepayofthefreeridersbypunishingthem.Table(6.1)summarizethestabilityof 238 pointsforcollectivehuntinggamewithnopunishment. Table6.1: Stabilityofcollectivehuntinggame'spointswhileplayersdon'tpunishfreeriders. Game cooperationdefectionmixedstrategy Solitaryhunting unstablestableifstableif ( =1) rk +1 AllorNothing stableifstableunstable ( = K +1) r>k +1 Smallgrouphunting unstablestableunstable Smallgrouphunting C R winsagainst D P C P winsagainst D R unstable betweenrichandpoor if r> Dk +1 D 1 if r> D + k 1 D 6.2.2ofpunishmentoncollectivehuntinggame Byintroducingpunishmentprobabilities( pc and pd ),thepaymatrixofsuchcollective huntinggamewouldbe: E = 0 B @ CD C r k +1 ( k +1 ) 1 r k +1 (1 ) 1 pc D r k +1 ( k ) pc ( + ) pd 1 C A (6.59) or: E = 0 B @ CD C 0 r k +1 (1 ) 1 pc +( + ) pd D r k +1 ( k ) r k +1 ( k +1 )+1 pc 0 1 C A (6.60) Here,besideshuntingthreshold\ ",therearefourpunishingparametersthatcan fateofthegame. 239 6.2.2.1Solitaryhunting( =1 ) Thepaymatrixforsolitaryhuntinggameis: E = 0 B @ CD C 0 r k +1 1 pc +( + ) pd D 1 pc 0 1 C A (6.61) Accordingtopaymatrix(6.61),wecanhaveallsortiteratedgames.Forexampleif ofpunishmentishighandcooperatorspunishmorefrequently,cooperationcanbe evolutionarystable.Ontheotherhand,ifcostofpunishmentishigh,anddefectorsdonot punishmorefrequently,evenifkillsizeisgreaterthanhuntinggroupsize(ifthereisno punishment,defectioncannotbeESSinthiscase)defectioncanbeevolutionarystableas well.Tothestabilityconditionsofpoints,wehavetoderivethereplicatorequation forthisgame.Wehave: _ ˆ c = ˆ c (1 ˆ c ) r k +1 1 pc +( + ) pd r k +1 ( + ) pc pd ˆ c (6.62) Thepointsforthisgameare: ˆ c =0,defector ˆ c =1,cooperator ˆ c = r ( k +1) 1+ pc ( + ) pd r ( k +1)( + ) pc pd ,mixed(6.63) 240 Bytakingthederivativeofthisreplicatorequationrespecttothefractionofcooperatorsin thepopulation,wecanidentifytheevolutionarystablepoints.Wehave: d _ ˆ c dˆ c =(1 ˆ c ) r k +1 1 pc +( + ) pd r k +1 ( + ) pc pd ˆ c ˆ c r k +1 1 pc +( + ) pd r k +1 ( + ) pc pd ˆ c r k +1 ( + ) pc pd ˆ c (1 ˆ c ) (6.64) Cooperationisrobustagainsttheinvasionofdefectorsifandonlyif: pc > 1(6.65) Cooperationcaninvadeapopulationofpuredefectorsifandonlyif: r k +1 1 pc +( + ) pd > 0(6.66) Ifmixedstrategyexists,itwillbeevolutionarystableifandonlyif: r k +1 ( + ) pc pd > 0(6.67) Ifcooperatorsanddefectorsequallypunishfreeriders,paymatrix(6.61)issito: E = 0 B @ CD C 0 r k +1 1 pc D 1 pc 0 1 C A (6.68) 241 Replicatorequation(6.62)isto: _ ˆ c = ˆ c (1 ˆ c ) r k +1 1+ pc r k +1 ˆ c (6.69) Thepointofthisequationare: ˆ c =0,defector ˆ c =1,cooperator ˆ c = r ( k +1) 1 pc r ,mixed(6.70) Mixedstrategyifexists,isstableifandonlyif: pc < 1 min 1 ; r k +1 1 (6.71) Cooperationisrobustagainsttheinvasionofdefectorsifandonlyif: pc > 1 (6.72) Cooperationcaninvadeapopulationofpuredefectorsifandonlyif: pc < 1 r k +1 1 (6.73) 242 Ifdefectorsaresmartenoughtostoppunishingtheirowntype,thepaymatrix(6.61)is to: E = 0 B @ CD C 0 r k +1 1 pc D 1 pc 0 1 C A (6.74) Replicatorequation(6.62)isto: _ ˆ c = ˆ c (1 ˆ c ) r k +1 1 pc r k +1 ( + ) pc ˆ c (6.75) Consideringthisreplicatorequation,thepointofthisgamewouldbe: ˆ c =0,defector ˆ c =1,cooperator ˆ c = r ( k +1) 1+ pc r pc ( k +1)( + ) ,mixed(6.76) Cooperationisevolutionarystableifandonlyif: pc > 1 (6.77) Defectionisevolutionarystableifandonlyif: pc > r ( k +1) ( k +1) (6.78) 243 Mixedstrategyisstableifandonlyif: pc < min r ( k +1) ( k +1) ; 1 (6.79) 6.2.2.2Allornothing( = k +1 ) Thepaymatrixforallornothinggameis: E = 0 B @ CD C 0 1 pc +( + ) pd D 1 r k +1 pc 0 1 C A (6.80) Similartosolitaryhuntinggame,wecaninferbycheckingthepaymatrix(6.80)thatall ornothinggamecanbetaketheformofallsortofiteratedtwoplayersgame.Forexample, ifofpunishmentishighandcooperatorspunishmorefrequently,evenifkillsizeis small,cooperationcanbeevolutionarystable.Ontheotherhand,ifcostofpunishmentis high,anddefectorsdonotpunishoftencomparetocooperators,evenifkillsizeisgreater thanhuntinggroupsize,defectioncanbeevolutionarystableaswell.Tothestability conditionsofpoints,wehavetoderivethereplicatorequationforthisgame.Wehave: _ ˆ c = ˆ c (1 ˆ c ) 1 pc +( + ) pd + r k +1 +( + ) pc pd ˆ c (6.81) Thepointsforthisgameare: ˆ c =0,defector ˆ c =1,cooperator ˆ c = ( k +1) 1+ pc ( + ) pd r +( k +1)( + ) pc pd ,mixed(6.82) 244 Bytakingthederivativeofthisreplicatorequationrespecttothefractionofcooperatorsin thepopulation,wecanidentifytheevolutionarystablepoints.Wehave: d _ ˆ c dˆ c =(1 ˆ c ) 1 pc +( + ) pd + r k +1 +( + ) pc pd ˆ c ˆ c 1 pc +( + ) pd + r k +1 +( + ) pc pd ˆ c + r k +1 +( + ) pc pd ˆ c (1 ˆ c ) (6.83) Cooperationisrobustagainsttheinvasionofdefectorsifandonlyif: pc + r k +1 > 1(6.84) Cooperationcaninvadeapopulationofpuredefectorsifandonlyif: ( + ) pd pc > 1(6.85) Mixedstrategyisevolutionarystableifandonlyif: pd pc > r ( k +1)( + ) (6.86) Ifcooperatorsanddefectorsequallypunishfreeriders,paymatrix(6.80)issito: E = 0 B @ CD C 0 1+ pc D 1 pc r k +1 0 1 C A (6.87) 245 Replicatorequation(6.81)isto: _ ˆ c = ˆ c (1 ˆ c ) 1+ pc + r k +1 ˆ c (6.88) Thepointofthisequationare: ˆ c =0,defector ˆ c =1,cooperator ˆ c = ( k +1) 1 pc r ,mixed(6.89) Mixedstrategyexistsifandonlyif: 1 r k +1 < pc < 1(6.90) Thispointisalwaysunstable.Cooperationisrobustagainsttheinvasionofdefectors ifandonlyif: pc + r k +1 > 1(6.91) Cooperationcaninvadeapopulationofpuredefectorsifandonlyif: pc > 1(6.92) 246 Ifdefectorsaresmartenoughtostoppunishingtheirowntype,thepaymatrix(6.80)is to: E = 0 B @ CD C 0 1 pc D 1 r k +1 pc 0 1 C A (6.93) Replicatorequation(6.62)isto: _ ˆ c = ˆ c (1 ˆ c ) 1 pc + r k +1 +( + ) pc ˆ c (6.94) Consideringthisreplicatorequation,thepointofthisgamewouldbe: ˆ c =0,defector ˆ c =1,cooperator ˆ c = ( k +1) 1+ pc r + pc ( k +1)( + ) ,mixed(6.95) Cooperationisevolutionarystableifandonlyif: pc + r k +1 > 1(6.96) Defectionisalwaysevolutionarystable;thus,mixedstrategycannotbeevolutionarysta- ble.Thus,thisgamecanonlybecoordinationgame,andwecannothavecooperationand defectionsimultaneouslyinthepopulation. 247 6.2.2.3Smallgrouphunting( 1 < 1 (6.100) Cooperationcaninvadeapopulationofpuredefectorsifandonlyif: 1 pc +( + ) pd > 0(6.101) Ifmixedstrategyexists,itwillbeevolutionarystableifandonlyif: pc < pd (6.102) Ifcooperatorsanddefectorsequallypunishfreeriders,paymatrix(6.61)issito: E = 0 B @ CD C 0 1+ pc D 1 pc 0 1 C A (6.103) Replicatorequation(6.98)isto: _ ˆ c = 1+ pc ˆ c (1 ˆ c )(6.104) Here,mixedstrategydoesnotexist.Cooperationisrobustagainsttheinvasionofdefectors ifandonlyif: pc > 1(6.105) 249 Cooperationcaninvadeapopulationofpuredefectorsifandonlyif: pc > 1(6.106) Ifdefectorsaresmartenoughtostoppunishingtheirowntype,thepaymatrix(6.97)is to: E = 0 B @ CD C 0 1 pc D 1 pc 0 1 C A (6.107) Replicatorequation(6.98)isto: _ ˆ c = ˆ c (1 ˆ c ) 1 pc +( + ) pc ˆ c (6.108) Consideringthisreplicatorequation,thepointofthisgamewouldbe: ˆ c =0,defector ˆ c =1,cooperator ˆ c = 1+ pc pc ( + ) ,mixed(6.109) Cooperationisevolutionarystableifandonlyif: pc > 1 (6.110) 250 Here,defectorisalwaysevolutionarystable;thus,mixedstrategycannotbeevolutionary stable.Tables(6.2-6.10)summarizethestabilityofxedpointsforcollectivehuntinggame withnopunishment. Table6.2: Stabilityofcollectivehuntinggame'spoints,whileplayerspunishfreeriders,and onehunterisenoughtokilltheprey(generalrule). Strategy Solitaryhunting( =1) Cooperation pc > 1 Defection pc ( + ) pd > 1 k +1 1 Mixedstrategy pd pc < r ( + )( k +1) Table6.3: Stabilityofcollectivehuntinggame'spoints,whileplayerspunishequallyfree riders,andonehunterisenoughtokilltheprey. Strategy Solitaryhunting( =1and pc = pd ) Cooperation pc > 1 Defection pc > r ( k +1) ( k +1) Mixedstrategy pc < 1 min 1 ; r k +1 1 Table6.4: Stabilityofcollectivehuntinggame'spoints,whileonlycooperatorspunishfree riders,andonehunterisenoughtokilltheprey. Strategy Solitaryhunting( =1and pd =0) Cooperation pc > 1 Defection pc > r ( k +1) ( k +1) Mixedstrategy pc < min r ( k +1) ( k +1) ; 1 251 Table6.5: Stabilityofcollectivehuntinggame'spoints,whileplayerspunishfreeriders,and cooperationofallgroupisnecessarytokilltheprey(generalrule). Strategy Allornothing( = k +1) Cooperation pc + r k +1 > 1 Defection ( + ) pd pc < 1 Mixedstrategy pd pc > r ( + )( k +1) Table6.6: Stabilityofcollectivehuntinggame'spoints,whileplayerspunishequallyfree riders,andcooperationofallgroupisnecessarytokilltheprey. Strategy Allornothing( = k +1and pc = pd ) Cooperation pc > 1 r k +1 Defection pc < 1 Mixedstrategy Unstable Table6.7: Stabilityofcollectivehuntinggame'spoints,whileonlycooperatorspunishfree riders,andcooperationofallgroupisnecessarytokilltheprey. Strategy Allornothing( = k +1and pd =0) Cooperation pc > 1 r k +1 Defection Stable Mixedstrategy Unstable Table6.8: Stabilityofcollectivehuntinggame'spoints,whileplayerspunishfreeriders,and cooperationofsmallgroup(gameofhyenas)isnecessarytokilltheprey(generalrule). Strategy Smallgrouphunting(1 < k ) Cooperation pc > 1 Defection pc +( + ) pd < 1 Mixedstrategy pd > pc and pc < 1 6.3Discussion Hyenalivein\clanswherethehigh-rankingfemalesenjoybetteraccesstofood,areprotected fromaggression,andhavemorethanlowerrankedfemales[175,173,176,177]. Thisdespoticsocietalstructureisblatantlyunjust,yetitappearsthatitisanevolutionarily 252 Table6.9: Stabilityofcollectivehuntinggame'spoints,whileplayerspunishequallyfree riders,andcooperationofsmallgroup(gameofhyenas)isnecessarytokilltheprey. Strategy Smallgrouphunting(1 < k and pc = pd ) Cooperation pc > 1 Defection pc < 1 Mixedstrategy Doesnotexist Table6.10: Stabilityofcollectivehuntinggame'spoints,whileonlycooperatorspunishfree riders,andcooperationofsmallgroup(gameofhyenas)isnecessarytokilltheprey. Strategy Smallgrouphunting(1 < k and pd =0) Cooperation pc > 1 Defection Stable Mixedstrategy Unstable stableadaptation.Inthischapter,wetriedtoanalyzehyenasgrouphuntingpatternto whatecologicalfactorsfavorsuchdespoticsocietalstructureandwhataretheselective pressuresthatmakesuchanunjustsocietypossible.Todoso,weappliedevolutionarygame theorysinceithasbeenusedsuccessfullytostudyanumberoftbehaviorsthat atappearedparadoxicalinthelightofevolution[16,36,48].Amongwell-established gametheoreticmodel,wechosethestandardPublicGoodsgame(PGG)tomodelandstudy societalstructureofhyenassincethisgamedescribesthedilemmaofcooperationfromthe pointofviewofagroup.Consideringtheaggressionofhigherrankfemalestowardsthelower ranksintheclan,wealsointroduced\punishmenttothegameasapossiblemechanismthat canmaintaincooperationandprotectinvestorsagainstthefreeridersinthegroup.Our resultsdepictthatwhilepunishmentestablishescooperationamongawell-mixedgroupof defectorsandcooperators,itisnotanestrategysinceinthestandardpublicgoods gamecooperatorsanddefectorsdonotcoexist.Coexistenceofdefectorsandcooperators isanassumptiontotheunjustsocietalstructureofhyenasthatcannotbeavoidedsincein 253 hyenasclancooperatorsanddefectorscoexist. Thus,wehadtomodifythestandardpublicgoodsgametomakeitapplicabletogroup huntingbehaviorofhyenas.Oneproblemwiththestandardpublicgoodsgameisthatthe paysarelinearfunctionsofthekillsize;however,inrealitythepaydoesnotincrease asnumberofhuntersincreases.Hyenacantakedownmediumtolargepreythattheyalone couldnotkill[174].Participatinginthehuntcanbecostly,butthepayisworthit. However,thesizeofthekill(thetotalpay)doesnotdependlinearlyonthenumberof hunters:eitherthereareenoughhuntersornot.Andifthereareenoughtotakedownthe prey,addingmorehuntersdoesnotincreasethesizeofthekill(ifweassumethatthehyena alwayshuntthesamesizeanimal). Inthischapter,weintroducedthe\CollectiveHuntingGame(CTG)asamoto thestandardpublicgoodsgamebysubstitutingthelinearpayofthegamewithathreshold dependencepayThismochangesthedynamicsofthegamedramatically.In collectivehuntinggamethethresholdparameterisverycriticalandbasedonthisparameter wecanhavethreetgameregimes:(1)Thesolitaryhunting( =1):asinglehunter isttotakedowntheprey).Thisgameissomewhatuninterestingasitisnota collectivegame.Weseethat,withoutpunishment,itispossibletohavecoexistenceof cooperatorsanddefectorswhenthekillsizeisbig.Moreover,ifcooperatorspunishfree ridersandtherateoftheirpunishmentisgreaterthan 1 ,theyareevolutionarystable. Ontheotherhand,ifpunishmentrateofcooperatorsislessthanacriticalvalue(please refertotable(6.2-6.10)formoredetails),cooperatorsanddefectorscoexist,andpunishment becomesanestrategy.(2)TheAll-or-nothinggame:(( = k +1,where k isthesize ofthegroup.)Inthisgame,asingledefectoristtodoomthehuntingsuccess:all mustparticipate.Thisgameisinterestingmathematically,becausetheevolutionary 254 point(theoptimalrationalstrategy)isfrequencydependentintheabsenceofpunishment. Ifmorethanacriticalnumberofagentsparticipate,theevolutionarydynamicsdrivethe populationtowardsthe\all-cooperatepoint,whileiffewerthanthecriticalnumber participate,evolutionarydynamicsdrivethepopulationtowardsall-defection.Moreover, ifdefectorspunishmentrateisgreaterthancooperatorspunishmentrate( pd pc > r ( + )( k +1) ),theycancoexist,andagainpunishmentbecomesanestrategy.(3)The Hyenagame(1 <," The- oreticalpopulationbiology ,vol.38,no.2,pp.219{232,1990. [108]L.S.Shapley,\Stochasticgames," ProceedingsoftheNationalAcademyofSciencesof theUnitedStatesofAmerica ,vol.39,no.10,p.1095,1953. 290 [109]H.OhtsukiandM.A.Nowak,\Evolutionarygamesoncycles," Proceedingsofthe RoyalSocietyB:BiologicalSciences ,vol.273,no.1598,pp.2249{2256,2006. [110]D.Iliopoulos,A.Hintze,andC.Adami,\Criticaldynamicsintheevolutionofstochas- ticstrategiesfortheiteratedprisoner'sdilemma," PLoScomputationalbiology ,vol.6, no.10,p.e1000948,2010. [111]W.H.PressandF.J.Dyson,\Iteratedprisonersdilemmacontainsstrategiesthatdom- inateanyevolutionaryopponent," ProceedingsoftheNationalAcademyofSciences , vol.109,no.26,pp.10409{10413,2012. [112]A.J.StewartandJ.B.Plotkin,\Extortionandcooperationintheprisonersdilemma," ProceedingsoftheNationalAcademyofSciences ,vol.109,no.26,pp.10134{10135, 2012. [113]C.AdamiandA.Hintze,\Evolutionaryinstabilityofzero-determinantstrategies demonstratesthatwinningisnoteverything," Naturecommunications ,vol.4,2013. [114]G.Hardin,\Thetragedyofthecommons," science ,vol.162,no.3859,pp.1243{1248, 1968. [115]A.HintzeandC.Adami,\Punishmentcatalyzestheevolutionofcooperation," arXiv preprintarXiv:1210.5233 ,2012. [116]D.J.Rankin,K.Bargum,andH.Kokko,\Thetragedyofthecommonsinevolutionary biology," TrendsinEcology&Evolution ,vol.22,no.12,pp.643{651,2007. [117]R.MacLean,\Thetragedyofthecommonsinmicrobialpopulations:insightsfrom theoretical,comparativeandexperimentalstudies," Heredity ,vol.100,no.5,pp.471{ 477,2007. [118]S.A.Frank,\Modelsofparasitevirulence," QuarterlyreviewofBiology ,pp.37{78, 1996. [119]S.Brown,\Cooperationandinhost{manipulatingparasites," Proceedingsof theRoyalSocietyofLondon.SeriesB:BiologicalSciences ,vol.266,no.1431,pp. 1899{1904,1999. [120]M.OlsonandM.Olson, Thelogicofcollectiveaction:publicgoodsandthetheoryof groups. HarvardUniversityPress,2009,vol.124. 291 [121]D.D.Davis, Experimentaleconomics .Princetonuniversitypress,1993. [122]J.Ledyard,\Publicgoods:Asurveyofexperimentalresearch,"DavidK.Levine,Tech. Rep.,1997. [123]D.G.Rand,A.Dreber,T.Ellingsen,D.Fudenberg,andM.A.Nowak,\Positive interactionspromotepubliccooperation," Science ,vol.325,no.5945,pp.1272{1275, 2009. [124]M.A.JanssenandT.Ahn,\Adaptationvs.anticipationinpublic-goodgames,"in annualmeetingoftheAmericanPoliticalScienceAssociation,Philadelphia,PA ,2003. [125]A.Gunnthorsdottir,D.Houser,andK.McCabe,\Disposition,historyandcontribu- tionsinpublicgoodsexperiments," JournalofEconomicBehavior&Organization , vol.62,no.2,pp.304{315,2007. [126]V.Capraro,\Amodelofhumancooperationinsocialdilemmas," PloSone ,vol.8, no.8,p.e72427,2013. [127]E.FehrandS.achter,\Altruisticpunishmentinhumans," Nature ,vol.415,no.6868, pp.137{140,2002. [128]E.FehrandU.Fischbacher,\Thenatureofhumanaltruism," Nature ,vol.425,no. 6960,pp.785{791,2003. [129]M.NakamaruandY.Iwasa,\Thecoevolutionofaltruismandpunishment:roleofthe punisher," Journaloftheoreticalbiology ,vol.240,no.3,pp.475{488,2006. [130]C.F.CamererandE.Fehr,\Whendoes"economicman"dominatesocialbehavior?" Science ,vol.311,no.5757,pp.47{52,2006. [131]K.Sigmund,C.Hauert,andM.A.Nowak,\Rewardandpunishment," Proceedingsof theNationalAcademyofSciences ,vol.98,no.19,pp.10757{10762,2001. [132]H.Brandt,C.Hauert,andK.Sigmund,\Punishmentandreputationinspatialpub- licgoodsgames," ProceedingsoftheRoyalSocietyofLondon.SeriesB:Biological Sciences ,vol.270,no.1519,pp.1099{1104,2003. [133]D.Helbing,A.Szolnoki,M.Perc,andG.Szabo,\Evolutionaryestablishmentofmoral anddoublemoralstandardsthroughspatialinteractions," PLoScomputationalbiology , vol.6,no.4,p.e1000758,2010. 292 [134]K.DengandT.Chu,\Adaptiveevolutionofcooperationthroughdarwiniandynamics inpublicgoodsgames," PloSone ,vol.6,no.10,p.e25496,2011. [135]J.Y.Wakano,M.A.Nowak,andC.Hauert,\Spatialdynamicsofecologicalpublic goods," ProceedingsoftheNationalAcademyofSciences ,vol.106,no.19,pp.7910{ 7914,2009. [136]M.A.NowakandK.Sigmund,\Gamesongrids," Thegeometryofecologicalinterac- tions:simplifyingspatialcomplexity ,pp.135{150,2000. [137]R.E.Lenski,C.Ofria,R.T.Pennock,andC.Adami,\Theevolutionaryoriginof complexfeatures," Nature ,vol.423,no.6936,pp.139{144,2003. [138]C.OfriaandC.O.Wilke,\Avida:Asoftwareplatformforresearchincomputational evolutionarybiology," Alife ,vol.10,no.2,pp.191{229,2004. [139]C.Shannon,W.Weaver,R.Blahut,andB.Hajek, Themathematicaltheoryofcom- munication .UniversityofIllinoispressUrbana,1949,vol.117. [140]N.Wiener,\Cybernetics;orcontrolandcommunicationintheanimalandthema- chine."1948. [141]J.Smith,\Theconceptofinformationinbiology," Philosophyofscience ,pp.177{194, 2000. [142]T.D.Schneider,\Evolutionofbiologicalinformation," Nucleicacidsresearch ,vol.28, no.14,pp.2794{2799,2000. [143]C.Adami,C.Ofria,andT.C.Collier,\Evolutionofbiologicalcomplexity," Proceedings oftheNationalAcademyofSciences ,vol.97,no.9,pp.4463{4468,2000. [144]C.Adami,\Whatiscomplexity?" BioEssays ,vol.24,no.12,p.10851094,2002. [Online].Available:http://dx.doi.org/10.1002/bies.10192 [145]C.AdamiandN.Cerf,\Physicalcomplexityofsymbolicsequences," PhysicaD:Non- linearPhenomena ,vol.137,no.1,pp.62{69,2000. [146]R.BadiiandA.Politi, Complexity:Hierarchicalstructuresandscalinginphysics . CambridgeUniversityPress,1999,vol.6. 293 [147]M.Donaldson-Matasci,C.Bergstrom,andM.Lachmann,\Thevalueofinfor- mation," Oikos ,vol.119,no.2,pp.219{230,2010. [148]C.ShannonandW.Weaver, Themathematicaltheoryofcommunication .University ofIllinoisPress,1962.[Online].Available:http://books.google.com/books?id= CugtAQAAIAAJ [149]C.Adami,\Digitalgenetics:unravellingthegeneticbasisofevolution," NatureRe- viewsGenetics ,vol.7,no.2,pp.109{118,2006. [150]B.J.-P.Delahaye,andP.Mathieu,\Completeclassesofstrategiesforthe classicaliteratedprisoner'sdilemma,"in EvolutionaryProgrammingVII .Springer, 1998,pp.33{41. [151]H.J.MullerandR.Dieng, Computationalcmodelingfordistributed intelligentsystemswithcontributionsbynumerousexperts .Springer,2000. [152]K.LindgrenandM.G.Nordahl,\Evolutionarydynamicsofspatialgames," Physica D:NonlinearPhenomena ,vol.75,no.1,pp.292{309,1994. [153]D.Newth,\Asynchronousiteratedprisoner'sdilemma," AdaptiveBehavior ,vol.17, no.2,pp.175{183,2009. [154]D.Garcia,\Robustsmoothingofgriddeddatainoneandhigherdimensionswith missingvalues," ComputationalStatistics&DataAnalysis ,vol.54,no.4,pp.1167{ 1178,2010. [155]M.W.MacyandA.Flache,\Learningdynamicsinsocialdilemmas," Proceedingsof theNationalAcademyofSciences ,vol.99,no.suppl3,pp.7229{7236,2002. [156]F.C.Santos,J.M.Pacheco,andT.Lenaerts,\Evolutionarydynamicsofsocialdilem- masinstructuredheterogeneouspopulations," ProceedingsoftheNationalAcademy ofSciencesoftheUnitedStatesofAmerica ,vol.103,no.9,pp.3490{3494,2006. [157]J.M.Pacheco,F.C.Santos,M.O.Souza,andB.Skyrms,\Evolutionarydynamics ofcollectiveaction,"in TheMathematicsofDarwinsLegacy .Springer,2011,pp. 119{138. [158]F.C.SantosandJ.M.Pacheco,\Riskofcollectivefailureprovidesanescapefromthe tragedyofthecommons," ProceedingsoftheNationalAcademyofSciences ,vol.108, no.26,pp.10421{10425,2011. 294 [159]M.ArchettiandI.Scheuring,\Review:Gametheoryofpublicgoodsinone-shotsocial dilemmaswithoutassortment," JournalofTheoreticalBiology ,vol.299,pp.9{20,2012. [160]C.S.GokhaleandA.Traulsen,\Evolutionarygamesinthemultiverse," Proceedings oftheNationalAcademyofSciences ,vol.107,no.12,pp.5500{5504,2010. [161]M.PercandP.Grigolini,\Collectivebehaviorandevolutionarygames{anintroduc- tion," Chaos,Solitons&Fractals ,vol.56,pp.1{5,2013. [162]D.Balliet,L.B.Mulder,andP.A.VanLange,\Reward,punishment,andcooperation: ameta-analysis." Psychologicalbulletin ,vol.137,no.4,p.594,2011. [163]R.Boyd,H.Gintis,andS.Bowles,\Coordinatedpunishmentofdefectorssustains cooperationandcanproliferatewhenrare," Science ,vol.328,no.5978,pp.617{620, 2010. [164]G.BornsteinandO.Weisel,\Punishment,cooperation,andcheaterdetectioninnoisy socialexchange," Games ,vol.1,no.1,pp.18{33,2010. [165]E.FehrandS.achter,\Cooperationandpunishmentinpublicgoodsexperiments," InstituteforEmpiricalResearchinEconomicsworkingpaper ,no.10,1999. [166]R.BoydandP.J.Richerson,\Punishmentallowstheevolutionofcooperation(or anythingelse)insizablegroups," Ethologyandsociobiology ,vol.13,no.3,pp.171{ 195,1992. [167]J.H.Fowler,\Altruisticpunishmentandtheoriginofcooperation," Proceedingsofthe NationalAcademyofSciencesoftheUnitedStatesofAmerica ,vol.102,no.19,pp. 7047{7049,2005. [168]T.H.Clutton-BrockandG.A.Parker,\Punishmentinanimalsocieties," Nature ,vol. 373,no.6511,pp.209{216,1995. [169]J.M.Pacheco,F.C.Santos,M.O.Souza,andB.Skyrms,\Evolutionarydynamics ofcollectiveactioninN-personstaghuntdilemmas," ProceedingsoftheRoyalSociety B:BiologicalSciences ,vol.276,no.1655,pp.315{321,Jan.2009.[Online].Available: http://dx.doi.org/10.1098/rspb.2008.1126 [170]T.Yamagishi,\Theprovisionofasanctioningsystemasapublicgood." Journalof PersonalityandsocialPsychology ,vol.51,no.1,p.110,1986. 295 [171]E.Fehr,\Humanbehaviour:don'tloseyourreputation," Nature ,vol.432,no.7016, pp.449{450,2004. [172]A.M.Colman,\Thepuzzleofcooperation," Nature ,vol.440,no.7085,pp.744{745, 2006. [173]H.E.WattsandK.E.Holekamp,\Hyenasocieties," CurrentBiology ,vol.17,no.16, pp.R657{R660,2007. [174]K.E.Holekamp,L.Smale,R.Berg,andS.M.Cooper,\Huntingratesandhunting successinthespottedhyena(crocutacrocuta)," JournalofZoology ,vol.242,no.1, pp.1{15,1997. [175]K.E.HolekampandL.Smale,\Ontogenyofdominanceinfree-livingspottedhyaenas: juvenilerankrelationswithotherimmatureindividuals," AnimalBehaviour ,vol.46, no.3,pp.451{466,1993. [176]H.E.WattsandK.E.Holekamp,\Ecologicaldeterminantsofsurvivalandrepro- ductioninthespottedhyena," JournalofMammalogy ,vol.90,no.2,pp.461{471, 2009. [177]K.E.Holekamp,J.E.Smith,C.C.R.C.VanHorn,andH.E.Watts, \Society,demographyandgeneticstructureinthespottedhyena," MolecularEcology , vol.21,no.3,pp.613{632,2012. [178]S.A.Frank,\Georgepricescontributionstoevolutionarygenetics," JournalofTheo- reticalBiology ,vol.175,no.3,pp.373{388,1995. [179]M.vanVeelen,\Ontheuseofthepriceequation," JournalofTheoreticalBiology ,vol. 237,no.4,pp.412{426,2005. [180]G.R.Price,\Thenatureofselection," JournalofTheoreticalBiology ,vol.175,no.3, pp.389{396,1995. [181]M.vanVeelen,J.GarcM.W.Sabelis,andM.Egas,\Groupselectionandinclusive arenotequivalent;thepriceequationvs.modelsandstatistics," Journal oftheoreticalbiology ,vol.299,pp.64{80,2012. [182]B.KerrandP.Godfrey-Smith,\Generalizationofthepriceequationforevolutionary change," Evolution ,vol.63,no.2,pp.531{536,2009. 296 [183]S.F.Elena,C.O.Wilke,C.Ofria,andR.E.Lenski,ofpopulationsizeand mutationrateontheevolutionofmutationalrobustness," Evolution ,vol.61,no.3, pp.666{674,2007. [184]C.O.WilkeandC.Adami,\Thebiologyofdigitalorganisms," TrendsinEcology& Evolution ,vol.17,no.11,pp.528{532,2002. 297