LOCALLYLINEARMANIFOLDMODELFORGAP-FILLINGALGORITHMSOFHYPERSPECTRALIMAGERY:PROPOSEDALGORITHMSANDACOMPARATIVESTUDYBySuhaIbrahimSulimanATHESISSubmittedtoMichiganStateUniversityinpartialentoftherequirementsforthedegreeofElectricalEngineering-MasterofScience2016ABSTRACTLOCALLYLINEARMANIFOLDMODELFORGAP-FILLINGALGORITHMSOFHYPERSPECTRALIMAGERY:PROPOSEDALGORITHMSANDACOMPARATIVESTUDYBySuhaIbrahimSulimanLandsat7EnhancedThematicMapperPlus(ETM+)ScanLineCorrector(SLC)device,whichcorrectsforthesatellitemotion,hasfailedsinceMay2003resultinginalossofabout22%ofthedata.ToimprovethereconstructionofLandsat7images,LocallyLinearManifold(LLM)modelisproposedforgapsinhyperspectralimagery.Inthisapproach,eachspectralbandismodeledasanon-linearlocallymanifoldthatcanbelearnedfromthematchingbandsatttimeinstances.Moreover,eachbandisdividedintosmalloverlappingspatialpatches.Inparticular,eachpatchisconsideredtobealinearcombinationofasetofcorrespondingpatchesfromthesamelocationthatareadjacentintimeorfromthesameseasonoftheyear.Usingthisapproach,thegap-processinvolvesafeasiblepointonthelearnedmanifoldtoapproximatethemissingpixels.TheproposedLLMframeworkiscomparedtosomeexistingsingle-sourceandmulti-sourcemethodologies.WeanalyzetheenessoftheproposedLLMapproachthroughsimulationexampleswithknownground-truth.ItisshownthattheLLM-modeldrivenapproachoutperformsallexistingrecoverymethodsconsideredinthisstudy.ThesuperiorityofLLMisillustratedbyprovidingbetterreconstructedimageswithhigheraccuracyevenoverheterogeneouslandscape.Moreover,itisrelativelysimpletorealizealgorithmically,anditneedsmuchlesscomputingtimewhencomparedtothestate-of-the-artAWLHMapproach.Thisthesisisdedicatedtothesoulsofmybelovedfatherandbrother,mygreatmotherandmylovelyson.iiiACKNOWLEDGMENTSIwouldliketotakethisopportunitytoexpressmysincereappreciationtomyadviser,Prof.HayderRadha,forhercontinuoussupport,helpandencouragementthroughoutmymasterstudy.Thisthesiswouldnothavebeenpossiblewithouthisguidance.IwouldalsoliketothankProf.LalitaUdpaandProf.PercyPierrefromtheDepartmentofElectricalandComputerEngineeringforservingonmythesiscommittee.Iamdeeplygratefultothemfortheirmotivationandinsightfulcomments.SpecialthanksgotomycolleaguesintheWirelessandVideoCommunications(WAVES)Laboratory.IamparticularlyindebtedtoMohammadAghagolzadaforhishelpandinvalu-ablecommentsandsuggestionsontheresearchissues.Lastbutnotleast,Iamdeeplygratefultomyfamilyandfriendsfortheirtremendoussupportandencouragement.ivTABLEOFCONTENTSLISTOFTABLES....................................viiLISTOFFIGURES...................................viiiChapter1INTRODUCTION...........................11.1Overview......................................11.2Survey.......................................31.2.1Single-SourceTechnique.........................31.2.1.1SimpleInterpolationApproach................31.2.1.2SegmentationModelApproach................51.2.2Multi-SourceTechnique.........................61.2.2.1LandsatMulti-SourceApproach................61.2.2.1.1USGSandNASAApproach.............71.2.2.1.1.1PhaseOneModel...............71.2.2.1.1.2PhaseTwoModel...............81.2.2.1.2KrigingGeostatisticalApproach...........91.2.2.1.3SpectralProjectionTransformationApproach...111.2.2.1.4NeighborhoodSimilarPixelInterpolator(NSPI)Ap-proach.........................111.2.2.1.5AMorphology-StitchingApproach.........131.2.2.2Non-LandsatMulti-SourceApproach............141.3ThesisOrganization................................16Chapter2EXISTINGRECOVERYMETHODOLOGIES.........172.1Single-SourceTechnique.............................172.1.1MovingAverageFilter..........................172.1.2SpatialInterpolationTechnique.....................182.1.2.1InverseDistanceWeight(IDW)Approach..........192.2Multi-SourceTechnique..............................202.2.1PhaseOne/LocalLinearHistogramMatching(LLHM)Algorithm..202.2.1.1LLHMMathematicalAnalysis.................222.2.2PhaseTwo/AdaptiveWindowLinearHistogramMatching(AWLHM)Algorithm.................................232.2.2.1AWLHMMathematicalAnalysis...............272.3Summary.....................................28Chapter3THEPROPOSEDLOCALLYLINEARMANIFOLD(LLM)MODEL..................................303.1ABriefOverviewofLLMModel.........................30v3.2TheEstimationProcessoftheMissingPixelsUsingtheLLM-modelDrivenApproach.....................................313.3LLMMathematicalAnalysis...........................343.4Summary.....................................38Chapter4EXPERIMENTALRESULTSANDPERFORMANCEEVAL-UATIONOFGAP-FILLINGALGORITHMS..........394.1ReconstructedResultsofSimulatedImages..............404.1.1AWLHMUsingMultipleImages................454.1.2LLMUsingMultipleSLC-onFillImages................484.2ReconstructedResultsofRealImages.................49Chapter5CONCLUSIONSANDFUTUREWORK.............545.1Conclusions....................................545.2FutureWork....................................55BIBLIOGRAPHY....................................57viLISTOFTABLESTable1.1:ThecharacteristicsofLandsat7(ETM+)bands(mofrom[1]).2Table4.1:ThequantitativeassessmentofAverage,IDW,LLHM,AWLHM(us-ingoneSLC-onimage)methods.....................50Table4.2:ThequantitativeassessmentofLLMandAWLHM(usingmultipleimages)methods.........................50viiLISTOFFIGURESFigure1.1:Landsat7ETM+SLC-onandoperation(obtainedandmod-from[2,3])."Forinterpretationofthereferencestocolorinthisandallotherthereaderisreferredtotheelectronicversionofthisthesis."..............................2Figure2.1:Alineartransformationbetweentheimageandtarget)image[4]..................................21Figure2.2:Alinearhistogramadaptivewindowprocess,thesmallredsquareatthecenterrepresentsthegappixelsneededtoberecovered,thereddashedlinesrepresenttheincreasingprocessofwindowsizeandthebigredsquarerepresentsthewindowsizethatcouldbereached[4,5]................................24Figure2.3:Aschematicdiagramoftheadaptivewindowprocess(mofrom[6]).PixelPthatneedstoberecoveredisonthesamelocationintargetandtrainingimagesasaresultofimageregistration.In(a),thewindowsizeusedis55,theresultedcommonpixels=12.Ifthisvaluedoesnotmeettheminimumrequiredcommonpixels,thewindowsizewouldbeexpandedbyonepixeloneachsideasin(b).Thewindowsizein(b)willbe77,thecommonpixelsarecomputedagain.Ifthenewvaluedoesnotmeettheminimumrequiredcommonpixels,continueincreasingthewindowsizeuntilmaximumwindowsizeisreached..............................26Figure3.1:Anillustrationdiagramrepresentsselectingpatches(P1-Pm)fromsamelocationinband1(B1)throughttimeinstancest(1984-2011)...................................32Figure3.2:Anabstractillustrationofa1D(onedimensional)manifoldrepre-sentingaparticularlandcoverpatchovertime.Here,each(approx-imatelysegmentofthemanifoldmaycorrespondtoaparticularseasonoftheyear1984through2011............34viiiFigure3.3:Aschematicdiagramillustratingthematrixsegmentationintoknownandunknownregions.WhereXdenotesatrainingdatathatisseg-mentedintoknownandunknownregions,ndenotesthenumberofpatches,mdenotesthenumberofpixels,m1denotesthenumberofknownpixels,m2denotesthenumberoftheunknown(missing)pix-els,Adenotestheknownregion,Bdenotestheunknownregion,denotesthecotthatwillbeusedtopredictthemissingpixel,xjdenotesthepatchthathasmissingpixels(b)andneedstobepredicted.................................35Figure4.1:Aschematicdiagramillustratingtheextractionprocessofsubimagesfrombands(5,4,2)ofthewholescene.................40Figure4.2:SWIR-NIR-greencompositesofLandsat5/TMimages.(a)Ground-truthSLC-onimageacquiredonMarch19,2006.(b)targetimagesimulatedbasedon(a).(c)SLC-onimageacquiredonFebruary11,2002.............................41Figure4.3:ResultsusingsimulatedLandsat5image.(a)Simulatedtargetimage,(b)SLC-onllimage,(c-f)Reconstructedim-agesusingAverage,IDW,LLHMandAWLHM(usingSLC-onimage)methodsrespectively...........................43Figure4.4:Errorimagesobtainedfromband5.(a)Errorimagebeforeapplyingalgorithms,(b)AfterapplyingAverage(c)AfterapplyingIDW,(d)AfterapplyingLLHM,(e)AfterapplyingAWLHMusingoneSLC-onimage.........................44Figure4.5:TherelationshipbetweenPSNRandthewindowsizeusingAverage,IDW,LLHMandAWLHM(usingSLC-onimage)methods.Onleft,forband2;onright,forband4;onbottom,forband5......45Figure4.6:ResultsofAWLHMapproachusingmultiplesimulatedTMimages,(a)FirstimageacquiredonApril13,2005,(b)SecondllimageacquiredonFebruary23,2002,(c)TargetimageacquiredonMarch19,2006,(d)ReconstructedimageusingAWLHMapproach.46Figure4.7:Errorimagesobtainedfromband5usingAWLHMwithmultipleimages;fromlefttoright,errorimagesbeforeandafterusingAWLHMrespectively...........................47Figure4.8:ResultsusingLLMapproach;onleft,simulatedtargetimage;onright,reconstructedimageusingLLMapproach..........47ixFigure4.9:Errorimagesobtainedfromband5usingLLMapproach;onleft,errorimagebeforeusingLLM;onright,errorimageafterusingLLMapproach..................................48Figure4.10:TherelationshipbetweenPSNRandthewindowsizeusingLLMandAWLHM(usingmultipleinputimages)methods.Onleft,forband2;onright,forband4;onbottom,forband5......49Figure4.11:ResultsusingrealETM+targetimage.(a)RealtargetimageacquiredonAugust24,2003;(b)TMSLC-onllimageacquiredonApril23,2001;(c-f)ReconstructedimagesusingAverage,IDW,LLHMandAWLHM(usingSLC-onimage)methodsrespectively...52Figure4.12:ResultofAWLHMusingrealETM+inputimage.(a)and(b)RealETM+imagesacquiredonMay15,2004andAugust23,2005respectively;(c)ETM+targetimage;(d)ReconstructedimageusingAWLHM..................53Figure4.13:ResultofLLMusingrealETM+targetimage.(a)RealETM+targetimageacquiredonAugust24,2003;(b)Re-constructedimageusingLLM......................53xChapter1INTRODUCTION1.1OverviewLandsatsatelliteserieswerelaunchedinJuly1972asaninstrumenttocapturescenesofearth'stopographyusingremotesensingtohelpunderstandtheearth'senvironmentalsys-tem[7,8].TheseseriesincludesMulti-SpectralScanners(MSS)sensorsofLandsat1-5,ThematicMapper(TM)sensorofLandsat5,EnhancedThematicMapperPlus(ETM+)sensorofLandsat7andrecentlyOperationalLandImager(OLI)and/orThermalInfraredSensor(TIRS)sensorsofLandsat8[5,9].AftertheunsuccessfullaunchingofLandsat6in1993,Landsat7waslaunchedinApril1999withanimprovementrepresentedbyaddingpanchromaticbandwithaspatialresolutionof15mandimprovingthespatialresolutionofthethermalbandfrom120minLandsat5/TMto60minLandsat7/ETM+[2,10].CharacteristicsofLandsat7bandsareillustratedinTable1.1.Currently,bothLandsat7andLandsat8arestillbeingusedforearthobservation.TheScanLineCorrector(SLC)deviceofETM+sensoronboardoftheLandsat7satellite,whichcorrectsforsatellitefor-wardmotion,hasfailedsinceMay2003resultinginalossofabout22%ofthedata[4,11].Consequently,eachscanoverlapsinthemiddleofeachsceneinsteadofperformingaregularparallelscans[2].ThedefectedSLCdeviceisshowninFigure1.1.Therefore,theextentofmissingpixelsinproductsismuchgreaterwhentheyareclosertotheedgesofeachscene,whilethecenterpartofthesceneisfullysampled.1BandNumberBandNameBandWavelength(nm)BandSpatialResolution(m)1Blue450-520302Green530-610303Red630-690304NearInfrared780-900305ShortwaveIR-11550-1750306ThermalIR10400-12500607ShortwaveIR-22090-2350308Panchromatic520-90015Table1.1:ThecharacteristicsofLandsat7(ETM+)bands(mofrom[1]).TheradiometricandgeometriccharacteristicsofLandsat7SLC-productsstillmeetdesignspand80%ofthedataineachsceneisscannedfaultlessly[12].There-fore,itisusefultointerpolateorinthegapsforbothscienandaestheticmotives.Accordingly,manystudieshavebeenconductedinthistoachievethatgoal.Figure1.1:Landsat7ETM+SLC-onandoperation(obtainedandmofrom[2,3])."Forinterpretationofthereferencestocolorinthisandallotherthereaderisreferredtotheelectronicversionofthisthesis."21.2SurveyThissectionisdevotedtoshowingandlistingmostofthestudiesthathavebeenperformedtosolvetheprobleminLandsat7ETM+thathappenedafterMay2003asaresultoftheSLCfailure.Thisfailurecausedalossofabout22%ofthedataonlyandtheremainingundamagedpartoftheimagestillcontainsgoodinformationastheradiometricandgeometriccharacteristicsstillmeetthedesignspTherefore,scientistscouldtakeadvantageofscenesobtainedbyLandsat7aftergapareasintheimages.Consequently,manyapproachesandalgorithmshavebeenconductedbyresearchersinanattempttoobtainareconstructedimagethatisfreeofgaps.Generally,theseapproachescouldbedividedintotwomaincategories:single-sourceandmulti-sourcetechniques.1.2.1Single-SourceTechniqueInthistechnique,theestimationprocessofthemissingpixelsdependsonthecorruptedimageitself.Thegapareas(stripes)inLandsat7ETM+productsareusingthenon-gappedareasavailableinthesameLandsat7image.Thismethodologycanbedividedinto:simpleinterpolationapproachandsegmentationmodelapproach.1.2.1.1SimpleInterpolationApproachThismethodconsideredoneofthesimplestandmoststraightforwardwaysthatcouldbeusedtothegaps.Itismainlyaboutestimatingthevaluesofthegapareasusingthevaluesofneighboringnon-gappedareasthatlocateinthesameimage.Manymethodslieunderthiscategory:linear(i.e.mean,medianandmidpoint),nonlinear(i.e.nearestneighbor,bilinearandbicubic),andmanyothermethods[13].3Meanaverageisusedtoreplacethemissingpixelvalueinanimagewiththeaveragevaluesoftheneighboringnon-gappedpixelswhicharelocatedinaparticularsubimagewindowofthatimageitself.Themedianontheotherhand,substitutesthegappixelwiththemedianoftheneighboringnon-gappedpixels.Moreover,itperformsbetterthanthemeansinceitmaintainsthehighfrequencydetailsintheimage.Themidpointmainlycalculatesthemidpointbetweenthemaximumandminimumpixelvaluesinthesubimagewindowoftheimageandusesittosubstitutethemissingvalue[14].Thenearestneighborinterpolationinvolvesselectingthenearestknownpixelfromthefourpixelsthatsurroundtheunknownmissingonebythemodeamongthemifitisavailable.Ifthemodedoesnotexist,thenthebilinearinterpolationisappliedtothefourpixelsandthevaluethatresultsfromthisprocessissubtractedfromeachvalueofthefoursurroundingpixels.Then,itcomputesthemodeagainforthenewset[15].Inbilinearmethod,themainconditiontoestimatethemissingpixelvalueistohavefourknownpixelssurroundingthemissingone.ThekeyideabehinditistoapplyalinearinterpolationintheY-axisdirectionandapplyitagainintheX-axisdirectiontopredictthevaluescorrespondingtotheunknowngappixelofboththeYandXaxes.Althougheachstepislinear,thewholeprocessisnot,sincetheestimatedvalueofthegappixelresultsfromtheproductoftwolinearfunctions.Finally,thebicubicinterpolationisanextensiontothecubicinterpolation.Thismethodistthanthebilinearmethodbecauseitutilizes16ofthenearestknownpixelssur-roundingtheunknownmissingpixeltobeestimated.Moreover,thebicubicinterpolationgiveshigherqualityreconstructedimagesthanboththebilinearandnearestneighbormeth-ods,butitisrelativelycomplicatedtorealizealgorithmicallyanditneedsmorecomputa-tionaltime[16].4Althoughsimpleinterpolationmethodsneedlesscomputationaltimethanmethodsofmulti-sourcetechnique,theydonotgiveaccurateresults.Thisisbecausetheestimationprocessoftheunknownmissingpixeldependsontheneighboringknownpixelsofthesameimage,notonthepixelsthatcouldbetakenfromanotherscenewhichrepresenttheexactoflandcover[17].1.2.1.2SegmentationModelApproachThismodelmainlyusesthesinglesourcetechniqueintheinterpolationprocessasitdependsondatawhichexistsintheimageitself.Butsomehow,itissituatedbetweensingle-sourceandmulti-sourcetechniqueandcouldbealsounderahybridgaptechnique.Thatmeans,itdependsontheSLC-onimagetocreatethesegmentmodeland,simultaneously,itusesdatapointswhichexistintheimageforthegaps.Inthisapproach,amulti-scalesegmentationmodeliscreatedfromLandsat5(TM)orLandsat7(ETM+)SLC-onimage.Thesegmentmodelisoverlaidonanimagetopredictthespectraldatathatintersectwithsegmentmodelboundary.Thesecoincidentalspectraldatathatareextractedfromtheimagewouldbeusedtoinvaluesforthemissing(gap)pixels[12,13].Accordingto[18],thefundamentalreasonbehindusingasegmentationmodelinthisapproachistoobtaintheclosestqualitiesofthelandscapeofanimage.Consequently,SLC-onandscenesshouldbeselectedtohaveashorttimeintervalbetweenthemtoensurefewerchangesinvegetation.Moreover,highlysaturatedpixelscorrespondingtocloudsareexcludedfromtheanalysisandminimalsnowcoverisrequiredintheSLC-onimagetogetabettersegmentationmodel.Inordertoensurehavingmoreintersectedspectraldatabetweenthesegmentmodel5andtheimage,scalesegmentmodelofsizes10,15and20havebeenusedinthisapproach.Thescaleischangedaccordingtothegapsize(stripeswidth).Accordingtoresultsafterapplyingthismodel,thebestinterpolationwasobtainedinhomogeneouslandscapessuchassceneswithgrasslandandforestcoverusingscale10[18].Scenesincludingnarrowobjectslikeroadsandriverswouldgivelessaccurateresults,becausetheyhavelowaccuracyofthelandcover[11,19].1.2.2Multi-SourceTechniqueUnderthistechnique,themissing(gap)pixelsinanimageareestimateddependingonanotherimageobtainedfromLandsat7ETM+beforeorafterSLCfailure.Furthermore,imagesobtainedfromLandsat5TMalsocouldbeusedintheestimationprocess.InadditiontousingimagesfromLandsatsensor,auxiliaryimagesfromtothersensorscouldbealsoutilizedinthegapsofLandsat7images.Consequently,themulti-sourcetechniquecanbedividedintotwomaincategories:Landsatmulti-sourceandnon-Landsatmulti-sourceapproaches.Thesuperiorityofthistechniqueresultsfromreplacingtheunknowngappixelswithknownpixelswhichrepresenttheexactofthelandcover.1.2.2.1LandsatMulti-SourceApproachThedatausedforgapsinthisapproachisobtainedfromhyperspectralLandsat7ETM+scenesbeforeandaftertheSLCfailureorfromhyperspectralLandsat5scenes.Im-agesthatareselectedtothegapsarecalledscenesandthosethathavegapareasarecalledtargetscene[20].Minimumtemporal,spectralandspatialvariabilityarepreferredbetweenthesceneandthetargetscenetoensureaslowchangeoflandcover(i.e.vegeta-6tionandurbanexpansion).Also,thesimilarityofadjacentspectralbandsandgeometricalcharacteristicofnaturalimagesareessentialrequirementsforgapalgorithms.1.2.2.1.1USGSandNASAApproachInspiteofthelossofabout22%ofthedataduetotheSLCfailure,theradiometricandgeometriccharacteristicsoftheimagesstillmeetdesignspTherefore,aftertheSLCfailure,theU.S.GeologicalSurvey(USGS)andtheNationalAeronauticsSpaceAdministration(NASA)studyteamshaveevaluatedandproposedtwomainalgorithmstoreconstructimagesofETM+Landsat7satellite.Toensurethebestreconstructedimagesinbothalgorithms,therearesomestandardsthatshouldbetakenintoaccounttochoosethescenes.Thatis,sceneswithminimumcloudsandsnowcover,minimaltemporalcesandtargetsceneshouldbeplacedascloseaspossibletothescene[6,21].ThesetwoalgorithmsusetheknownpixelsinSLC-onandLandsat7ETM+imagestollthegappixels.Thus,thesealgorithmshavebeenintophaseoneandphasetwoapproachesdependingontheselectedimagesthatwereusedtothegaps.1.2.2.1.1.1PhaseOneModelAfteraworkshopconductedbyUSGS/NASAinOctober2003,theversionofphaseonealgorithmwasannouncedinJune,2004.Inthismethod,thesceneisselectedfromSLC-onproductsthatwereacquiredpriorto2003.Generally,thisphaseconsistoftwoalgorithms,GlobalHistogramMatching(GHM)andLocallinearHistogramMatching(LLHM)[4].AlineartransformationbetweenthetargetsceneandtheSLC-onscenehasbeenperformedtocalculatethemeanandstandarddeviationthroughalocalizedlinearscaling[13].GlobalHistogramMatching(GHM),itisthegenerallysimplemethod7thatisusedforthegapsinLandsat7images.TheoriginalbasisofthismethodistothelineartransformationofthewholeSLC-onscenethroughcalculatingthegainandbiasfromallpixelsoftheimage.Thus,itperformswellinscenesofinvariantterrainswhilesomeerrorsmayappearinscenesofheterogeneouslandscapes[5,22].LocallinearHistogramMatching(LLHM),toensureagreaterprecision,thismethodisproposedtothegaps.Inthisalgorithm,thelineartransformationcanbefoundbycalculatingthegainandbiasofSLC-onimagepixelswithinalocalizedmovingwindow[23].Toavoidacomputationallyexpensivelinearthegainandbiasarecalculatedusingthemeanandstandarddeviation.Themeanandstandarddeviationofthesceneareadjustedthroughalocalizedlinearscalingtomatchthestatisticsofthetargetscene.Highlysaturatedpixelscorrespondingtocloudsareexcludedfromanalysis[4].1.2.2.1.1.2PhaseTwoModelAsanimprovementtothephaseonemethodology,thisapproachhasbeenproposedinNovember2004togapsusingonlyproducts[20].AdaptiveWindowLinearHistogramMatching(AWLHM),itisbasedonthesamepresumptionasLLHM,exceptthatthemovingwindowsizeisnotconstant.Possibly,morethanonellscenesareusedinaniterativeregressionprocess.ThismethodtakesadvantageofthefactthatthemissinggapsinLandsat7productsaretime-varyingwhichallowsfusionofadjacentscenestoreducethegaps.Thelinearregressionprocessconsistsoftwoparameters(gainandbias)andisoverdetermined.Again,pixelscorrespondingtocloudandsnowareexcluded[20,24].Someresearchstudieshavebeen8proposedandcomparedtotheUSGSalgorithms.Asstatedin[24],acomparisonamongGHM,LLHM,AWLHMandadaptivewindowregressionalgorithmshasbeenperformed.Inthisresearch,thegapwasdividedintothreetypes:largeedgegap,largeinternalgap,andsingleinternalgap.Accordingtothegapwidthandposition,anappropriateUSGSalgorithmisselectedatatime.Ifalargegaplocatesattheedgeoftheimage,theGHMalgorithmbasedonmeanandvariancematchingisperformed.Ifalargegaplocateswithintheimage,thenGHMbasedonmeanmatchingisperformed.Finally,asmallgapinthemiddleoftheimageisrestoredusingAWLHMalgorithm.TheadaptationofselectingtheappropriateUSGSmethodgiveswhatiscalledtheadaptivewindowregressionalgorithm[13,22].Accordingto[25],thelocalcorrelationanalysismethodisusedtothegappixelsinLandsatimageswhereimages,withminimumtemporalvariabilityarerequired.Toconverttheimageintoaone,atransformationusinglocalrelativeradiometricnormalizationandsimilarspectralpixelsisused.TheresultshaveshownhigherqualityreconstructedimagesusingthismethodthanusingLLHMmethod.Alsoin[26],acomparativestudyhasbeenperformedbetweenthesingle-sourceandthemulti-sourcetechniques.Someclassicalsingle-sourcemethods(simpleandspatialinterpo-lation)havebeenselectedtobecomparedwithLLHMmethod.MSE,RMSEandvisualcomparisonwereusedtoevaluatetheresults.TheresultsshowedthesuperiorityofLLHMmethodoverthesinglesourcemethodsintermsoflessererrorsandsmootherrecoveredimages.1.2.2.1.2KrigingGeostatisticalApproachThisapproachdependsonspatialdatavaluesthatareclosetoeachotherduetothepossibilityofbeingmoresimilarthanthe9distantpixels.Ithasalimitationinprovidingacertainestimationofthemissingpixelsespeciallyatthejunctionareas(i.e.edgesbetweenriversandbareground).Furthermore,itmaynotbepracticaltouseitinthegapsduetoitscomplexityandslowcomputingtime[21,27,28].In[27],theordinarykrigingandthestandardizedordinarycokriginghavebeenusedinthegapsofLandsat7images.Theordinarykrigingisanon-stationaryalgorithmthatusestheprimarydataoftheindividualtargetimagestoestimatethemissingpixelvalues,whilethestandardizedordinarycokrigingusesthesecondarydataforpredictionwhentheprimarydatatobeestimatedissparseandimperfectlycorrelated.ResultsofbothkrigingandcokrigingwereassessedusingtheVariogramwhichcalculatesthedigitalnumbervalues(DN)oftheimageandthen,describesthespatialcorrelationoftheselectedsamples.Inadditiontotheaforementionedstudy,ageostatistical-basedinterpolationapproachusingkrigingandcokrigingalgorithmswasproposedin2009.Asaprinciple,toestimatetheexactoflandcoverinpixellevel,threecloud-freeimagesthatweretemporallyclosetoeachother,wereselectedandarrangedfromtheearliestdatetothelatestdate.Themiddleimagerepresentedthetargetimageandtheothertworepresentedtheimages.Cokrigingwasusedtoestimatetheunknownmissingvaluesinthetargetimageusingthesecondarydatafromimages.Ifthesecondarydatawerenotenoughtogaps,theestimationprocesswouldbeswitchedtousekrigingapproach.Inadditiontothevariogrmsofgeostatisticalapproach,somesimplecompositingmethodswereproposedtobecomparedamongstthemselves:simplecomposite(dependingonaverage),globalmeanandvariancecorrelationcomposite,andlocalmeanandvariancecomposite.Furthermore,threehybridestimatorsweredevelopedasareactiontotheslownessofthe10geostatisticalmethodandtheinabilityofcompositeestimatorstopredictalltheunknownsinthetargetimage[28].1.2.2.1.3SpectralProjectionTransformationApproachThisapproachmainlyde-pendsonthepropertyofspectralprojectiontransformationofthehyperspectralETM+im-ages.Thespectraltransformation,inthismethod,wasbasedonthePrincipleComponentTransformation(PCT)algorithm.Thatis,aftertheimages'co-registration,eigenvaluesandcorrelationmatrixwereobtainedusingsampledataselectedfromtheknownpixelvaluesoftheimage.Then,atransformationusingPCTwasappliedonSLC-onimagepixelvaluesthatcorrespondedtotheunknownmissingpixels(strips)oftheimage.Thistransformationwouldgiveanewsetofknowndata.Thestatisticsobtainedfromtheknownpixelswereusedtoinverselytransformthenewsetofknowndataobtainedpreviously.Finally,theinverse-transformedknownsetofdatawasusedinthegapsinimage.ThismethodwascomparedtoLLHMmethodusingvisualcomparisonandUniversalImageQualityIndex(UIQI).ResultsshowedthesuperiorityofthismethodoverLLHMespeciallyinagricultureandforestterritories[29].1.2.2.1.4NeighborhoodSimilarPixelInterpolator(NSPI)ApproachMostpre-viouslymentionedapproacheshaveaweaknessinnotperformingwelloverhetero-geneousregions,especiallywhenthereisalongtimeintervalbetweentargetandimages.Therefore,NSPIhasbeenproposedasasimpledeterministicinterpolationmethodtoover-comethisshortcoming.Thebasicideaofthisapproachisthattheknownneighboringpixelsintheimage,locatedwithinthesameclasswiththeunknown(gap)pixelsinthetargetimage,showsamespectralcharacteristicsandsubsequently,thesametemporalTherefore,neighboringpixelswithsimilarspectralproperties(similarpixels)andsmaller11spatialdistancetotheunknowngappixelhavemoreweightthanothers.ThisapproachwasappliedtooneSLC-onsceneandcomparedtoLLHM.Furthermore,itwasappliedtomultiplescenesandcomparedtoAWLHM.AccurateresultshavebeenacquiredusingNSPI,whichthesuperiorityofthismethodoverLLHMandAWLHM[19].ThedisadvantageofNSPIisitsinabilitytoprovidethecertaintyofpredictionduetolandcoverchangesandtheneedformorecomputingtime[21].Also,amoNSPIhasbeenpresentedbyZhutodealwithimagesthathavecloudcontaminationproblemandtoreducetheedge[30].AsanimprovementtoNSPI,amogeostatistical-basedmodelcalledGeostatisticalNeighborhoodSimilarPixelInterpolator(GNSPI)wasproposed.Inthismethod,allscenesshouldbegeometricallyrbeforeapplyingthealgorithmtomatchthetargetscene.Afterclassifyingtheimageintothreeclasses(water,baregroundandvegetation)dependingonthesimilarspectralcharacteristics,atemporalmodelcanbedeterminedbe-tweenthetargetandtheimages.Asaresult,aresidualimagecanbeobtainedfromthepixelvaluesoutsidethegaps.Usingthisimage,asemivariogramforeachclasscanbemodeleddependingonthespatialrelianceofitsresiduals.Moreover,atemporaltrendwasdeterminedutilizingsimilarpixels,wheresimilarpixelsfromeachclassproduceidenti-caltemporalvariabilitypatterns.Finally,ordinarykrigingwasusedtopredicttheresidualsofthetargetimage.Theseresidualsprocessedwiththetemporalmodelcangivethepredictionofuncertainty.ThismethodusedbothLandsat5andLandsat7scenesasinputimagesandtheresultsshowedthatGNSPIhadovercomeNSPI,buttheonlydisadvantagewastheslowcomputingtime[21].Also,anewintegratedmethodologywasproposedasaalgorithmforLandsat7ETM+images.Multi-temporalimages(temporallyclosetothetargetimage)12wereusedasimages.Thismethodemployedtwostagestothegaps;aWeightedLinearRegression(WLR)modelwasperformedtoestimatetheunknownmissingpixelvaluesusingmultipleimages,andthenaNon-referenceRegularizationmodelwasperformedonlyifgapareasinthetargetimagestillexisted.ResultsobtainedusingtheproposedmethodhaveeditssuperiorityoverNSPIandLLHM.However,theonlydisadvantagewastheslowcomputingtime.Accurateestimationoftheunknownpixelswasobtainedusingthisapproach,especially,intheedgeswherethemulti-temporalscenesvariedregularly(seasonalvegetation),whileinaccurateestimationwasobtainedwherethevariationwasabrupt(humanintervention)[11].AnotherstudydependingonthesameassumptionofNSPImethodwaspresentedtothegapsinimages.Targetandimageswereselectedtobetemporallyseparatedbyoneyeartocapturethetemporalvariabilityproperties.inthisstudy,eachsceneisintotclasses(water,forest,vegetationandbareground),thenthesameclassofbothtargetandimageswerecomparedtodeterminethesimilaritybetweenclassestoevaluatethealgorithm[31].1.2.2.1.5AMorphology-StitchingApproachAghamohamadniaandAbedinihaveproposedanewmethoddependsonusingthemovingneighbor-selectionkernelandappliesitonthetwolayersborderlinesbetweenthestripesanditsadjacentarea.AttheborderlinewasextractedusingthemasklayeroftargetimageandSLC-onimage.Afterthat,theadjacentneighboringKnownpixelsonbothsidesweredetectedandthenearestonetothecenterpixeloftheborderlinewasselected.Subsequently,twonewpixelsandastitch-lineweredetermined.Thisprocesswasrepeatedfromthecenterpixelstobeappliedonthetwoleftadjacentneighboringpixel.Thismethodwasappliedonband413ofLandsat7ETM+image,theresultsshowedanimprovementintermsoferrorpercentageandreconstructedimageappearancecomparedtoLLHMandAWLHMmethods[32].1.2.2.2Non-LandsatMulti-SourceApproachAsmentionedinsection1.2.2.1,theperformanceofgapalgorithmsdependsontheselectionoftargetandimages,wherecloud-freeimageswithlesstemporal,spatialandspectralarerequired[21].Therefore,thoseperfectcloudfreeimagesarese-lectedfromthesameseason;sensorsotherthanLandsatshouldbeavailabletoprovidetheneededauxiliaryinformation,incasetheywerecontaminatedintheLandsatsensor[33].Furthermore,thetemporalresolutionoftheLandsatsensorcannotcatchtherapidchangeinvegetation;thusnon-Landsatsensorswithhighfrequencytemporalresolutioncouldbeemployedtocatchthesedetails[34].Anapproachwasproposeddependingontheauxiliarymulti-temporalimagesprovidedbyModerateResolutionImagingSpectroradiometer(MODIS)BRDF/Albedolandsurfacecharacterizationproduct.Asemi-physicalfusionbetweenMODISBRDF/AlbedoandLand-sat7ETM+productswasimplementedperpixeltoestimatetheunknownmissingpixelsofLandsat7imagesandcloudcoveredimages.Astheshortwavelengthsaremoresusceptibletoatmosphericthismethodshowedmoreaccurateeestimationinnearinfraredbandsthaninvisiblebands.Thisapproachwassimpletoimplementandwasbytemporalchangesbutatthesametimeitalsorequiredtheco-registrationoftimagesandthedetectionofcloudpresenceforLandsat7scenes[35].OtherresearchershaveusedimagesfromIRS/IDLISS-IIIsensortoreconstructtheSLC-imagesofLandsat7ETM+.TothegapsinETM+imagesperfectly,twostagesarerequiredtoimplementthisapproach.First,usingalinearregressionmodel,bands14(2,3)ofLISS-IIIwereusedtothegapsinbands(3,4)ofETM+sincetheyweretheonlycomparablebandsbetweenthetwosensors.ThisstepwillproducetwonewSLC-onimagesinbands(3,4).Second,thetwonewreconstructedimagesinbands(3,4)areusedtothebands(1,2,5,6and7)throughtwomethods.Themethodistoalinearrelationshipbetweenbands(3,4)andETM+bands(1,2).Thesecondmethodistoapplyaplannerrelationshipbetweentherecoveredbands(3,4)andbands(5,6and7)[2].Anotherstudywasconductedtoestimatetheun-scannedpixelvaluesinETM+SLC-imagesusingthedataobtainedfromAdvancedLandImagery(ALI)onboardtheEarthObserverOne(EO-1).Basedonthestatisticalqualitiesoftheimages,thetwodatasetsfromLandsat7ETM+andEO-1/ALIhavethesamespectralrangeweredesignatedtobeusedinRegressionBasedDataCombination(RBDC)approachtothegapareas.Twoordinarytechniques,sceneBased(SB)andClusterBased(CB),andtwostatisticaltechniques,Based(BB)andWeightedBased(WBB),werecomparedandtheresultsshowedthatthestatisticalapproachesprovedtobemoretthantheordinaryapproache[36].DuetothespectralsimilaritybetweentheChinaBrazilEarthResourcesSatellite-02B(CBERS-02B)andLandsat7/ETM+,anewstudywaspresentedtothegapareasinETM+images.Thisstudywasbasedonimplementingandcomparingsomefamousalgorithmsinwhich,Simple,GHM,LLHMandAWLHMalgorithmswereused.ResultsindicatethesuperiorityofAWLHMoverotheralgorithms,whereimageswithlessererrorsandhigherqualitywereobtained[37].DuetothelackofresearchaboutrecoveringthethermalbandofLandsat7ETM+images,anadditionalapproachusingauxiliarydatafromtheCBERS-01sensorwaspresented.Alinearcombinationofbands3and4fromCBERS-01wasutilizedtogapareasinthethermalband.ThestudyproposedamoAWLHMtobeappliedonscenesfromboth15sensorsandcomparedtotheregularAWLHM.MoreaccurateresultswereobtainedusingthemoAWLHMmethodthantheregularAWLHMmethod.Duetotheunbiasedmannerofgappixelprediction,thereconstructedthermalbandcouldbeusedinstudyingtheurbanthermalscenes[10].1.3ThesisOrganizationTherestofthethesisisstructuredasfollows.InChapter2,wediscussindetail,someexistingsingle-sourceandmulti-sourcerecoveryapproaches.TheproposedLLM-modeldrivenap-proachanditsmathematicalanalysisareintroducedinChapter3.ExperimentalresultsandperformanceevaluationofbothselectedandproposedalgorithmsarediscussedinChapter4.Finally,thethesisisconcludedandfutureworkisprovidedinChapter5.16Chapter2EXISTINGRECOVERYMETHODOLOGIESInthischapter,wepresentsomealgorithmsandmethodsthathavebeenusedforthegap-inLandsat7ETM+products.Tocoverthetwomaintechniquesusedinthisareaofstudy,wediscuss,indetail,twomethodsofthesingle-sourcetechnique.Then,weexplainandanalyzetwomainalgorithmsofthemulti-sourcetechnique.AllofthesemethodswillbecomparedwiththeLocallyLinearManifold(LLM),whichwillbeintroducedinChapter3.2.1Single-SourceTechniqueInthistechnique,theestimationprocessofthemissingpixelsdependsonthecorruptedimageitself.Thegapareas(stripes)inLandsat7productsthatwereacquiredafter2003canbeusingthelinearandnonlinearorbyapplyingthespatialinterpolationapproach.2.1.1MovingAverageFilterIngeneral,linearestimatevaluesoftheunknown(gap)pixelsusingvaluesoftheneighboringnon-gappedpixelsthatarelocatedwithinthesamesubimagewindow.The17averageisoneofthemorestraightforwardmethodsthatcouldbeusedtoreplacethemissingpixelvalueinanimagewiththeaveragevaluesoftheneighboringnon-gappedpixelsthatarelocatedinaparticularsubimagewindowoftheimageitself.Sincetheestimatedreconstructedpixelsareacombinationofnon-gappedpixelsintheimage,thentheaverageisconsideredasalinear[38].Inthismethod,thesizeofthewindowshouldbeconsideredtoavoidblurring.Whenablurisnoticedintheoutputimage,thewindowsizeshouldbedecreasedanditerationshouldbeusedtointhegapsleftinthescene[14].ThismethodcanbeperformedbyextractingamovingsubimagewindowWofsizennfromthefullimage,thentheaveragevalueYofpixelsI(i;j)withinthiswindowiscomputedtoreplacethezero-valuepixelsofthegapareas[14].Y=1n2X(i;j)2WI(i;j)(2.1)2.1.2SpatialInterpolationTechniqueItisknownthatinterpolationisaprocessofpredictingthemissingvaluesbasedontheknownsamplepointslocatedwithinthesamearea.Itcanbedividedintodeterministicandgeostatisticaltechniques[39,40].Generally,thespatialinterpolationisbasedontheTobler'sLawofGeographyinwhichallpixelsinascenearerelatedtoeachother,butthecloserpixelshavemoresimilaritiesthanthedistantones[21].Therefore,thenumbersofknownsamplepointsandtheirlocationhaveatroleinestimatingtheunknownmissingvalues.182.1.2.1InverseDistanceWeight(IDW)ApproachIDWisoneofthedeterministicspatialinterpolationmethodsusedbythegeographicalinformationsystem(GIS)anditisbasedontheassumptionthatthenearestknowndatapointshavemoreweightinpredictingthemissingvaluethanthefarones[41,42].Itusesamathematicalfunctiontobetlyappliedinareaswherethedatapointsareuniformlydistributed[42,43].Inthisinterpolationmethod,themissing(gap)pixelsareestimatedusingasetofknown(non-gapped)pixelsthatfallwithinthesamearea.Theaccuratepredictionofthesemissingvaluescanbeobtaineddependingonthedistancebetweentheknownandunknownvalues,inwhichthelargerdistancebetweentheknownandtheunknownvaluesdenotesthelesserimpactoftheknownsamplepointsintheestimationprocess[43].UsingasetofneighboringknownsamplepointsLi=L(Ri),thevalueLthatinterpolatedfromthisneighboringsetatagivenpointRcanbeobtainedasinthefollowingequation[26,44]:L(R)=KXi=0Vi(R)LiPKj=0Vj(R)(2.2)whereKreferstothenumberofknown(non-gapped)pixels,ViisasimpleweightingfunctionwhereVi(R)=1d(R;Ri)p,dreferstothedistancebetweentheunknown(gap)pixelRandtheknownpixelRiandpisthepositivepowerparameter.Asitcanbeseenfromequation2.2,knownpixelsthatarefarfromtheunknownpixelRwouldhavelesserweightthanthecloserones.Furthermore,alargevalueofptendstogivelargerweightstotheclosestpointsandsmallerweightstothedistantpoints[45].192.2Multi-SourceTechniqueInthistechnique,theestimationprocessofthemissingpixels,inthecorruptedimage,dependsonanotherimageobtainedfromLandsat7ETM+beforeorafterSLCfailure.Furthermore,imagesobtainedfromLandsat5TMalsocanbeutilized.AfterLandsat7SLCfailureonMay2003,theUSGSEarthResourcesObservationSystems(EROS)DataCenter(EDC)hascreatedtwoalgorithmstothegapsinscenes.Phaseone/LocalLinearHistogramMatching(LLHM)algorithmandphasetwo/AdaptiveWindowLinearHistogram(AWLHM)algorithmarethetwomainalgorithmsthathavebeenselectedforthisstudytointhemissingvaluesinLandsat7productsthatwereacquiredafter2003.2.2.1PhaseOne/LocalLinearHistogramMatching(LLHM)Al-gorithmPhaseoneLLHMalgorithmisoneofthemainmethodsthathavebeenproposedbyUSGS/NASAduetotheSLCfailurein2003.OnlyonesceneisselectedfromSLC-onproductsofLand-sat7thatwereacquiredpriorto2003.AsshowninFigure2.1,thelinearhistogramcanbeappliedbyalineartransformationbetweentheSLC-otargetimageandtheSLC-onimage[4].AfterselectingtheSLC-onimageandthetargetimage,LLHMapproachcanbeperformedusingthefollowingsteps:1.Extractasuitablennmovingwindowaroundthemissing(gap)pixelfrombothscenes[5].2.Withinthismovingwindow,eachpixelisexamined.Thenon-gappedpixelvalueinthe20targetimagewillberetainedasitis,whilethemissingpixelhastoberetrieved.3.Toestimatethevalueofthemissingpixelinthetargetimage,acorrectivegainandbiasarecalculatedandappliedtothecorrespondingpixelintheSLC-onimage.4.Theabovestepswillbeiteratedtoallmissing(gap)pixelsintheStargetimage.Figure2.1:Alineartransformationbetweentheimageandtarget)image[4].TheLLHMmethodhastheadvantageintermsofspeedandsimplicityofimplementa-tion.Furthermore,toobtainabetter-reconstructedimageusingLLHMtechnique,therearesomerequirementsthatshouldbetakenintoconsideration.InordertoensureantperformanceofLLHM,thefollowingconditionsshouldbeconsideredforboththetargetandSLC-onimages[4,27]:Highlysaturatedpixelscorrespondingtocloudsareexcludedfromanalysis.Aminimalsnowcoverisrequiredforbothtargetandscene.21Temporalcorrelationsmustbeexploitedforinterpolationwhichmeansaslowchangeoflandcoveroveryearlyperiods.AllimagesforboththetargetandSLC-onscenesshouldberegisteredforanaccurateresult.2.2.1.1LLHMMathematicalAnalysisInphaseoneLLHMapproach,thegainandbiasarecalculatedusingmeanandstandarddeviationandsecondstatisticalmoments).Meanandstandarddeviationofthesceneareadjustedthroughalocalizedlinearscalingtomatchthestatisticsofthetargetscene.Thesecalculationsareshowninthefollowingequations[4,6]:g=˙t(off)˙f(on)(2.3)b=t(off)gf(on)(2.4)wheregdenotesthegainusedtohistogrammatchingbetweenthelandtargetimages,bdenotesthebiasusedtohistogrammatchingbetweentheandtargetimages,˙t(off)denotesthestandarddeviationofthetargetimage,˙f(on)denotesthestandarddeviationoftheSLC-onimage,t(off)denotesthemeanofthetargetimage,andf(on)denotesthemeanoftheSLC-onimage.Themeanandstandarddeviationforbothtargetandscenescouldbecalculatedasshownbelow:t(off)=1NNXi=1targetDNi(2.5)22˙t(off)=vuut1(N1)NXi=1(targetDNit(off))2(2.6)f(on)=1NNXi=1fillDNi(2.7)˙f(on)=vuut1(N1)NXi=1(fillDNif(on))2(2.8)whereNindicatesthetotalnumberofpixelsintheimage,fillDNiindicatesthevalueofpixelifromtheSLC-onimageandtargetDNiindicatesthevalueofthecorrespondingpixelifromthetargetimage.Afterobtainingthegainandbias,bycalculatingthemeanandstandarddeviationforbothscenes,thegainandbiasvaluesareappliedtothecenterpixelvalueXintheSLC-onimagetogetthenewpixelvalueYwhichwillbeusedtothegapsinthetargetimageasillustratedinFigure2.1[4,6,26]:Y=gX+b(2.9)2.2.2PhaseTwo/AdaptiveWindowLinearHistogramMatching(AWLHM)AlgorithmPhasetwoAWLHMalgorithm,adevelopmenttothepreviousphaseoneLLHMalgorithm,isalsosuggestedbyUSGA/NASAafter2003asaresulttoSLCfailure[6].TherearetwomainbetweentheLLHMandAWLHMalgorithms.Theebetweenbothalgorithmsistheselectionofthescenes.Probablymorethanonescenesareselected23fromproductsofLandsat7thatwereacquiredbefore2003.Theseconderenceisthepropertyofusinganadaptivemovingwindowaroundgapsinthetargetimage.Thesizeofthisadaptivewindowchangesaccordingtothesizeofstripsintheimagesandcouldbeincreasedtoreachthemaximumwindowsize[20].Figure2.2illustratestheprocessofincreasingthewindowsize.Figure2.2:Alinearhistogramadaptivewindowprocess,thesmallredsquareatthecenterrepresentsthegappixelsneededtoberecovered,thereddashedlinesrepresenttheincreasingprocessofwindowsizeandthebigredsquarerepresentsthewindowsizethatcouldbereached[4,5].ToapplyAWLHMmethod,asastep,morethanonelimagesmustbechosentothegapsinthetargetimage.Moreover,afewchangeshavetobedoneonLLHMapproachtogettheimprovedAWLHMalgorithmasexplainedinthefollowingsteps:1.Extractannnmovingwindowaroundthemissing(gap)pixelinthetarget24imageandtheimage[5].2.Withinthismovingwindow,startwithaminimumsub-windowsize11forbothscenes.Computethenumberofcommonnon-gappedpixelsandcompareittothepredeterminedvalueoftheminimumnumberofcommonnon-gappixels.Ifthisvalueisnotmet,thenincreasethesizeofthesub-windowbyonepixeloneachsidetobecome33;55;:::untiltheminimumcommonpixels'valueismetorthemaximumwindowofsizennisreachedasillustratedinFigure2.3.3.Whentheminimumcommonvalueismet,calculatethegainandstandarddeviation.Checkforacceptablegainbycomparingittothemaximumallowablegainasitwillbeexplainedinsection2.2.2.1.Applythecalculatedgainandstandarddeviationvaluestothecorrespondingpixelatthecenterofthesub-windowinimage.Eventually,usethisnewpixelvaluetothegappixelintargetimage.4.Repeattheabovestepsforeachgapinthetargetimageuntilallared.Ifsomegapsstillexistinthereconstructedimage,thenusethisimageastheSLC-targetimageandpickanotherimagefromalistofimagesanditeratetheprocedureuntilallgapsarerecovered.Theabovestepswillbeiteratedtoallmissingpixelsinthetargetimage.Again,allscenesmustberegisteredbeforeapplyingthealgorithm.TheadvantageofAWLHMapproachisduetoallowingfusionofadjacentscenestoreducethegapsbecausethemissingpixelsinETM+Landsat7productsaretime-varying.Therefore,itgivesmoreaccurateresultsthanLLHM,butneedsextratimeforcomputation.ToobtainabetterrecoveredimageusingAWLHMtechnique,thesamerequirementsforselectingscenes,whichhavebeendiscussedinLLHMapproach,shouldbetakeninto25Figure2.3:Aschematicdiagramoftheadaptivewindowprocess(mofrom[6]).PixelPthatneedstoberecoveredisonthesamelocationintargetandtrainingimagesasaresultofimageregistration.In(a),thewindowsizeusedis55,theresultedcommonpixels=12.Ifthisvaluedoesnotmeettheminimumrequiredcommonpixels,thewindowsizewouldbeexpandedbyonepixeloneachsideasin(b).Thewindowsizein(b)willbe77,thecommonpixelsarecomputedagain.Ifthenewvaluedoesnotmeettheminimumrequiredcommonpixels,continueincreasingthewindowsizeuntilmaximumwindowsizeisreachedconsiderationforthismethodaswell.Thus,toensureagoodperformanceofAWLHMandtopreventbadresults,aminimalsnowcoverispreferredinthemultiplescenes,highlysaturatedpixelscorrespondingtocloudsareexcludedfromanalysisandaslowchangeoflandcoveroveryearlyperiodsmustbeexploitedforinterpolation[20].262.2.2.1AWLHMMathematicalAnalysisInphasetwoAWLLHMalgorithm,thelinearregressionprocessconsistsoftwoparameters(gainandbias)anditisusedtothetransformationbetweenthetargetimageandimage.Thisprocessisappliedontheover-determinedlinearsystemtocalculateitsleastsquaressolutionasshownbelow[20]:266666666641fillDN11fillDN2......1fillDNN37777777775264bg375=26666666664targetDN1targetDN2...targetDNN37777777775Thentheleastsquaressolutioniscalculatedasfollowing:264bg375=26664NNPi=1fillDNiNPi=1fillDNiNPi=1(fillDNi)237775126664NPi=1targetDNiNPi=1(fillDNi)(targetDNi)37775wherebdenotesbias,gdenotesgain,Nisthetotalnumberofpixelsintheimage,fillDNidenotesthevalueofpixelifromSLC-onimageandtargetDNidenotesthevalueofthecorrespondingpixelifromtargetimage.Aftercomputingthegainandbias,afurtherstepisperformedtodistinguishAWLHMfromLLHM.Thisstepistocheckwhetherthecomputedgainissuitabletopreventoutliersfromhavingastrongonthetransformation[5].Therefore,thegainiscomparedtothemaximumandminimumallowablegain;andaccordingtoreferences[6,20]themaximumallowablegainisdeterminedtobeequalto3.Ifthecomputedgainisfoundtobereasonableinwhichitlieswithintherange13<g<3,thenthegainandbiaswillbecalculated27accordingtoequations2.3and2.4.Otherwise,accordingtoreference[20],thegainissettobeequaltoonewhileonlyabiasadjustmentisappliedtothegapsasshownbelow:g=1b=t(off)f(off)(2.10)Finally,equation2.9wouldbeusedtocalculatethenewmergedpixelvaluefromtheimagetoestimatethemissinggappixelinthetargetimage.Theaboveproceduresshouldbeappliedoneachgappixelexistsinthetargetsceneinordertoallmissingpixels.2.3SummaryInthischapter,wereviewedsomeexistingrecoveryapproachesthathavebeenusedforgap-inLandsat7ETM+products.First,wediscussedindetailtwosingle-sourcemethodswhicharethemovingaverageandtheInverseDistanceWeight(IDW).Theaverageismainlyaboutcalculatingtheaveragevalueofallneighboringpixels,whenthepixelvalueisequaltozero,thenreplacesthecenterpixelwithinthemovingwindowwiththisaveragevalue.Then,wediscussedtheIDWmethodwhichisadeterministicspatialinterpolationmethod.Thatis,themissing(gap)pixelsareestimatedusingasetofknown(non-gapped)pixelsthatarelocatedwithinthesamearea.Theaccuratepredictionofthesemissingvaluescanbeobtaineddependingonthedistancebetweentheknownandunknownvalues,wherethelargerdistancebetweentheknownandthemissingvaluesindicatesthelesserimpactoftheknownsamplepointsintheestimationprocess.Bothpreviousmethodsthatbelong28tosingle-sourcecategoryneedlessercomputingtimethanmulti-sourcemethods,buttheydonotprovideaccurateresults.Thisisduetopixelsrepresentinganinaccurateoflandscape.Thatis,theestimationprocessoftheunknownpixeldependsontheknownneighboringpixelsexistinginthesameimage,notoncorrespondingpixelsfromauxiliaryscenesthatrepresenttheexactoflandcover.Second,weexplainedtwomainalgorithmsofthemulti-sourcetechnique,whichareLLHMandAWLHM.InphaseoneLLHMmethod,thesceneisselectedfromtheSLC-onproductsthatwereacquiredpriorto2003.Themeanandstandarddeviationofthesceneisadjustedthroughalocalizedlinearscalingtomatchthestatisticsofthetargetscene.Highlysaturatedpixelscorrespondingtocloudsareexcludedfromtheanalysis.Finally,theAWLHMapproachisdiscussedwherepossiblymorethanoneSLCscenesareselectedinaniterativeregressionprocessandanadaptivemovingwindowisextractedaroundthegappixelthatneedstobeThepixelscorrespondingtocloudandsnowareexcludedfromanalysisaswell.AWLHMoutperformsLLHMbecauseitgivesmoreaccurateresults.However,itneedsmuchmorecomputingtimethanLLHM.29Chapter3THEPROPOSEDLOCALLYLINEARMANIFOLD(LLM)MODELInthischapter,wepresentthenewLocallyLinearManifold(LLM)modeltogapsinSLC-productsofLandsat7EnhancedThematicMapperplus(ETM+).First,wewillgiveageneralideaabouttheLLMapproachandhowitworks.Then,theestimationprocessofthemissingpixelsusingthisapproachwillbeexplainedindetailthroughtheuseofillustrationgraphs.Furthermore,amathematicalanalysiswillbeperformedtorecoverthemissingpixelsinhyperspectralimagesusingtrainingdatatoachieveaccurateresults.Finally,asummaryofthechapterwillbegiventodiscussallsections.3.1ABriefOverviewofLLMModelInthissection,wetrytoinvestigateageneralapproachtotheprobleminLandsat7(ETM+)productsacquiredafterMay2003asaresultofthefailureinthescanlinecorrector(SLC)whichcausedanestimatedlossofabout22%ofthedataineachscene[4].Inthisapproach,eachbandismodeledasanon-linear,locallymanifoldthatcanbelearnedfromthematchingbandsatttimeinstances.Furthermore,eachmissing30pixelvalueisestimatedusingknownpixelsonthesamebandandsamelocationofttimes.Usingthisapproach,theprocessinvolvesthemostfeasiblepointonthelearnedmanifoldtoapproximatethemissingpixels.Eachbandisdividedintosmalloverlappingpatches.Inparticular,eachpatchisconsid-eredtobealinearcombinationofasetofcorrespondingpatchesfromothertimeinstances.Inahigh-dimensionalEuclideanspacewhereeachpatchismappedintoapoint,thecollec-tionofregisteredpatchesovertimeconstitutesanon-linearonedimensional(1D)manifold.Therefore,inthismodel,theentirefourdimensional(4D)multispectraldata(2Dspatial,1Dspectral,1Dtemporal)consistsofasmanyindividual1Dmanifoldsasthenumberofpatchesineachbandtimesthenumberofspectralbands.3.2TheEstimationProcessoftheMissingPixelsUs-ingtheLLM-modelDrivenApproachLetusrefertopatcheswithgaps(tobeinterpolated)asthetargetpatchesandtothecompletepatchesfromothertimeinstances(tobeusedduringinterpolation)asthepatches.AsillustratedinFigure3.1,patchescanbeselectedfromLandsat7(ETM+)productspriortotheSLCfailureorfromLandsat5ThematicMapper(TM)productswhichspansabout27years(1984-2011)andhavesimilarspatialandradiometricresolutionasLandsat7(ETM+)products.Theinterpolationprocessconsistsofestimatinganapproximatelyhyperplanethatistangenttothemanifoldoverthetargetpatchandapointontheestimatedhyper-planethathasthesmallestEuclideandistancetotheobservedpartofthetargetpatch.Theconceptofthemanifoldmodelandthesetofadjacentpointsonthemanifold31Figure3.1:Anillustrationdiagramrepresentsselectingpatches(P1-Pm)fromsamelocationinband1(B1)throughrenttimeinstancest(1984-2011).thatcanbeusedtoestimatethetangenthyperplaneareillustratedinFigure3.2.Inthispatchesfromthesamelocationthatareadjacentintimeorfromthesameseasonoftheyearareapproximatelyonanspace(eachapproximatelylinearsegmentofthemanifoldcancorrespondstoaparticulartime/seasonoftheyear).ToperformtheLLMalgorithmsuccessfully,thefollowingstepsshouldbeperformedtocompletetheestimationprocessofthemissingpixelsinthetargetpatches:1.Eachpatchisextractedfrominputimageattheparticularposition.2.Torecoverthemissingpixelsinonepatchxj,weconsiderthecorrespondingdatasetatthesamelocationbutttime(onepatchtakenfromatrainingimageatthesamelocation).3.Thetrainingdatawillbesegmentedintotwoparts,whichdependsonthemissingpixelsinXthatweneedtoestimateasillustratedinFigure3.3.ThepartAcorresponds32withdatathatpatchesinXhasapixelvalue.ThesecondpartBcorrespondswithdatathatpatchesinXdoesnothavepixelvalue.4.ThecotiscomputedbasedonpartAbyprojectingthepatchthathaspixelvalueontothelinearspacespannedbyA.5.Thecotthatwascomputedfromthepreviousstepwillbeusedtopredictthemissingpixel.6.Afterthepredictionofthemissingvector,thevectorwillbecombinedwiththenon-missingvector,bereshapedfromvectorintopatchshape,andthenputbackintothepredictedimage,countingthenumberofoverlapforeachpixel.Theestimatedimagewillbeaveragedovertheoverlap.7.Finally,alltheothermissingpixelswillbeinterpolatedbyiteratingthestepsabovefortheoutputimage.AsinLLHMandAWLHMalgorithms,therequirementsofselectingscenesarealsorequiredintheLLMalgorithmtoobtainbetterresults.Thetemporal,spectralandspatialcorrelationsmustbeexploitedforinterpolation.Temporalcorrelationisduetotheslowchangeoflandcoveroveryearlyperiods,spectralcorrelationcorrespondstothesimilarityofadjacentspectralbandsandspatialcorrelationcorrespondstothegeometricalcharacteristicsofnaturalimages.Furthermore,thehighlysaturatedpixelscorrespondingtocloudsareexcludedfromanalysisandaminimalsnowcoverispreferredinthemultipleSLC-onscenes.33Figure3.2:Anabstractillustrationofa1D(onedimensional)manifoldrepresentingapar-ticularlandcoverpatchovertime.Here,each(approximatelysegmentofthemanifoldmaycorrespondtoaparticularseasonoftheyear1984through2011.3.3LLMMathematicalAnalysisThismodelaimstorecoverthemissingpixelsinhyperspectralimages,usingtrainingdatawhichisalsohyperspectralimagescapturedaboutmorethan10yearsagowhentherewasnoproblemwithLandsat7orusingtrainingdatacapturedabout27yearsago(1984-2011).Thetrainingdataiscapturedexactlyfromthesamelocation,butttimes.Theinputimageincludesimageswithmissingpixelsandweusedthemaskwith0and1valuestoindicatethemissingpixellocation.Foreachinputimage,weworkwithpatchesofsizeequalto16.Thepatchwillbemovingfromlefttoright,uptodown,andeach34missingpixelinthepatchwillberecovered.Sincethereisoverlap,weneedtosumupalltheestimatedmissingpixelsanddividethembythetotalnumberofoverlaps.Figure3.3:Aschematicdiagramillustratingthematrixsegmentationintoknownandun-knownregions.WhereXdenotesatrainingdatathatissegmentedintoknownandunknownregions,ndenotesthenumberofpatches,mdenotesthenumberofpixels,m1denotesthenumberofknownpixels,m2denotesthenumberoftheunknown(missing)pixels,Adenotestheknownregion,Bdenotestheunknownregion,denotesthecoientthatwillbeusedtopredictthemissingpixel,xjdenotesthepatchthathasmissingpixels(b)andneedstobepredicted.TheunderlyingideaofmatrixsegmentationfortheestimationprocessofthemissingpixelsusingtheLLMmodelisillustratedbyFigure3.3.ThesetofpatchescanbewritteninamatrixformX=[x1;x2;x3;:::;xn]2Rmn[46].Inthismodel,wedonotconcentrateonthesetoftangentspacesthatarespinadvanceasinreferences[46,47].Alterna-tively,anhyperplanethatistangenttothemanifoldoverthetargetpatchisestimated.Then,apointontheestimatedhyperplanewhichisxj2Rmanditwillbesetastheclosestpointoftheobservedpartofthetargetpatch[48].LetxjrepresentsapatchinamatrixXthathasknownandunknownpixels,therefore:X=xj(3.1)35whereisacots'vectorthatneedstobepredicted.AsillustratedinFigure3.3,thematrixXcanbesegmentedintotwoparts.PartAcorrespondswithdatathathasknownpixelvalues,andpartBcorrespondswithdatathathasunknown(missing)pixelvalues.InsteadofusingthewholematrixX,onlytheknownpartAwillbeusedtoestimatethecots'vector:=a(3.2)1.Assumethatm1<n,wherem1denotesthenumberofknownpixelsinA,thenwehavemoreunknownsnthanknownequationsm1whichleadstoansetofsolutionsasappearedinequation3.2.Therefore,oneofthesolutionscouldbefoundusingtheminimumEuclideannormregression.Thissolutioncouldbefoundasshownbelow:=argminkk2subjectto=a(3.3)Asasolutiontotheequationabove,isgivenby:=A+a(3.4)Herek:kisthe`2normthatscanstheEuclideandistanceofapointontheestimatedhyperplanetotheobservedpartofthetargetpatch,istheestimatedcots'vector,andA+isthepseudoinverseofthematrixA.36Afterhasbeenfound,itwouldbeusedtoestimatethemissingpixelb:B=b(3.5)2.Whenm1>n,thenwehavemoreknownequationsm1thanunknownsn.Therefore,aleastsquaresregressionisusedtominimizethesumofsquaresofallerrorsofeverysingleequation.Consequently,theequation3.3wouldbechangedtobecomeasshownbelow:mink()ak2(3.6)Inthiscase,wehavetoomanyequationsandthen=amayormaynothaveasolution.However,ourgoalistominimizetheerrorsintheresults.AsitwillbeshownanddiscussedinChapter4,theLLM-modeldrivenapproachoutper-formsallsingle-sourceandmulti-sourcemethodsthathavebeenmentionedinChapter2.Itgiveshighqualityresultswithlesspossibleerrors.TheonlyconcerninLLMapproachisthecomputingtime.However,comparingtothestate-of-the-artAWLHMapproach,LLMneedsmuchlesstimethanAWLHMandconcurrentlyitgivesbetterreconstructedimages.Therefore,Intermsofcomputingtimeneededforgaps,thisapproachliesinthemid-dlebetweenLLHMandAWLHM.Whileintermsofaccuracy,itperformsbetterthanbothapproaches.373.4SummaryInthischapter,wepresentedtheLocallyLinearManifold(LLM)modelasageneralapproachtotheprobleminLandsat7(ETM+)productsacquiredafterMay2003.Adetailedestimationprocessandmathematicalanalysishavebeenintroducedinthischapter.Forperfectinterpolationprocess,multipleSLC-onimageswereusedasimagesinthisapproach.Generally,eachbandismodeledasanon-linear,locallymanifoldthatcanbelearnedfromthematchingbandsatttimeinstances.Eachbandisdividedintosmalloverlappingpatches.Particularly,eachpatchisconsideredtobealinearcombinationofasetofcorrespondingpatchesfromothertimeinstances.FillpatchescanbeselectedfromLandsat7(ETM+)productspriortotheSLCfailureorfromLandsat5ThematicMapper(TM)productsthathavesimilarspatialandradiometricresolutionasLandsat7ETM+products.TheminimumEuclideannormregressionisusedtoestimatethetangenthyperplaneoverthetargetpatchandapoint,ontheestimatedhyperplane,thathasthesmallestEuclideandistancetotheobservedpartofthetargetpatchisfounded.Asmentionedpreviouslyforallalgorithms,aminimalsnowandcloudcoverarerequiredinallscenestoacquiresatisfyingresults.Therefore,pixelscorrespondingtocloudandsnowareexcludedfromanalysisinLLMapproachaswell.TheLLM-modeldrivenapproachoutperformsallaforementionedalgorithmsinChapter2duetothehighqualityreconstructedimages,thelesscomputingtimeandthesimplicitytorealizealgorithmically.38Chapter4EXPERIMENTALRESULTSANDPERFORMANCEEVALUATIONOFGAP-FILLINGALGORITHMSThischapterisdevotedtocomparinganddiscussingtheexperimentalresultsofthegap-algorithmsandLLMmodel.Allresultsareevaluatedquantitativelyandperceptually.Forthepurposeofcomparison,thisstudyisperformedforbothsimulatedandrealimages.AllalgorithmsareappliedonasceneofPath02andRow71obtainedfromtheUSGSwebsite.Subimagesofsize512512pixelfrombands5,4and2areextractedfromthefullsceneforsimulationanalysisasillustratedinFigure4.1.Bands5,4and2(Short-WaveInfrared(SWIR),NearInfrared(NIR)andgreen)areusedinthisstudyduetotherelationshipbetweentheselectedsceneandthesuitablespectralbandswereuse.OurselectedsceneisforalakeinSouthAmericawithsomevegetation,wetsoilandariveraroundit.Therefore,bands(5,4,2)arethebesttobeusedforthissceneaccordingtotheircharacteristicsindistinguishingmoisturecontentsofsoilandvegetation,discriminatingsnowandcloud-coveredterritories,emphasizingthebiomasscontentsandshorelines,furthermoreevaluatingtheplantvigorbymeasuringtheectancepeak[49].39Figure4.1:Aschematicdiagramillustratingtheextractionprocessofsubimagesfrombands(5,4,2)ofthewholescene.4.1ReconstructedResultsofSimulatedIm-agesTogetasimulatedtargetimage,sceneacquiredonMarch19,2006fromLandsat5TMisusedaftermultiplyingitwiththestriped(mask)imageextractedfromarealLandsat7imagesasshowninFigure4.2.Also,sceneobtainedonFebruary11,2002isselectedtorepresentthe(training)image.Allsceneswereacquiredtobeinthesameseasonandwithminimumsnowandcloudcovertoobtainbetterresults.40Figure4.2:SWIR-NIR-greencompositesofLandsat5/TMimages.(a)Ground-truthSLC-onimageacquiredonMarch19,2006.(b)targetimagesimulatedbasedon(a).(c)SLC-onimageacquiredonFebruary11,2002.Toassesstheresultsquantitatively,eachpixelvalueinthereconstructedimageiscom-paredtoitscorrespondingpixelvalueinthegroundtruthdataofthesimulatedLandsat5tar-getimage.ThecomparisonisperformedusingtheRootMeanSquareError(RMSE)[26,37]andPeakSignaltoNoiseRatio(PSNR)usingthefollowingequations:RMSE=pMSE(4.1)where,MSE=NPi=1(^LiLi)2N(4.2)PSNR=10log10Max2MSE(4.3)Here,MSEdenotestheMeanSquareError,^Lidenotesthepredictedithpixelvalueinthereconstructedimage,Lidenotestherealithpixelvalueinthegroundtruthdataofthetargetimage,Ndenotesthetotalnumberofmissing(gap)pixelsandMaxdenotesthemaximumpixelvalueoftheimage.AsshowninTable4.1,largerRMSEandsmallerPSNRindicateslargererrorprediction.Figure4.5illustratestherelationship41betweenPSNRandthewindowsize(forallthreebands)usingmethodsofsingle-sourceandmulti-sourcetechniques.Inaddition,resultsarecomparedperceptually(visually)inordertodetectwhethertherearegapsshownintheimagesandwhetherthesereconstructedimagesarespatiallycontinuous[19].AsillustratedinFigure4.3,(c)and(d)representthereconstructedimagesusingthesingle-sourcetechnique.TheresultingimagesshowthatallgapswerecompletelybutthereisayinpreservingthedetailedhighfrequencyinformationintheimageandalowperformanceinterritorieswithdetailedspatialpatternsasshowninFigure4.4.Thehighererrorobtainedusingthistechniqueisduetotheestimationprocessthatdependsonknownneighboringpixelsinthesameimageratherthanusingknownpixels(fromanauxiliarySLC-onimage)thatrepresenttheexactoflandcover.ThereconstructedimagesshowninFigure4.3,usingmulti-sourcetechnique(LLHMandAWLHM),havelessererrorsandprovideanaccuratepredictionofthemissingpixelsduetotheuseofanexternalSLC-onimageandsubsequentlydiscriminatespatialpatternsbetweentargetandSLC-onimage.ItcanbenoticedthatAWLHM,usingthesameimagesareusedforLLHM,providesslightlybetterresultsthanLLHM.Thisisduetotheuseofanadaptivewindow,tocalculategainandbias,inAWLHM,whileLLHMutilizesaxed-sizewindowthatleadstotheuseoftheuncorrelatedneighboringpixelsintheestimationprocess.TheweaknessofLLHMandAWLHMapproaches(usingthesameimagesinbothmeth-ods)wouldbeshowninsmalledges,sceneswithheterogeneouslandcoverandscenesfromtseasonsduetotheintargetradiancebetweentargetandimages.42Figure4.3:ResultsusingsimulatedLandsat5image.(a)Simulatedtargetimage,(b)SLC-onimage,(c-f)ReconstructedimagesusingAverage,IDW,LLHMandAWLHM(usingSLC-onimage)methodsrespectively.43Figure4.4:Errorimagesobtainedfromband5.(a)Errorimagebeforeapplyingalgorithms,(b)AfterapplyingAverage(c)AfterapplyingIDW,(d)AfterapplyingLLHM,(e)AfterapplyingAWLHMusingoneSLC-onimage.44Figure4.5:TherelationshipbetweenPSNRandthewindowsizeusingAverage,IDW,LLHMandAWLHM(usingSLC-onimage)methods.Onleft,forband2;onright,forband4;onbottom,forband5.4.1.1AWLHMUsingMultipleImagesAWLHMapproachwasdesignedtobeusedonlywithimages.WeusedAWLHMwithoneSLC-onimageintheprevioussectiontoclarifytheslightimprovementinresultsduetotheadaptive-sizewindow.Here,twosimulatedLandsat5imagesareusedasimagesandonesimulatedLandsat5isusedastargetimage.Sincethestripesofimagesdonotoverlap,thentwoinputimageswereenoughtothegapsinthetargetimage(morethantwoimagescouldbeuseduntilallgapsarebeingThesametargetimagementionedintheprevioussectionisusedhere,imagewasacquiredonApril13,2006andthesecondonewasacquiredonFebruary23,2002asshowninFigure4.6.Sinceboththetargetandimagescontainstripes,themaximumwindow45sizeisincreasedtobe1515ratherthan77inthepreviousalgorithms.Inaddition,theminimumnumberofcommonpixelsbetweenthetargetandimagesaredeterminedtobeequalto20inordertoobtainanaccuratepredictionofthemissingpixelvalues.Figure4.6:ResultsofAWLHMapproachusingmultiplesimulatedTMimages,(a)FirstimageacquiredonApril13,2005,(b)SecondimageacquiredonFebruary23,2002,(c)TargetimageacquiredonMarch19,2006,(d)ReconstructedimageusingAWLHMapproach.AWLHMprovideshigherqualityresultswithlessererrorsduetotheadaptivewindowsizeandmultipledatausedtothegaps.ErrorimageafterapplyingtheAWLHMalgorithmisshowninFigure4.7.Also,thequantitativeassessmentthatshowsanoticeabledecreasing46inRMSEandhighPSNRcouldbeillustratedinTable4.2.Figure4.7:Errorimagesobtainedfromband5usingAWLHMwithmultipleimages;fromlefttoright,errorimagesbeforeandafterusingAWLHMrespectively.ThelimitationofAWLHMcouldbenoticedwhenatvariationoccurinregionssmallerthanthelocalwindowsize[13].Moreover,thismethodneedsmorecomputingtimethanallalgorithmsmentionedinthisstudybecauseofusingmultipleinputimagesandconsequentlyincreasingthewindowsize.Figure4.8:ResultsusingLLMapproach;onleft,simulatedtargetimage;onright,reconstructedimageusingLLMapproach.47Figure4.9:Errorimagesobtainedfromband5usingLLMapproach;onleft,errorimagebeforeusingLLM;onright,errorimageafterusingLLMapproach.4.1.2LLMUsingMultipleSLC-onFillImagesTheLLM-modeldrivenapproachisproposedinthisstudytogapsinLandsat7products.ThismodelbasicallydependsonusingmultipleSLC-onimagestothegapsintargetimage.Fillimagesfrom1984to2011areobtainedfromLandsat5TM.Patchesofsizeequalto16areextractedfrominputSLC-onimagesataparticularposition.Sincethereisoverlapequalsto4,weneedtosumupalltheestimatedmissingpixelsanddividethembythetotalnumberofoverlaps.SametargetimageemployedintheaforementionedalgorithmsisalsousedinLLMalgorithm.AllgapsinthereconstructedimageusingLLMshowninFigure4.8areperfectly.Inaddition,Figure4.9showsthaterrorscanhardlybeobservedafterapplyingLLMapproach.Moreover,thequanti-tativeassessmentinTable4.2showsthatRMSEhasdecreasedandPSNRhasincreasedforallbands.RelationshipbetweenPSNRandwindowsizeforbothLLMandAWLHMapproachesisshowninFigure4.10.ThevisualandquantitativeresultshaveproventhesuperiorityofLLMoverallalgorithmsusedinthisstudy.48Figure4.10:TherelationshipbetweenPSNRandthewindowsizeusingLLMandAWLHM(usingmultipleinputimages)methods.Onleft,forband2;onright,forband4;onbottom,forband5.Thissuperiorityisduetopreservingalldetailsinthereconstructedimagebecauseofthelinearcombinationofpatchesinthehyperplanespace.Furthermore,inspiteofthehighcomputingtimeneededforthismethod,itneedsmuchlesstimethanAWLHMalgorithm.Inthismethod,sceneswithminimumcloudandsnowcoverarerequired.4.2ReconstructedResultsofRealImagesAllaforementionedalgorithmshavebeenappliedonactualtargetimageobtainedfromLandsat7/ETM+.TargetimagewasacquiredfromLandsat7onAugust24,2003andSLC-onimagewasacquiredfromLandsat5onApril23,2001.Thereconstructedimages,usingthesamesingle-sourceandmulti-sourcemethodsthat49MethodologyBand2Band4Band5RMSEPSNRRMSEPSNRRMSEPSNRAverage(mean)Filter0.019234.36040.023732.52910.038128.4462IDWFilter0.016435.72870.020633.70160.033029.6276LLHM0.012637.96920.016035.89650.022333.0177AWLHM0.011938.47570.015136.45050.021033.5889Table4.1:ThequantitativeassessmentofAverage,IDW,LLHM,AWLHM(usingoneSLC-onimage)methods.MethodologyBand2Band4Band5RMSEPSNRRMSEPSNRRMSEPSNRAWLHM(only)0.009240.76760.011738.66460.015736.0906Manifold(LLM)0.008341.67290.010139.87730.014236.9525Table4.2:ThequantitativeassessmentofLLMandAWLHM(usingmultipleimages)methods.mentionedinsection4.1,areshowninFigure4.11.ThesameETM+targetimageusedintheestimationprocessforsingle-sourcemethodsandoneTMSLC-onimagesusedformulti-sourcemethods.InAWLHMalgorithmusingmultipleimages,twoimagesfromLandsat7/ETM+wereacquiredonMay15,2004andAugust23,2005respectively.ResultsusingthismethodareshowninFigure4.12.TheLLMmethodemployedthesameLandsat5scenesacquiredon(1984-2011)asimagestothemissingvaluesinLandsat7targetimagementionedabove.Re-constructedimageusingLLMisshowninFigure4.13.AlmostthesameperformanceisobtainedusingrealimagesasthatobtainedusingsimulatedimagesduetospatialandradiometricresolutionsimilarityforbothLandsat7/ETM+andLandsat5/TMproducts.Furthermore,thesizeofstripesisalmost50thesameinoursimulatedandactualimages.51Figure4.11:ResultsusingrealETM+targetimage.(a)RealtargetimageacquiredonAugust24,2003;(b)TMSLC-onimageacquiredonApril23,2001;(c-f)Recon-structedimagesusingAverage,IDW,LLHMandAWLHM(usingSLC-onimage)methodsrespectively.52Figure4.12:ResultofAWLHMusingrealETM+Sinputimage.(a)and(b)RealETM+imagesacquiredonMay15,2004andAugust23,2005respectively;(c)ETM+targetimage;(d)ReconstructedimageusingAWLHM.Figure4.13:ResultofLLMusingrealETM+targetimage.(a)RealETM+targetimageacquiredonAugust24,2003;(b)ReconstructedimageusingLLM.53Chapter5CONCLUSIONSANDFUTUREWORK5.1ConclusionsInthisthesis,algorithmsfromsingle-sourcetechnique(AverageandIDW)andmulti-sourcetechnique(LLHMandAWLHM)wereselectedandexplainedtobecomparedtotheproposedLLMmodel.LLMapproachwasmodeledinwhich,multipleLandsat5SLC-onimages(1984-2011)wereutilizedasimagesandeachbandwasmodeledasanon-linear,locallymanifoldthatcanbelearnedfromthematchingbandsatdittimeinstances.Eachbandwasdividedintosmalloverlappingpatchesconsideredtobealinearcombinationofasetofcorrespondingpatchesfromothertimeinstances.Theselectedandproposedalgorithmswereappliedonthesimulatedandrealimages.Landsat5/TMsceneswereusedtothegapsinLandsat7/ETM+images.ResultsobtainedinthisstudyhaveproventhesuperiorityoftheLLM-modeldrivenapproachoverallotheralgorithms.Simulationresultsshowedthatsingle-sourcemethodsmaynotbesuitableforgapsinLandsat7imagesbecauseofutilizingpixelsrepresentinaccurateestimationoflandscape.However,multi-sourcemethodsprovidedmoreaccurateresultsbecauseofusingauxiliaryLandsatSLC-onorimages.Therefore,multi-sourcemethodsweremoreappropriateforgapsinETM+images.LLHM54approachprovidedgoodreconstructedimageswithlessercomputingtime.WhileAWLHMoutperformedLLHMbyprovidingmoreaccurateresults,itslowcomputingtime.TheproposedLLMapproachwasthebestmodelforgapsintermsofprovidinghigherqualityreconstructedimages,almosterror-free,andconcurrentlyneededlesscomputingtimethanAWLHM.ThesuperiorityofLLMwasduetopreservingalldetailsinthereconstructedimagebecauseofthelinearcombinationofpatchesinthehyperplanespace.Forthismethodandallotherselectedmethodsinthisthesis,sceneswithminimumcloudandsnowcoverwererequired.5.2FutureWorkInlandcoveranalysis,cloudcontaminationisconsideredanomaliesthatchangeovertimeandspace.Pixelsthataremaskedbycloudslosetheirscienreliabilityforlandcoveranalysessuchastheassessmentofvegetationindices.Itisnottoassesswhetherapixeliscoveredbyclouds,snoworwater.Therefore,contaminatedpixelscanbeanalyzedseparatelyfromlandcover.Meanwhile,cloudcoveredpixelsconstitutesmoothtexturesthatcanbeinterpolatedspatiallywithinasinglescene.Clearly,interpolatingcloudcoversmerelyservesvisualpurposessincecloudmaskedpixelsaretypicallyexcludedfromanyscienanalysis.AsfortheproposedLLMapproach,cloud-coverscenesareexcludedfromanalysis.Thisprocesswasperformedmanuallyinwhicheachsceneacquiredfrom(1984-2011)wasobservedtodeterminewhetheritisacloud-coversceneornot.Forfuturework,analgorithmthattestsscenestodeterminewhethertheyhaveregionscoveredwithcloudscouldbedeveloped.Sincepixelscorrespondingtocloudsarecalledcoldpixels,thisalgorithmcouldgenerallybebasedonutilizingthermalbandofeachimageinwhichpixels55ofanauxiliaryclearimageforthesamescenewouldbecomparedtocorrespondingpixelsofthecontaminatedimage.Pixelswithlowertemperaturegiveanindicationofcloudpresenceintestedimages[50].56BIBLIOGRAPHY57BIBLIOGRAPHY[1]USGS.(2016)FrequentlyAskedQuestionsabouttheLandsatMissions.[Online].Available:http://landsat.usgs.gov/banddesignationslandsatsatellites.php[2]M.RezaandS.Ali,\UsingIRSproductstorecover7ETM+defectiveimages,"AmericanJournalofAppliedSciences,vol.5,no.6,pp.618{625,June2008.[Online].Available:http://dx.doi.org/10.3844/ajassp.2008.618.625[3]USGS.(2016)Products:Background.[Online].Available:http://landsat.usgs.gov/productskground.php[4]P.Scaramuzza,E.Micijevic,andG.Chander,\SLCProductsPhaseOneMethodology,"LandsatTechnicalNotes,March2004.[Online].Available:http://landsat.usgs.gov/documents/SLCGapFillMethodology.pdf[5]F.Chen,H.Ye,andX.Zhao,\MakingUseoftheLandsat7ETM+ImageThroughtRecoveringApproaches,"inDataAcquisitionApplications,Z.Karakehayov,Ed.INTECHOpenAccessPublisherRijeka,Croatia,August2012,ch.13,pp.317{342.[Online].Available:http://dx.doi.org/10.5772/48535[6]P.Scaramuzza,E.Micijevic,andG.Chander,Gap-FilledProductsGap-FillAlgorithmMethodologyPhase2Gap-FillAlgorithm,"USGeologicalSurveyEarthResourcesObservationandScience(EROS)Center,july2004.[Online].Available:hFilled%20Products%20Gap-Fill%20algorithm%20methodology.pdf[7]M.Moraniec,\Landsat:AnEarth-ObservingTrailblazer,"ArticlesEarthObserva-tionEarthzine,IEEE,2011.[Online].Available:http://earthzine.org/2011/12/28/landsat-an-earth-observing-trailblazer/[8]D.L.Williams,S.Goward,andT.Arvidson,\Landsat:Yesterday,Today,andTomorrow,"PhotogrammetricEngineering&RemoteSensing,vol.72,no.10,pp.1171{1178,2006.[Online].Available:http://dx.doi.org/10.14358/PERS.72.10.1171[9]USGS.(2016)Landsat8.[Online].Available:http://landsat.usgs.gov/landsat8.php58[10]F.Chen,L.Tang,C.Wang,andQ.Qiu,\RecoveringofthethermalbandofLandsat7ETM+imageusingCBERSasauxiliarydata,"AdvancesinSpaceResearch,vol.48,no.6,p.10861093,September2011.[Online].Available:http://dx.doi.org/10.1016/j.asr.2011.05.012[11]C.Zeng,H.Shen,andL.Zhang,\RecoveringmissingpixelsforLandsatETM+imageryusingmulti-temporalregressionanalysisandaregularizationmethod,"RemoteSensingofEnvironment,vol.131,pp.182{194,April2013.[Online].Available:http://dx.doi.org/10.1016/j.rse.2012.12.012[12]S.Maxwell,\FillingLandsatETM+GapsUsingaSegmentationModelAp-proach,"PhotogrammetricEngineeringandRemoteSensing,vol.70,no.10,pp.1109{1112,October2004.[13]A.S.ABDULJABAR,G.SULONG,andL.E.GEORGE,\SURVEYONGAPFILLINGALGORITHMSINLANDSAT7ETM+IMAGES,"JournalofTheoretical&AppliedInformationTechnology,vol.63,no.1,pp.136{146,2014.[Online].Available:http://www.jatit.org/volumes/Vol63No1/16Vol63No1.pdf[14]S.M.AliandM.J.Mohammed,\Gap-FillingRestorationMethodsforETM+SensorImages,"IraqiJournalofScience,vol.54,no.1,pp.206{214,2013.[15]R.OlivierandC.Hanqiang,\NearestNeighborValueInterpolation,"InternationalJournalofAdvancedComputerScienceandApplications(IJACSA),vol.3,no.4,pp.25{30,2012.[Online].Available:http://dx.doi.org/10.14569/IJACSA.2012.030405[16]D.Lancaster,\AReviewofSomeImagePixelInterpolationAlgorithms,"Retrieved,vol.1,no.7,p.2013,2007.[Online].Available:http://carolinepetitjean.free.fr/enseignements/ti/pixintpl.pdf[17]W.Hu,M.Li,Y.Liu,Q.Huang,andK.Mao,\Anewmethodofrestoringetm+imagesbasedonmulti-temporalimages,"inGeoinformatics,201119thInternationalConferenceon.Shanghai,China:IEEE,June2011,pp.1{4.[Online].Available:fhttp://dx.doi.org/10.1109/GeoInformatics.2011.5981182g,ISSN=f2161-024Xg,[18]S.Maxwell,G.Schmidt,andJ.Storey,\Amulti-scalesegmentationapproachtogapsinLandsatETM+SLCimages,"InternationalJournalofRemoteSensing,vol.28,no.23,pp.5339{5356,December2007.[19]J.Chen,X.Zhu,J.E.Vogelmann,F.Gao,andS.Jin,\AsimpleandemethodforllinggapsinLandsatETM+images,"RemoteSensingof59Environment,vol.115,no.4,pp.1053{1064,April2011.[Online].Available:http://dx.doi.org/10.1016/j.rse.2010.12.010[20]J.Storey,P.Scaramuzza,G.Schmidt,andJ.Barsi,\LANDSAT7SCANLINECORRECTOR-OFFGAP-FILLEDPRODUCTDEVELOPMENT,"inPecora16,GlobalPrioritiesinLandRemoteSensing,SiouxFalls,SouthDakota,October2005,pp.1{13.[21]X.Zhu,D.Liu,andJ.Chen,\AnewgeostatisticalapproachforgapsinLandsatETM+images,"RemoteSensingofEnvironment,vol.124,pp.49{60,2012.[22]Q.He,B.Shan,H.Ma,Y.Chen,andX.Wang,\ResearchonAlgorithmsforRecoveringLandsat-7GapData,"inControl,AutomationandSystemsEngineering(CASE),2011InternationalConferenceon.Singapore:IEEE,July2011,pp.1{4.[Online].Available:http://dx.doi.org/10.1109/ICCASE.2011.5997656[23]M.DesaiandA.Ganatra,\SurveyonGapFillinginSatelliteImagesandInpaintingAlgorithm,"InternationalJournalofComputerTheoryandEngineering,vol.4,no.3,pp.341{345,June2012.[Online].Available:http://dx.doi.org/10.7763/IJCTE.2012.V4.479[24]V.Rulloni,O.Bustos,andA.G.Flesia,\Largegapsimputationinremotesensedimageryoftheenvironment,"ComputationalStatistics&DataAnalysis,vol.56,no.8,pp.2388{2403,August2012.[Online].Available:http://dx.doi.org/10.1016/j.csda.2012.02.022[25]W.Hu,M.Li,Y.Liu,Q.Huang,andK.Mao,\ANewMethodofRestoringETM+ImagesBasedonMulti-temporalImages,"inGeoinformatics,201119thInternationalConferenceon.Shanghai,China:IEEE,June2011,pp.1{4.[Online].Available:http://dx.doi.org/10.1109/GeoInformatics.2011.5981182[26]G.Sulong,A.Sadiq,andL.Edwar,\SingleandMulti-sourceMethodsforReconstructiontheGapsinLandsat7ETM+Images,"ResearchJournalofAppliedSciences,EngineeringandTechnology,vol.11,no.4,pp.423{428,2015.[Online].Available:http://dx.doi.org/10.1016/j.rse.2012.04.019[27]C.Zhang,W.Li,andD.Travis,ofLandsatETM+satelliteimageusingageostatisticalapproach,"InternationalJournalofRemoteSensing,vol.28,no.22,pp.5103{5122,November2007.[Online].Available:http://dx.doi.org/10.1080/0143116070125041660[28]M.Pringle,M.Schmidt,andJ.Muir,\GeostatisticalinterpolationofSLC-LandsatETM+images,"pp.654{664,November2009.[Online].Available:http://dx.doi.org/10.1016/j.isprsjprs.2009.06.001[29]A.D.Boloorani,S.Erasmi,andM.Kappas,\Multi-SourceRemotelySensedDataCombination:ProjectionTransformationGap-FillProcedure,"Sensors,vol.8,no.7,pp.4429{4440,July2008.[Online].Available:http://dx.doi.org/10.3390/s8074429[30]X.Zhu,F.Gao,D.Liu,andJ.Chen,\AMoNeighborhoodSimilarPixelInterpolatorApproachforRemovingThickCloudsinLandsatImages,"GeoscienceandRemoteSensingLetters,IEEE,vol.9,no.3,pp.521{525,May2012.[Online].Available:http://dx.doi.org/10.1109/LGRS.2011.2173290[31]M.Mohammdy,H.Moradi,H.Zeinivand,A.Temme,H.Pourghasemi,andH.Alizadeh,\ValidatingofLandsatETM+satelliteimagesintheGolestanProvince,Iran,"ArabianJournalofGeosciences,vol.7,no.9,pp.3633{3638,2014.[Online].Available:http://dx.doi.org/10.1007/s12517-013-0967-5[32]M.AghamohamadniaandA.Abedini,\Amorphology-stitchingmethodtoimproveLandsatimageswithstripes,"GeodesyandGeodynamics,vol.5,no.1,pp.27{33,February2014.[Online].Available:http://dx.doi.org/10.3724/SP.J.1246.2014.01027[33]T.K.Alexandridis,I.Cherif,C.Kalogeropoulos,S.Monachou,K.Eskridge,andN.Silleos,\RapiderrorassessmentforquantitativeestimationsfromLandsat7images,"RemoteSensingLetters,vol.4,no.9,pp.920{928,June2013.[Online].Available:http://dx.doi.org/10.1080/2150704X.2013.815380[34]D.Yang,H.Su,Y.Yong,andJ.Zhan,\MODIS-LandsatDataFusionforEstimatingVegetationDynamics{ACaseStudyforTwoRanchesinSouthwesternTexas,"inInProceedingsofthe1stInternationalElectronicConferenceonRemoteSensing,June2015,pp.1{14.[Online].Available:http://dx.doi.org/10.3390/ecrs-1-d016[35]D.P.Roy,J.Ju,P.Lewis,C.Schaaf,F.Gao,M.Hansen,andE.Lindquist,\Multi-temporalMODIS-Landsatdatafusionforrelativeradiometricnormalization,gapandpredictionofLandsatdata,"RemoteSensingofEnvironment,vol.112,no.6,pp.3112{3130,2008.[Online].Available:http://dx.doi.org/10.1016/j.rse.2008.03.009[36]A.D.Boloorani,S.Erasmi,andM.Kappas,\Multi-sourceimagereconstruction:exploitationofEO-1/ALIinLandsat-7/ETM+gapinElectronicImaging2008.InternationalSocietyforOpticsandPhotonics,2008,pp.681219{681219.[Online].Available:http://dx.doi.org/10.1117/12.76686661[37]F.Chen,L.Tang,andQ.Qiu,\ExploitationofCBERS-02BasAuxiliaryDatainRecoveringtheLandsat7ETM+Image,"inGeoinformatics,201018thInternationalConferenceon.Beijing,China:IEEE,June2010,pp.1{6.[Online].Available:http://dx.doi.org/10.1109/GEOINFORMATICS.2010.5567696[38]R.GonzalezandR.Woods,DigitalImageProcessing,2nded.UpperSaddleRiver,NewJersey07458:PrenticeHall,January2002.[39]P.Manuel,ofDEMinterpolationmethodsinDrainageAnalysis,"LectureNotesofCE394KGISinWaterResources,2009.[40]O.OhashiandL.Torgo,\SpatialInterpolationUsingMultipleRegression,"inDataMining(ICDM),2012IEEE12thInternationalConferenceon.Brussels:IEEE,2012,pp.1044{1049.[41]D.NusretandS.Dug,\ApplyingtheinversedistanceweightingandkrigingmethodsofthespatialinterpolationonthemappingtheannualprecipitationinBosniaandHerzegovina,"Ph.D.dissertation,InternationalEnvironmentalModellingandSoftwareSociety(iEMSs),2012.[42]C.Childs,\InterpolatingSurfaceinArcGISSpatialAnalyst,"ArcUser,July-September,pp.32{35,2004.[Online].Available:hes/interpolating.pdf[43]M.A.AzpuruaandK.D.Ramos,\ACOMPARISONOFSPATIALINTERPO-LATIONMETHODSFORESTIMATIONOFAVERAGEELECTROMAGNETICFIELDMAGNITUDE,"ProgressInElectromagneticsResearchM,vol.14,pp.135{145,2010.[44]M.JingandJ.Wu,\Fastimageinterpolationusingdirectionalinversedistanceweight-ingforreal-timeapplications,"OpticsCommunications,vol.286,pp.111{116,January2013.[45]G.Y.LuandD.W.Wong,\Anadaptiveinverse-distanceweightingspatialinterpolationtechnique,"Computers&Geosciences,vol.34,no.9,pp.1044{1055,September2008.[46]C.Dang,M.Aghagolzadeh,andH.Radha,\ImageSuper-ResolutionviaLocalSelf-LearningManifoldApproximation,"SignalProcessingLetters,IEEE,vol.21,no.10,pp.1245{1249,October2014.[Online].Available:http://dx.doi.org/10.1109/LSP.2014.233211862[47]C.T.Dang,M.Aghagolzadeh,A.A.Moghadam,andH.Radha,\SingleImageSuperResolutionviaManifoldLinearApproximationusingSparseSubspaceClustering,"inGlobalConferenceonSignalandInformationProcessing(GlobalSIP),2013IEEE.Austin,TX:IEEE,December2013,pp.949{953.[Online].Available:http://dx.doi.org/10.1109/GlobalSIP.2013.6737049[48]C.Dang,A.Safaie,M.Phanikumar,andH.Radha,\WindSpeedandDirectionEstimationUsingManifoldApproximation,"inProceedingsofthe14thInternationalConferenceonInformationProcessinginSensorNetworks.Seattle,WA,USA:ACM,April2015,pp.328{329.[Online].Available:http://dx.doi.org/10.1145/2737095.2742998[49]N.R.Canada,\Tutorial:Fundamentalsofremotesensing,"2016.[On-line].Available:hes/earthsciences/pdf/resource/tutor/fundam/pdf/fundamentalse.pdf[50]J.-F.CayulaandP.Cornillon,\CloudDetectionfromaSequenceofSSTImages,"RemoteSensingofEnvironment,vol.55,no.1,pp.80{88,January1996.[Online].Available:http://dx.doi.org/10.1016/0034-4257(95)00199-963