DEVELOPMENTOFPLENOPTICINFRAREDCAMERAUSINGLOWDIMENSIONALMATERIALBASEDPHOTODETECTORSByLiangliangChenADISSERTATIONSubmittedtoMichiganStateUniversityinpartialoftherequirementsforthedegreeofElectricalEngineering-DoctorofPhilosophy2016ABSTRACTDEVELOPMENTOFPLENOPTICINFRAREDCAMERAUSINGLOWDIMENSIONALMATERIALBASEDPHOTODETECTORSByLiangliangChenInfrared(IR)sensorhasextendedimagingfromsubmicronvisiblespectrumtotensofmicronswavelength,whichhasbeenwidelyusedformilitaryandcivilianapplication.TheconventionalbulksemiconductormaterialsbasedIRcamerassufferfromlowframerate,lowresolution,tem-peraturedependentandhighlycost,whiletheunusualCarbonNanotube(CNT),lowdimensionalmaterialbasednanotechnologyhasbeenmademuchprogressinresearchandindustry.TheuniquepropertiesofCNTleadtoinvestigateCNTbasedIRphotodetectorsandimagingsystem,resolvingthesensitivity,speedandcoolingdifinstateoftheartIRimagings.Thereliabilityandstabilityiscriticaltothetransitionfromnanosciencetonanoengineeringespeciallyforinfraredsensing.ItisnotonlyforthefundamentalunderstandingofCNTphotore-sponseinducedprocesses,butalsoforthedevelopmentofanovelinfraredsensitivematerialwithuniqueopticalandelectricalfeatures.Intheproposedresearch,thesandwich-structuredsensorwasfabricatedwithintwopolymerlayers.Thesubstratepolyimideprovidedsensorwithisolationtobackgroundnoise,andtopparylenepackingblockedhumidenvironmentalfactors.Atthesametime,thefabricationprocesswasoptimizedbyrealtimeelectricaldetectiondielectrophoresisandmultipleannealingtoimprovefabricationyieldandsensorperformance.Thenanoscaleinfraredphotodetectorwascharacterizedbydigitalmicroscopyandpreciselinearstageinorderforfullyunderstandingit.Besides,thelownoise,highgainreadoutsystemwasdesignedtogetherwithCNTphotodetectortomakethenanosensorIRcameraavailable.Toexploremoreofinfraredlight,weemploycompressivesensingalgorithmintolightsampling,3-Dcameraandcompressivevideosensing.Theredundantofwholelightinclud-ingangularimagesforlightbinocularimagesfor3-Dcameraandtemporalinformationofvideostreams,areextractedandexpressedincompressiveapproach.Thefollowingcomputationalalgorithmsareappliedtoreconstructimagesbeyond2Dstaticinformation.Thesuperresolutionsignalprocessingwasthenusedtoenhanceandimprovetheimagespatialresolution.Thewholecamerasystembringsadeeplydetailedcontentforinfraredspectrumsensing.ACKNOWLEDGMENTSIfeeltremendouslyluckytohavehadtheopportunitytoworkwithDr.NingXi,Dr.LixinDongontheideasinthisdissertation,andIwouldliketothankthemfortheirguidanceandsupport.Dr.Xiinstilledinmealovefordesigningnanosensorbasedinfraredcamera,agreedtotakemeonasagraduatestudent,andencouragedmetoimmersemyselfinsomethingIhadapassionfor.Dr.Donginspiredmemuchonnanophotonicsenhancementandnanomanipulation.Ihavenevermetaprofessormoregenerousandfriendlywithhistimeandexperience.IamgratefultoDr.TimothyGrotjohn,Dr.FathiM.Salem,Dr.DonnieReinhardandDr.ZhengfangZhouforservingonmythesiscommittee.Theyofferedmetimelyhelpandunfailingsupportthatimprovethetechnicalsoundnessandthepresentationofthisdissertation.Iwouldliketoacknowledgetheworkoftheotherindividualswhohavecontributedtothiscameraresearch.Dr.BaokangBiandDr.RezaLoloeeinDepartmentofphysicsandastronomy,helpedmeonnanosensorfabricationandassemblydevice.Dr.Loloeegenerouslydonatedhistimeandexpertisetohelpverifyfabricationprocess.Dr.MingYaninDepartmentofmathematicscon-tributedthemosttoexplaininghowthenonconvexworks,andmanyoftheoptimizationprobleminthesepagesareduetohisartistry.Dr.WeihongGuoincasereserveuniversity,supportedmeonsingleimagesuperresolution.Mr.DavidSmithinwintechdigital,gavemuchguidanceondigitalmicromirrorsetupandapplications.Inaddition,IwouldliketothankDr.KingWaiChiuLai,Dr.CarmenKarManFung,Dr.HongzhiChen,Dr.RuiguoYang,Mr.BoSong,Dr.ZhanxinZhou,Dr.YongliangYang,Mr.ZhiyongSun,Dr.ChiZhang,Dr.ErickNieves,Dr.JianguoZhao,Dr.YunyiJia,Mr.YuCheng,Mr.MustaffaAlfatlawi,Mr.EmadAlsaedi,Mr.LaiWeiandMr.XiaoZengfortheirsupportintheexperimentsanddiscussionattheMSURoboticsandAutomationLab.ThanksalsotoDr.ivTongJia,Dr.JianyongLouforhelpfuldiscussionsrelatedtothiswork.Finally,Iwouldliketothankmyfamily,mywife,QiaozhiSun,fortheirloveandsupport.Thisdissertationwouldnothavebeenpossiblewithouttheiryearsofencouragementandcontin-uoussupport.Mywifehavemadecountlessforme,andhaveprovidedmewithsteadyguidanceandencouragement.Thisdissertationisdedicatedtothem.vTomyparents,mywifeQiaozhifortheirloveandsupportviTABLEOFCONTENTSLISTOFTABLES.......................................xLISTOFFIGURES......................................xiChapter1Introduction..................................11.1InfraredEverywhere.................................11.1.1FundamentalofInfrared...........................11.1.2ConventionalInfraredSensor.........................41.1.3InfraredDetectorMarket...........................51.2CarbonNanotubeBasedInfraredSensor.......................71.2.1SensorsCharacterization...........................71.2.2NanoMaterialIRPhotodetector.......................111.3ComputationalImaging................................121.4HighDimensionalPlenopticFunctionofLight....................151.5DissertationOverview.................................161.6OrganizationoftheStudy...............................17Chapter2LowDimensionalMaterialBasedInfraredPhotodetector.........192.1PreviousWork.....................................192.2CNTIRSensorDesign................................212.3RealtimeDEPFabrication..............................222.3.1IntroductionofDEP.............................242.3.2AssemblyMethodandSystem........................262.3.3QuantitativelyCNTDepositionandDeviceFabrication...........282.4SensorReliabilityandResponseEnhancement....................292.4.1SubstrateEffectandPackagingonNanoSensor...............302.4.2ExtrinsicSurfaceStateEffect.........................332.4.3SensorResponseEnhancement........................352.5NanoscaleIRSensorCharacterization........................372.5.1SensorsandMeasurementMethod......................372.5.2ExperimentalResults.............................402.6ChapterSummary...................................45Chapter3SinglePixelInfraredCamera........................473.1PreviousWork.....................................473.2SpatialLightModulatorbasedImager........................503.2.1CompressiveSensing.............................503.2.2SinglePixelImager..............................513.3WeakSignalReadoutMethod.............................52vii3.3.1CurrenttoVoltageConversionMethod....................523.3.2ResistorbasedCurrentReadoutMethod...................543.3.3CapacitorBasedCurrentReadoutMethod..................543.4ROICStructureforCNTIRSensor..........................553.4.1ZerobiasReadoutCircuits..........................553.4.2HighGainCurrenttoCurrentConverter...................563.4.3HighSpeedReadout.............................563.5HardwareExperimentalPerformanceandApplications...............573.5.1ReadoutSystemTesting...........................573.5.2ReadoutApplications.............................583.6ChapterSummary...................................59Chapter4LightFieldImaging..............................624.1PreviousWork.....................................624.24DLightFieldModel.................................654.2.1LightFieldModelinLens..........................654.2.2LightFieldModelinMirror.........................694.3MaskbasedSinglePixelLightFieldSensing.....................714.3.1OpticsandSystemDesign..........................714.3.2ExperimentalPerformance..........................734.4DoubleCompressiveLightFieldSensing.......................734.4.1ModelingofDoubleCompressiveLightFieldSensing...........744.4.2RecoveryAlgorithm.............................744.4.3ExperimentswithDoubleCompressiveSensing...............774.5Chaptersummary...................................80Chapter53-DImaging..................................825.1PreviousWork.....................................825.2TimeofFlight3DImaging..............................845.2.1WorkingPrinciple...............................845.2.2TimeofFlightModelingandApplication..................855.3StereoVision3DImaging...............................885.3.1WorkingPrinciple...............................885.3.2StereoVisionModelingandApplication...................885.4Compressive3DImaging...............................915.4.1Sparsityin3-DImage.............................915.4.2Compressive3DSampling..........................935.4.3ExperimentswithPrototypeCamera.....................955.5Chaptersummary...................................95Chapter6SuperResolutionImaging..........................976.1PreviousWork.....................................976.2ObservationModeling.................................1016.3MultipleImagesbasedSuperResolution.......................1036.3.1NonuniformInterpolationApproach.....................103viii6.3.2FrequencyDomainApproach.........................1046.3.3RegularizedSRReconstructionApproach..................1056.3.4MultipleFramesSamplinginSinglePixelCamera.............1066.3.5ExperimentswithPrototypeCamera.....................1096.4SingleImagesbasedSuperResolution........................1106.4.1ObservationModeling............................1126.4.2SplinebasedReproducingKernelHilbertSpaceandApproximativeHeav-isideFunctionModel.............................1136.4.3IterativeReconstructionAlgorithm......................1156.4.4SimulationsandExperiments.........................1166.5Chaptersummary...................................120Chapter7CompressiveVideoSensing..........................1227.1PreviousWork.....................................1227.2SparsityofVideo...................................1257.2.1IntraframeCompression...........................1267.2.2InterframeCompression...........................1277.2.3VideoCompression..............................1317.3CompressiveVideoSensing..............................1347.3.1Introduction..................................1347.3.2CombinedSparsitySamplingforVideo...................1357.3.3Non-convexProblemSolver.........................1357.3.4Non-convexSorted`1Method........................1387.3.5NumericalAnalysis..............................1427.3.6ExperimentalImplementationandResults..................1467.4Chaptersummary...................................148Chapter8ConclusionsandFutureWork........................1498.1Conclusions......................................1498.2FutureResearch....................................152BIBLIOGRAPHY.................................154ixLISTOFTABLESTable1.1Infraredsub-divisionscheme.........................4Table2.1Au-CNT-Austructureanditsphotoresponse................38Table2.2Eucentricveaxistable..................39Table2.3CNTmetalcontactlengthandthedirectionofoutputphotocurrent.....44Table4.1CharacterizingangularimagesaccumulationresidualerrorbyPSNRandRMSE.....................................79Table6.1RKHSbasedsingleimagesuperresolutionalgorithm............116Table6.2RMSEvalueofmedicalIRimageandindoorIRimage...........120Table7.1Iterativelyreweighted`1minimizationwiththresholding..........142Table7.2CharacterizingvideoframesaccumulationresidualerrorbyPSNRandRMSE.....................................146xLISTOFFIGURESFigure1.1Planck'slaw(coloredcurves)andclassicaltheory(blackcurve).Forin-terpretationofthereferencestocolorinthisandallotherthereaderisreferredtotheelectronicversionofthisthesis...........2Figure1.2Conventionaldigitalphotography......................14Figure1.3Computationalphotography.........................14Figure1.4Optics............................15Figure1.5P(x;y)andP(x;y;l).............................17Figure1.6Dissertationoverview.............................18Figure2.1BanddiagramofCNTmetalcontact.....................20Figure2.2BandbendingofaSchottkybarrierinCNT-FET,fortwogatevoltages...21Figure2.3CNTmetalcontactwiththreedistinctpositions.a)CNTonthetopmetal;b)CNTundermetal;c)CNTbetweenmetal.................22Figure2.4Illustrationofthedielectrophoreticmanipulation..............26Figure2.5DEPforceonCNTinanon-uniformelectrical(sideview)......27Figure2.6Real-timemonitoringDEPsystem.RedrowshowscurrentloopwhenCNTisbridgedbetweengap(yellow).ThesystemwillshutdownACsourcethroughfeedbackwhenimpedancechanges.............27Figure2.7SEMimageofmultiwallcarbonnanotubes.ThereisonlyoneCNT(MC1)ontopdevice,twoCNTs(MC2andMC3)onmiddledeviceandthreeCNTsonbottomdevice(MC4,MC5andMC6)............29Figure2.8SEMimageofsinglewallcarbonnanotube.TopissingleCNT(SC1)bridged.ThebottomdeviceshowssinglewallCNTusingrealtimeDEPdeposition................................30Figure2.9DarkcurrentmeasurementresultsonCNTIRdetector...........32Figure2.10LinearitymeasurementresultsonCNTIRdetector.............32xiFigure2.11ParasiticcapacitancemodelofCNTmetalSchottkybarrier.........34Figure2.12Surfacechargestorageonsubstrate.....................35Figure2.13CNT-basedIRsensorresponseenhancementbyhelicalantenna......36Figure2.14I-VcurveofCNTIRSensor.a)deviceA;b)deviceB...........36Figure2.15Top:SEMimageofAu-CNT-Austructure.Bottom:TherelativesizebetweenCNTdetectorandIRlaserbeamspot................38Figure2.16a)Proposedtestingbenchusingdigitalmicroscope,laserandveaxissubstage.b)Hardwaresetup,insetissubstage.c)Fourpointscalibra-tionmarkerfordetector.d)Rasterscanning:experimentalmeasurementpathwayforcentroidofphotodetector....................40Figure2.17Focusedandunfocusedlightraysondigitalmicroscope..........41Figure2.18Photocurrentmeasurementalongxdirectionwithdistincty.........43Figure2.19Photocurrentmeasurementalongydirectionwithdistinctx.........44Figure2.20Photoresponseanddarkcurrentondifferentbiasvoltage..........45Figure2.21PhotocurrentcomparisononAu-CNT,Cu-CNTandAg-CNT........46Figure3.1SystemsetupofcompressivesensingbasedimagingsystemusingaCNTphotodetector,responseenhancedbyphotoniccavity............53Figure3.2SchematicofRtypeIVconverter......................54Figure3.3SchematicofCtypeIVconverter......................55Figure3.4Zerobiasreadoutcircuit...........................56Figure3.5CurrenttocurrentconverterforCNTIRsensor...............57Figure3.6Diagramofreadoutsystem..........................58Figure3.7ReadoutlinearitytestonCNTbasedIRdetector...............58Figure3.8Readoutcomparisonbetweenproposedsystemandsemiconductorpa-rameteranalyzer(Agilent4155c)......................59Figure3.9HardwaresetupofsinglepixelIRimagingsystem.............60xiiFigure3.10ImagesrecoverybasedonsingleCNTdetector...............61Figure4.1Concaveobjectradiance(left)andconvexobjectradiance(right)......63Figure4.2Parameterizinglightrayin3Dspacebyposition(x,y,z)anddirection(q,f).63Figure4.3Twoplaneparameterizationforlight.................67Figure4.4TwoplaneparameterizationinSLRcamera.................67Figure4.5TwoplaneparameterizationinCartesiancoordinates............68Figure4.6Lightraydiagramofcamera(unfocused)..................69Figure4.7RaysinCartesiancoordinates(unfocused)..................70Figure4.8TwoplaneparameterizationinDMDbasedimagingsystem........71Figure4.9Schematicdiagramofsinglepixellightsensing............72Figure4.10Distinctangularimagefromtwoaperture..................72Figure4.11Syntheticapertureimaging.Thefocusplaneisbecomingfarawaytomainlensfromlefttoright..........................73Figure4.1255angularimagesofStanfordjellybeans.................75Figure4.13Adjacentangularimagedifference.a)intensitydifference,b)signichanges(nonzerochanges)ofangularimage................76Figure4.14Angularimagerecoverycomparisonbetweenbasiccompressivesensinganddoublecompressivesensing.......................77Figure4.15AngularimagerecoveryresidualerrorbyRMSEandPSNR........78Figure4.16Angularimagerecoveryfromdoublecompressivesensing,thecol-umnisreferenceimageandtheotherfourarerestoreddependsonitsleft......................................81Figure5.13Dcameraoperationprinciple................85Figure5.2Twomethods:pulsed(left)andcontinuouswave(right)...87Figure5.3Retinaldisparityfromeyes..........................89xiiiFigure5.4Stereopsisdepththroughdisparitymeasurement(left)andstereovisionsystem(right).............................89Figure5.5SparsityinDMDbased3Dsampling....................92Figure5.6Dualdetectors3Dimagingsystem......................94Figure5.7Maskbased3Dimagingsystem.......................94Figure5.83Dimagereconstructioninred/cyancolor..................96Figure6.1Multipleimagessuperresolution.......................98Figure6.2FourcausesofLRimageacquisition.....................98Figure6.3Schematicdiagramofsinglepixelcamera..................107Figure6.4TwoaperturesdesigninDMDimagingsystem...............108Figure6.5Multipleaperturesdesignforhighresolutionimaging............108Figure6.6MeasurementcoveredinneighborhoodofX0point.............109Figure6.7Prototypehardwareofsuperresolutionsinglepixelcamera.........111Figure6.8Experimentalresults,fromtop:4,9,16................111Figure6.9Classicalmultipleimagesandsingleimagesuperresolution........113Figure6.10NearIRimageofbuilding.a)bicubicmethod;b)nearestneighbor;c)proposedmethod;d)groundtruthimage...................117Figure6.11Handandheadinfraredimagefromsuperlattice(SLS)cooledFPA.a)bicubicmethod;b)nearestneighbor;c)proposedmethod;d)groundtruthimage.....................................118Figure6.12Handprintthermalimagefromcooledthermalcamera.a)bicubicmethod;b)nearestneighbor;c)proposedmethod;d)groundtruthimage......118Figure6.13Theuncooledthermalimagesuperresolutioncomparison.........119Figure6.14HandinfraredimagefromuncooledthermalIRcamera,a)bicubicmethod;b)proposedmethod;c)groundtruthimage.................119Figure7.1ConventionalNyquistShannonsignalsamplingandcompressivesampling.123xivFigure7.2LightilluminationinSLRcamera......................124Figure7.3Sparsityinvideosignal............................125Figure7.4Sub-samplingofimagecompression.....................126Figure7.5DCTbasedtransformcodingimagecompression..............127Figure7.6Differencebetweenadjacentframe......................128Figure7.7Histogramplotofadjacentframedifference................129Figure7.8Flowchartofmotioncompensationprocess.................130Figure7.9Macroblock(4:2:0)..............................131Figure7.10H.261framesequence............................132Figure7.11MPEG-1framesequence...........................133Figure7.12MPEGcompression.............................133Figure7.13Spatialandtemporalresolutiontrade-offinvideostream..........134Figure7.14Framedifferencesampling..........................135Figure7.15Countourmapsofdifferentpenaltiesandfeasiblesetofy=Fxatp=0;1=2;1;and2................................136Figure7.16Countourmapsofproposednonconvexsorted`1withM(M=4)values..139Figure7.17Signalrecoveryondistinctsparsity,4096inlength.............144Figure7.18Adjacentframeintensitydifference.....................144Figure7.19Signalrecoveryondifferentsamplingrate..................145Figure7.20Accumulationresidualerror.........................146Figure7.21Movingobjectvideorecording........................147Figure7.22Rotatingobjectvideorecording.......................147xvChapter1Introduction1.1InfraredEverywhere1.1.1FundamentalofInfraredThediscoveryofelectromagneticradiationotherthanvisiblelightcamein1800,whenWilliamHerscheldiscoveredinfrared(IR)radiation[1].Itiswidelyusedincivilianapplicationfromindus-trial,agricultural,nightvision,buildinginspection,medicalthermographyandmeteo-rology,medicaldiagnosisduetothattheIRcameracanexploremoreinformationthanvisiblelightcamera.Mostimportantly,itoperatesinnightandlongdistancecomparedtovisiblelightcamerasothatitisbecomingoneofmostpopularnon-destructivediagnostictechnologyinindustry.Therearethousandsofcommercializedapplications,includinghyperspectralimaginginbiologicalandmineralogicalmeasurements,targetacquisitionandtracking,night-vision[2],IRdatacommuni-cationsbystandardspublishedbyIrDA,Infraredtelescopeinastronomy,environmentmonitoringinmeteorology,etc.Besides,infraredphotographyisnotonlyapplicableinindustry,butalsoforresearch.ThemostpopularapplicationisFourierTransformInfraRedspectroscopy(FTIR)[3],whichdetermineswhatfractionoftheincidentradiationisabsorbedbypassinginfraredradiationthroughasample.AnothercounterpartapplicationisthatStimulatedRamanScattering(SRS)fromtheuseofpulsednear-infraredlasers,whichgenerateshighsignallevelsatamoderateaveragepowerinbiomedicalcuttingedgeresearch[4].Meanwhile,theinfraredphotog-1Figure1.1Planck'slaw(coloredcurves)andclassicaltheory(blackcurve).Forinterpretationofthereferencestocolorinthisandallother,thereaderisreferredtotheelectronicversionofthisthesis.raphydiscoversbeneathapaintingssurfaceandviewsdetailthatwouldotherwiseremainunseeninartscience.Itisalsoappliedtodetectdisease,insectinfestationinplantsscience[5].Intheofmedicinescience,medicalinfraredthermographyisanon-invasive,non-radiatinglowcostdetectionmethodforanalyzingphysiologicalfunctionsinsportsmedicineastraumatickneein-juries[6],cancerdiagnostics[7].Themostwidelyapplicationscomefrommilitary,whereinfraredlightisextensivelyemployedfortargetacquisitioninwideIRbandwidth.Infraredthermographydetectselectromagneticspectrumfrom720nmupto14mm.SinceIRradiationisemittedbyallobjectsaboveabsolutezerokelvinandtheirradiatedwavelengthdependsonitstemperatureaccordingtotheblackbodyradiationlaw[8],thermographyallowsonetoobservevariationsintemperature.Thistechnologyisespeciallyrelatedtohumanbodybecausehumansatambientroomtemperaturecanradiatearound12mmwavelengthinfraredlightbasedonWiensdisplacementlaw[8].Theplanck'slawdescribestheblackbodyelectromagneticradiationinthermalequilibrium.2AsshowninEq.(1.1),BvrepresentsthespectralradiancewhichtellstheamountofenergyatdifferentwavelengthinabsolutetemperatureT,wherekBistheBoltzmannconstant,histheplanckconstant,cisthespeedoflightinthemedium.InFigure1.1,thecoloredcurvesshow5000K,4000Kand3000Kenergyradianceaccordingtowavelengthrespectively.Inlongwavelengthrange,thePlanck'slawtendstobeRayleighJeanslaw,whileitisclosetoWienapproximationinshortwavelength[9].Thepeakwavelength(lmax)canbenumericallyevaluatedbysolvingmathematicalequationEq.(1.1)andconcludedasEq.(1.2).Bv(v;T)=2hv3c21ehvkBT1(1.1)lmax=hcx1kT=2:89776829106nmKT(1.2)TheIRimagesmeasuretheinfraredirradiationanditsdistributions.TherearetwoIRsourcesforimaging,oneisinternalemissivityasPlanck'slawdepictedandanotherisexternalsimilarasvisiblelight.Thelight,e.g.sunlight,indoorlighting,isnotonlythepre-dominateelementforvisibleimagesensors,butalsocontributesonnearIR,SWIRimaging.Innaturally,bothsunlightandairglowatnightgeneratenearIRlightwaveandradiateonobjectsothatthecameracouldcapturethevelighttoformanearIRimage.Thewavelengthofman-madeIRilluminatoralsolocateswithinnearIRandSWIRspectrum,e.g.lightbulbsandsolidstateLightEmittingDiode(LED).Theincandescentlightbulbsheatatungstentohightemperatureandproducevisiblelightbuttogetherwithinfraredradiation.HoweverthesolidstateLEDaremoreefwithnearmonochromaticinfraredenergy,whichdependsonthesponta-neousandstimulatedemission.Whenanelectronorbitsthenucleusofanatom[10]inhighenergystate,ithaschancespontaneouslydecaypathtolowenergy.Theelectrondecayinsuchamanner3Table1.1Infraredsub-divisionscheme.DivisionAbbreviationWavelengthFrequencyPhotonCharacteristicsName(mm)(THz)Energy(meV)Near-infraredNIR,IR-ADIN0.72-1.4214-400886-1653PassivenightvisiondevicesShort-wavelengthinfraredSWIR,IR-BDIN1.4-3100-214413-886Waterabsorptionandlong-distancetelecommunicationsMid-wavelengthinfraredMWIR,IR-CDIN3-837-100155-413Atmosphericwin-dowandthermalinfraredabovebodytemperatureLong-wavelengthinfraredLWIR,IR-CDIN8-1520-3783-155Thethermalimagingregion,requiringnoilluminationFar-infraredFIR15-10000.3-201.2-83Far-infraredlaserwillintroduceaphotonemittedatexactlythesamewavelengthandphase.Itbecomesman-madeinfraredlightsourcewhenphotonenergyiswithininfraredspectrumasshowninTable1.1.Itliststheinfraredbywavelength,frequency,photonenergyandapplication.1.1.2ConventionalInfraredSensorIRsensorcanbebroadlydividedintotwocategories:cooledanduncooleddetector.Italsocanbedbydetectionmechanism:athermaltypethathasnowavelengthdependence(thermaldetector)andaquantumtypethatiswavelength-dependent(photondetector)[11].Infraredradia-tionenergyisequaltothevibrationalorrotationalenergyofmoleculesrangingfrom1.24eVat1mmdownto0.12eVat10mm.ManyspecializedIRsensorstructuresandmaterialshavebeenmanufacturedandcomparedasthermistorforIRdetection[12].Itsensestheheatirradiation,throughwhichtheresistancewillbechanged,andareadoutcircuitmonitorsresistancevariationtodeterminephotocurrentresponseof4IRsensor.Thewholeprocesswillcosttoomuchtimeandmakethesensorrelativelyslowresponsebecausethedetectorelementissuspendedonlagswhichareconnectedtotheheatsink,thoughitworksinroomtemperatureandindependentofwavelength.Thequantumdetectors,includingindiumantimonide,indiumarsenide,MCT[13],leadandleadselenide[14],usuallyneedacryogenicallydewarfortheoperationofsemiconductormaterials.Theforeinfraredphotodetectorsarealsoknownasquantumdetectorswhichdependonthebandgapofmaterials.Theyofferhigherdetectionperformanceandafasterresponsespeed.However,theyalwaysworkinverylowtemperaturetokeephighresponsibility,sothatacryogeniccooledsystemisneededtomaintainIRsensorworkstable.Eventhoughsomeotherphotoelectriceffectquantumdot(QD)andquantumwell(QW)IRsensorsimproveworkingtemperatureanditcoulddetect10mmwavelengthatroomtemperature[15],thesensitivitydeterioratesmuch.Atthesametime,thebulkycoolingsystemisagainstwithportabledesignofimagingsystem.1.1.3InfraredDetectorMarketInfraredcamera(imager)isalsonamedfocalplanearray(FPA)orIRsensorsininfraredindustryandresearch.Itisathermalsystemwhichconvertsinfraredradiationintoavisibleimageandthecoreofcameraincludeselectronics,IRlensesandsensor.Inoverall,themilitarydemanddominatesinfraredcameramarketinlast50years.However,sincethemilitarymarketforuncooledinfraredimagingtechnologiesdeclines,itturnstowardcommercialbusinessessuchaspersonalvisionandsmartphonesbyYolereport[16].AlthoughthewidelyapplicationofIRimagingcoversconsumerelectronics,surveillance,aerospaceanddefense,automotive,industrial,medical,andetc.,theIRcamerasmar-ketgrowthisbeingdrivenbytheultra-low-endmarketwhichconsistsoflow-resolutioncamerasforbasicradiometricpurposes.Itisestimatedthattheglobalinfraredimagingmarketwillreach5$8,450millionby2020[17].Theglobalinfraredmarketisdividedintofourgeographicsegments,theAmerican,Europe,Asia-PandRestoftheWorld.MajorplayersintheIRimagingmar-ketareFLIRSystemsInc.(U.S.),DRSTechnologiesInc.(U.S.),FlukeCorporation(U.S.),AxisCommunicationsAB(Sweden),SamsungTechwin(SouthKorea),SeekThermalInc.(U.S.)andSofradirGroup(France).Undoubtedly,AmericasisconsideredtobetheleaderintheoverallIRimaging/sensorsmarket.Thereareatleastvetrendsfromapplicationinsight.-Thermography:theultra-low-endcameraswithattractivepricing,lowerthan$1,000drivethemarket.Theywillwidenthecustomersofthermaltechnology.Therearemanynewmodelsreleasedfromleadingcompanies,FLIR[18]andFluke[19]in2014and2015,andtheyaregoingtoleadthepricewarduetotheirverticalbusinessmodel.-Automotive:themarketleaderAutolivwillcontinuetointroducenightvision3rdgenerationonnewcarmodels.NewEuroNCAPcouldboostthemarketbypromoting,in2018,nighttimepedestriancollisionmitigationsolutionspotentiallyusingathermalsolution,butonlyifthecostissuflow.FLIRSystemsalsoprovidescameracoresforRollsRoyce,BMW,AudiandMercedesBenzmodelsthroughapartnershipwithAutolivElectronics.-Surveillance:infraredcameraisagoodsurveillancerobustequipmentthatcanhandleanyenvironmentalconditionsfornaval,airandgroundsecurity.SWIRworksforcounter-aerial,verylongrangelandandsea,largeareanavalsurveillance.Priceerosionwillcontinue(-12%/year)andwillenlargethescopeofcommercialapplicationsliketrafparking,andpowerstations.-Consumerapplications:thisisthefastgrowthin2013-2014.Personalvisionsystems(gog-gles,sightforsecurity,andhunting,outdoorobservation)dominatecivilapplications.Sincetherearemanynewentrantsarrivingfromtheoutdoorvisiblebusiness,itwillcontinuetogrowinfutureyears.6-Smartphones:Therearetwosmartphonemodules(FLIROne,OpgalAndroid)andSeekTher-malbeenintroducedatthegroundbreakingpriceof$349byFLIRand$249bySeekThermalin2014.Ahighnumberofpre-releasereservationsfortheFLIROnealreadyprovesthecommercialsuccessofthissmartphoneplatform.Thesmartphonebusinessisanalmostbillionmarket.ThosehighvolumeswillonlybepossibleifahugecostreductionisobtainedbytheIRimagingindustry.Atsensorlevel,majormanufacturers(suchasDRS,FLIR,Raytheon,ULIS,GWIC)havenowmovedto8inchproductionlinesinsteadof6inchtoreducewafermanufacturingcost.Severalhaveevenoutsourcedtheirproductiontofoundriestofurtherreduceproductioncost.Thesetwoelementsarepreliminarysignsofastrongmicrobolometercostreductionthatwillopenupcostdrivenapplicationssuchassmartphones.Atcameralevel,verticallyintegratedplayers,withinternalsensormanufacturing,canbefromtheirefcoststructurestoenteranycommercialmarketwithanaggressivecameraprice.Forinstance,DRSandFLIRleadthepricewarinsurveillancewhileFLIRhasintroducedalow-costinfraredcameraforin2012.Thisrepresentsamajoradvantageforverticallyintegratedplayersbecausetheycanleveragehighvolumemanufacturingthatthesinglemarketcamerasspecialistscannot.1.2CarbonNanotubeBasedInfraredSensor1.2.1SensorsCharacterizationThemajorIRdetectorperformancecriteriaindicatinginfrareddetectorperformanceareoperatingtemperature,photosensitivity,noiseequivalentpower(NEP)anddetectivity.-Photonsensitivity(Responsivity)Whennoiseisnotamainconsideration,thephotonsensitivitycanbecalculatedbytheoutput7(voltageorcurrent)perwattofincidentenergy,showinEq.(1.3).R=SPA(1.3)R:Responsivity,[V/W]S:SignalOutput,[V]or[A]P:Incidentenergy,[W/cm2]A:Detectoractivearea,[cm2]Inphotovoltaicinfrareddetectors,theoutputsignalsareextractedasphotocurrent.Itisex-pressedasEq.(1.4),whenthelightisatagivenwavelengthirradiatedondetector.ISC=hqPAhcl=hqPAlhc(1.4)q:Electroncharge,[C]Theresponsivityofphotovoltaicphotodetectoratwavelength(l)willbeasEq.(1.5).Rl=ISCPA=hqlhc=hl1:24(1.5)h:QuantumefyHowever,theoutputofphotoconductivedetectorisvoltage.TheoutputvoltageVOwillvary(DVO)duetochanges(DRi)ofinternalresistanceRiwhenexposingtoinfraredlight.VO=RLRi+RLVB(1.6)DVO=RLVBRi+RL2DRi(1.7)8DRi=Riq(me+mh)shtlPAlwdhc(1.8)t:Carrierlifetime[S]me:Electronmobility[cm2/(Vs)]mh:Holemobility[cm2/(Vs)]s:Electricalconductance[S/m]DRl=DVOPA=q(me+mh)lthslwdhcRLRiVB(Ri+RL)2(1.9)Althoughthephotonsensitivityissocomplicated,itcanbeexpressedbyEq.(1.9).ItisonlyapplicabletofewcasesbecauseRi,me,mh,tandsaredependentwitheachother.Moveover,itisalsorelatedtobiasvoltage(VB)appliedonthedetector.-Noiseequivalentpower:NEPNoiseequivalentpower(NEP)isanothercriticalvaluetomeasurethesensitivityofaphotode-tector.Itisasthesignalpowerthatgivesasignal-to-noiseratioofoneinaonehertzoutputbandwidth[20].TheNEP(W/Hz1=2)measuresthequantityofincidentlightwhenthesignaltonosieratio(S/N)isone.NEP=PAS=NpDf(1.10)N:Noiseouput,[V]Df:Noisebandwidth,[Hz]ThesmallerNEPis,thephotodetectorwillbemoresensitive.Forexample,adetectorwithanNEPof1012W=pHzcandetectasignalpowerofonepicowattwithaSignal-to-NoiseRatio(SNR)ofoneafteronehalfsecondofaveraging.TheSNRimprovesasthesquarerootofthe9averagingtime,andhencetheSNRinthisexamplecanbeimprovedto10byaveragingfor50seconds.Eq.(1.10)onlyreferstotheelectricalNEP.ThereisanotherNEPrelatedtothedetectorsystem,calledopticalDEP.TheopticalNEPisequaltotheelectricalNEPdividedbytheopticalcouplingefyofthedetectorsystem.-Detectivity:D(D-star)Detectivityisthephotonsensitivityperunitactiveareaofaphotodetector.Thisiswidelyusedtocomparethecharacteristicsofdifferentdetectors[21].ThedetectivityisgiveninEq.(1.11),whereAistheareaofthephotosensitiveregionofthedetector,fisthefrequencybandwidth[22].WhenmeasuringDinexperiments,itisrelatedtotemperature(T:[K]),wavelengthofaradiantsource(l:[mm])andthechoppingfrequency(f:[Hz]).Basedonexperimentalresultsreported,thedetectoralwayshasapeaksensitivitywavelength.D=pAfNEP(1.11)-OperatingtemperatureTherearetwosensingmechanismsforsemiconductorbulkphotondetector:themajoritycarrierandtheminoritycarrier[23].Thesensingwillbephotoconductiveinnatureifitismajoritycarrierdominant,whileitisbothphotoconductiveandphotovoltaicmodesiftheminoritycarrierdominatesdeviceoutput.BothcarriermobilityandthermalnoisearetemperaturedependentsothatoperatingtemperatureiscriticalincharacterizingofIRsensor.Inpractically,itisnecessarytoconsiderwavelength,responsetime,temperature,coolingmethod,sensingarea,numberofsensingelementsforinfraredapplication.101.2.2NanoMaterialIRPhotodetectorCarbonnanotubeshavebroughtextensivelyattentionboththeoreticallyandexperimentallysinceitsdiscoverysothattherearemanynanoelectronicdevicesandNEMSusingsingleormultiwallcarbonnanotubes[24].Theultrahighsurface-area-to-volumeratioandquasi1Dnearballisticelectronictransportmakeitattractiveinsupercapacitors[25],solarcells[26],mechanicaloscilla-tors[27],gassensors[28].CarbonnanotubebasedMEMSalsodrewincreasedattentioninsingleCNT,CNT[24].TheobservationofphotoelectriceffectinCNThasopenedanumberofavenuesofresearchinbothcharacterizationandphotonicapplicationsofcarbonnanotubes[29].InfrareddetectionusingCNTswasrealizedandreportedin[30]and[31].TheCNTbasedIRsensors,includingindividualsinglewallcarbonnanotube(SWNT)basedSchottkydiodestructure[32][33][34]andCNTeffecttransistor(CNTFET)modulatedstructure[35]werealsoreported.CNTisonedimensionalnanomaterialswithhexagonhollowcylinderstructures,whichshowsoutstandingmechanical,electricalandoptoelectronicproperties[36].Withdevelopmentofmorethan20years,thetheoreticalanalysisandpotentialapplicationsarefound.Dependingonitschirality,carbonnanotubeshavearmchair,zigzagandchiralstructure[37],whichareassemiconductorandmetalmaterial.ThebandgapenergyofsemiconductorCNTcanbemodulatedbycontrollingthediameterofCNTinordertodetectdifferentwavelengthofIRlight[38].TheelectrontransportinCNTisinonedimensionsothatthethermalnoiseofCNTIRdetectorisextremelylowduetophononscatteringsuppression[39].ThiscreatestheCNTdevicewithstableresponseinroomtemperature.Thedistinctpropertiesofnanomaterialdistinguishitfromtraditionalbulkmaterials.Therearemanyresearchesturningtonovel1D,2Dmaterialsanditsderivatives.Thecarbonnanotubes11areverygoodthermalconductorswithballisticconduction,sothatitisusedasaultrasmallscaletemperaturesensor[40],whichmakesthesensorperformancebetterthanotherdevicesinsimi-larsize.Theresultingdeviceexploitsthesuperiorthermalandelectricalpropertiesbyderivingthetemperaturebasedonachangeinelectricalresistance[41].Carbonnanotubenotonlyworksforthermaltypebutalsoquantumtypeinfrareddetector.Therearemanycontributionsonsin-glewallcarbonnanotubebasedinfrareddetector,inwhichthecarbonnanotubeischaracterizedassemiconductormaterialswithbandgapwithinnearinfraredwavelength.ThesecondinfrareddetectorsaredemostratedusingMWCNT.In[42],differentmorphologiesofMWCNTaresyn-thesizedtodetectinfraredradiationinroomtemperatureviameasuringphotoconductance.ThethirdisCNTbasedIRphotodetector.MostexperimentaldataofthesedetectorssuggestthattheIRphotoresponsearisesmainlyfromthethermaleffect,asin[43][30].ItwasalsoreportedthatphotoexcitationeffectpredominatedtheIRphotoresponseinCNTat[44].Alloftheseinfrareddetectorscanworkinroomtemperaturewithfastresponsivityinsmallbandgap.1.3ComputationalImagingComputationalimaging(Photography)referstodigitalimagecapturingandprocessingtechniqueswhichusedigitalcomputationalmethodinsteadofopticalprocesses.Thegoalistoovercomethelimitationsoftraditionalphotographyandenhancethewayofcapturing,manipulating,andinteractingwithvisualmedia.AsshowninFigure1.2,thetraditionalcameraismim-icshumaneyeforasinglesnapshot,singleview,singleinstant,eddynamicrangeanddepthofeldforgivenilluminationinastaticworld.Thecameracomesfrombasicgeometryopticstoformimage.However,thecomputationalimageuseonemoreprocesstorecoveryimage,asshowninFigure1.3.Computationalphotographywhichenhancesorextendsthecapabilitiesof12digitalimaging,isoneofmostrapidlydevelopingresearchincomputervision,imagepro-cessingandappliedoptics[45].Theoutputofthesetechniquescanreconstructinformationofscenewhichisnotobtainedbytoday'sdigitalcamera[46].ThecurrentresearchhasevolvedmanyFirstofsuchtechniquesiscomputationalimaging,includinghighdynamicrangeimaging(HDR),lightimaging,colormanagement,etc.[45].Thehighdynamicrangeimagingin[47],isachievedbyplacinganopticalmaskadjacenttoimagesensorarray,followedbyanefimagereconstructionalgorithms.MorereconstructionsalgorithmonHDRimagearein[48][49].Lightimaging[50][51]isusedtoanalyzeimagepartsthatarenotinfocusandextractdepthinformationbyraytracing.Thenovelcomputationalphotographyinvolvesopticscodedexposureimaging,whichinsertsapatternedcodedaperturetorecoverbothdepthinformationandanallfocusimagefromsinglephotographs.Compressivecodedaperture[52]iscombinedinsupperresolutionimagereconstructionfromlowresolution[53].Anothernovelcomputationalphotogra-phy,whichisbasedonnewmathematicaltheoryandalgorithmsofcompressivesensing,combinessamplingwithcompressionintoasinglenon-adaptivelinearmeasurementprocess,namedsinglepixelimaging[54].ThesinglepixelcamerautilizesSpatialLightModulator(SLM),comprisedofmillionsofelectrostaticallyactuatedmicromirrors,toprojecttargetimageintoalowdimension.The2-DmirrorsworkasopticalswitchwithtwostatesONandOFF,sothatthesinglepixelcam-erameasurestheinnerproductbetweenanMN-pixelimageandtwodimensionalfunctionsinmatrix.IthasbeenprovedbyEmmanuelCandes,TerenceTaoandDavidDonoho[55][56][57]thatthesparsitysignalmaybereconstructedwithfewersamplesthanNyquistShannontheoremrequires.Extraconstraintsareimposedsoastogetauniquesolutioninunderdeterminedsystem.SincecompressivesensingisNon-deterministicPolynomialtimehardproblem(NPhard),oneapproachisdirectlytousegreedyselectionalgorithms.Becauseofitsnon-convex,thereisnoguaranteetoglobalminimizerandthesolutionisunreliable,thoughitisfastinthisapproach.13Figure1.2Conventionaldigitalphotography.Figure1.3Computationalphotography.14Figure1.4Optics1.4HighDimensionalPlenopticFunctionofLightThelightbehaviourhasbeeninvestigatedwhenMichaelFaradayproposedthatthelightshouldbeinterpretedasamagneticTheradiometrydescribeshowenergyistransferredfromlightsourcestosurfacepatches.Inperviousresearch,thestudyoflightcanbedividedasquantumoptics,physicalopticsandgeometryopticsfrommicrotomacroscale.TheinclusionrelationisshowninFigure1.4.Fromtheresearchareapointofview,thequantumopticsisthestudyoftheinteractionofobjectwithlight.Iftheobjectisinsub-wavelengthornanometerscale,itisreferredtonanophotonics[58].Thephysicaloptics'stopicsincludeinterference,diffraction,polarizationetc.,whichdiscussphysicaltheory.Theabstractgeometricalopticsarealsonamedrayopticswhichuseapproximationmethodtodescribethelightintermsofrays.Theresearchwillconcentrateonhowtomathematicallyexpressraysandgetimagesbyrayoptics.Inrayoptics,theplenopticfunctionwasproposedforlightsince1991byAdelsonandBergen[59].Inordertodescribethenatureluminousenviornment,therearesevenparametershighdimensionalmodel(7-Dimensionalfunction)considered,showninEq.(1.12)(polarcoordinates),Eq.(1.13)(Cartesiancoordinates),whereV=(Vx;Vy;Vz)istheviewpoint,S=(q;f)/S=(x;y)isthedirectionoftheraylightpassingthroughtheviewpoint.15P=P(q;f;l;t;Vx;Vy;Vz)(1.12)P=P(x;y;l;t;Vx;Vy;Vz)(1.13)Foragrayscalephotographtakenbyapinholecamera,itonlyshowstheintensityoflightfromasingleviewpointinstatic.Thewavelengthisaveragedoverthespectrum,showninFigure1.5left.ThegrayscaleimagecanbeparametreizedbyEq.(1.14)orEq.(1.15)intwodistinctcoor-dinates.Acolorimagewilladdwavelengthinformationtomakeitasafunctionofwavelength(P(q;f;l)),asFigure1.5right.Thecolorvideostreamwillextendtheinformationtocovertimevariable(P(q;f;l;t)).Moreover,theholographicmoviewouldreconstructofscenefromeveryviewpointsuchthatsevenparametersmodelisrequired.Inordertomakethemodelapplicable,theassumptionincludesofrayspassingthroughfreespace,theregionsfreeofoccluderssuchasopaqueobjectsandthelighttravelingviaarayinconstantalongitslength[60].The6Dlightwillbediscussedinthisresearch,asEq.(1.16).P=P(q;f)(1.14)P=P(x;y;l)(1.15)P=P(x;y;l;t;u;v)(1.16)1.5DissertationOverviewThecentralcontributionofthisdissertationistodesignareliablelowdimensionalmaterialbasedIRsensor(CNT)andimagingsystem.Ageneraloverviewoftheresearchisdiscussedinsection16Figure1.5P(x;y)andP(x;y;l).ofintroduction.ThefollowingaresixthemesincludingnanomaterialIRsensor,computationalcamera(singlepixelIRcamera),andfourbeyondIRcamerasystems,asshowninFigure1.6.1.6OrganizationoftheStudyThereliableCNTIRphotodiodedesign,fabricationandtestingmethodarepresentedinChapter2,includingnovelsensorstructuredesign,anefsensorfabricationmethodandcharacter-izationofnanoscaleIRsensor.ItisfollowedbydevelopinglowdimensionalmaterialbasedIRcamera,whichcomprisedofcompressivesensingandsingleIRsensorinChapter3.Thecamerareadoutsystemwasalsodiscussed.InChapter4,thecompressivelightsensingmethodforIRspectrumisintroduced,whichmakesIRlightsensingavailable.InChapter5,3-DimagingmethodisdescribedtoexploremoreofsceneinIRwavelength.InChapter6,thesuperresolutionfrommultipleimagestosingleimageanditsimportancetosinglepixelimagingsystemaredis-cussed,whichopenthedoorofnanosensorIRimagingsystemtocommercialization.InChapter7,thenonconvexcompressivevideosensinganditsimplementationmethodwasproposedtodis-covermoreIRmessagesintemporaldomain.Finally,Chapter8presentsthesummaryofresearchandfutureworks.17Figure1.6Dissertationoverview.18Chapter2LowDimensionalMaterialBasedInfraredPhotodetector2.1PreviousWorkAlthoughnanotechnologyhadattractedahugenumberofattentionssincecarbonnanotube(CNT)wassynthesizedandreportedbySumioIijimain1991atNEC[61],andtheopticalab-sorptionspectraoftheSWNTswereobservedfromvisibletonearinfraredregion[62],ahighperformanceCNTbasedinfrareddetectorwasstillunavailable.Overthepastseveralyears,nu-merousstudieshavebeenperformedintheofcarbonnanotubebasedphotodetectors.Inall,itcanbeassinglewallCNT,multi-wallCNTsandCNTForsinglewallCNT,ithasarmchair,zigzagandchiralstructuredependingonitschirality[63],whichareassemi-conductorandmetalmaterial.InordertodifferentiatewavelengthofIRlight,thebandgapenergyofsemiconductorCNTcanbemodulatedbycontrollingthediameterofCNT[38].TheelectrontransportinCNTisinonedimensionsothatthethermalnoiseofCNTIRdetectorwillbeextremelylowduetophononscatteringsuppression[39].ThiscreatestheCNTdevicewithstableresponseinroomtemperature.Sofar,nearlyallCNTbasedIRphotodetectorssensingschemearebasedonCNTmetaljunctionsforphotonelectrontransition[32].TheSchottkybarriers,showninFigure2.1,willbeformedatthecontactregionsbetweenmetalandsemiconductorCNT.WhenIRlightirradiatesontocontactregion,photoninducedelectronswillbeinjectedfromCNTtometal19surface,togeneratephotocurrentinclosecircuitry.Figure2.1BanddiagramofCNTmetalcontact.However,CNTandmetalcontacthasnegativeeffectforphotoconductancebasedIRphotode-tector[42][44]sothatthesmallcontactresistanceispreferred.TheexperimentalresultssuggestedthatPdandTicontactsweresuperiortoAuandPtcontacts,buttheresultsforTiwereerratic,pos-siblyduetothehighchemicalreactivityofTicomparedtoothermetals[64].ThecalculationsalsodemonstratedthattherewasnoSchottkybarriertoelectrontransferbetweenPdandnanotubeattheinterface,becauseinterfacestates,duetothechargetransferatthePd/CNTcontact,thebandgapofthesemiconductingCNT,resultinginacontactofmetallicnature[65].InordertoreducecontactresistanceinCNT-FETs,theuseofagraphiticcarbon(G-C)interfaciallayertosemiconductingCNTcanimprovetheelectricalcontacttothesemiconductingCNTandreducethesubthresholdswingoftransistorswiththeseimprovedcontacts[66].Notonlyontherelation-shipbetweencontactandphotoresponse,itisknownthattheone-dimensionalgeometryofCNTsmakesthemhighlysensitivetotheirelectrostaticandelectrochemicalenvironment[67].In[67],italsodemonstratesthatanelectrochemicalchargetransferreactionistheunderlyingphenomenongoverningthesuppressionofelectronconductioninCNTsdevices.Besides,thedeviceisalsosensitivetoelectricalAsshowninFigure2.2,thefiofffl-stateandfionfl-staterepresenttwodifferentgatevoltageconditions,whichgeneratedistinctbanddiagraminCNT-FETSchottkybar-rier.Thiskindofbandbendingaffectsthewofelectronsfromsourcetodrainandaltersdeviceproperties.Unfortunately,CNTmetalcontactsensingmethodologysuffersfromlimitedsensingareaand20Figure2.2BandbendingofaSchottkybarrierinCNT-FET,fortwogatevoltages.weakopticalabsorption.Thefullyexplanationofhownanoscalematerialrespondtoinfraredspectrumiscriticalinlowdimensionmaterialsresearch.Inthiswork,itisnotonlyforthefunda-mentalunderstandingofCNTphotoinducedprocesses,butalsofordevelopmentofanewinfraredphotosensitivematerialwithuniqueopticalandelectricalfeatures.2.2CNTIRSensorDesignAsdiscussedinprevioussection,therearetwokindsphotoresponsemechanismsofCNTbasedinfraredphotodetector.Theyarequantumphotovoltaicandphotoconductanceeffectrespectively,inwhichthesingleCNTphotodetectorhappensonquantumphotovoltaicwhiletheCNTorarraysbasedIRsensorsdependonthephotoconductance.IndifferentCNTdevices,theCNTsworkasdistinctfunctions.However,theCNTs-metalcontactismostwidelyusedstructurefornanoelectricalcircuits[68]andnanosensors[69].Inthisresearch,asingleCNTispreferredduetoitsuniqueperformancesothatCNT-metalcontactisinvestigatedmoredeeply.TherearetwotypesofCNT-metalcontactareainmostresearches,theradialdirectionsidecontactandsidewallcontact.AsshowninFigure2.3,therearethreelocationsofCNTinthesidecontact.TheCNT-metalcontactcanbeformedatbottomside(Figure2.3a),topside(Figure2.3b),andmiddlecontact(Figure2.3c).TheCNTontopofmetal(bottomcontact)canbemanipulatedbygrowthdirectlyandDielectrophoretic(DEP)assembly.However,theDEPmethodisnotsuitableforCNTunder21Figure2.3CNTmetalcontactwiththreedistinctpositions.a)CNTonthetopmetal;b)CNTundermetal;c)CNTbetweenmetal.metal.Intheproposedapplication,CNTsareintegratedintoinfraredopticalsensorsothattheFigure2.3(a)structureismorephotoelectricconversionefyandthecontactareawillbetransparenttoIRirradiations.Inthisdesign,thereisnotopmetaltoblockinfraredlightandtheIRpenetrationdepthonCNTcanreach450nm[70]toincreaseefy.2.3RealtimeDEPFabricationThetechniquesformanufacturingCNTbasednanodevicescanbegenerallyintobottomupandtop-downtwodistinctmethods[36].IntheCNTsaregrowndirectlyontodevicesub-strateusingchemicalvapordepositionmethod[71].Withthedirectlygrowthmethod,singleCNTdevicesarefabricatedbygrowingasingleCNTbetweenapairofprefabricatedmicroelectrodestomakeconnections.ThedirectlygrowthmethodisabletofabricatemultiplesingleCNTdevicesatonce.ThusitisgoodformakingCNTbasednanodevicearrays.However,thelimitationofthismethodisthatthepropertiesoftheCNTscannotbeeffectivelycontrolled.DifferentCNTsmayhavedistinctpropertieseventheyareproducedatonebatchbasedonitschirality[72].Moreover,22theproductionprocessmaygenerateimpuritiesaroundthemicroelectrodesandCNTs,whichwillaffecttheelectronicpropertiesoftheCNTdevice.Thirdly,itisdiftogrowonlyasingleCNTbetweenthemicroelectrodeswhiletheyarebundlesorlosingtheuniquepropertiesbroughtbythe1-DstructureofCNTs.ThemostdisadvantageofdirectlygrowthmethodcomesfromCVDprocessinveryhightemperature[73].ThislimitsthesubstratessuchassiliconsapphiresothatitcouldnotwidelyworkforxiblematerialsandsomebiocompatibleMEMSapplications.Thesecondcategoryistogrow,purifyandsortCNTs,andalignCNTsusingassemblymanipulation.TherearetwowaystomanipulateCNTsincludingDEPmanipulation[74]andnanoprobebasedmechanicalmanipulation[75].WiththeDEPdepositionmethod,microelectrodesarefabricatedusingconventionalmicrofabrication.DuringlocalizingCNTprocess,adropletofCNTsuspension(appropriateconcentrationinethanol)isdroppedbetweenthemicroelectrodesandanACvoltageisappliedacrossthemicroelectrodes.CNTswillbeattractedbythedielec-trophoreticforceandbridgedontheelectrodestoformanelectricalconnection.AlthoughthenumberofCNTsattractedtotheelectrodescanberoughlycontrolledbyvaryingtheACvolt-ageandtheconcentrationoftheCNTsolution,itisverydiftodepositasingleCNTusingthismethod.HencetheDEPdepositionmethodisnormallyusedtofabricatedeviceswithCNTorCNTbundles.Thenanoprobebasedmechanicalmanipulationwasalsocallednanorobotmanipulation[75].In[75],asingleCNTattachedatthetipendwasmanipulatedusingfocusedionbeam.Inthismethod,thenanotubehastobemetal-coatedformanipulationandnotgoodforbuildingnanoelectronicdevices.DEPbasedassemblyispotentiallyoneofmostimportantbottom-uptechnologiesforfabricat-ingnanomaterialsbasedMEMSandNEMSblocks.Itisliquidmediumbasedmethodtotransportmicro/nanoparticlesatroomtemperature.However,CNTshaveverylowdispersibilityinmanysolventsduetoitshighVanderwaalsforcesbetweeneachnanotube,inducingstrongtendencyto23aggregatewitheachother.ThemostdifincontrollingnanotubedepositionusingDEPisthatalargenumberoftubesorbundlesoftubeswillbeaccumulatedbetweengapsothatareal-timecontrolledsystemisrequiredtomonitorthequantity.In[76],aconductancemeasuringwasusedtoestimatealargenumberofmultiwallcarbonnanotubeswithsmallimpedancebutnotapplicabletosinglewallCNTs.Theapproachin[77],usedaninsitudetectionsystemandlock-intoreadDCandACcurrentthroughDEPloopinordertoalignsinglewallCNTbetweengap.DuetothelargeimpedanceofSWCNT,thecurrentissuchweakbetweenelectrodesthatthelockinneedstimetointegrate.Therefore,thequantityisoutofcontrolinthispoint.ThewholeDEPbasedCNTassemblyprocesshasmillionsapplicationsfromelectricaldevicetobio-sensors.ItincludesgrowthofCNT,depositionofas-grownCNTsonelectrodesbydielec-trophoresis(DEP).Inbio-application,theextraprocessistomakeself-assemblybyfunctionalizingCNTswithdifferentchemicalsorevenDNAmolecules[78].Toacertainextent,thesemethodshavetheirshortcomingsintermsofrepeatability,massproduction.Ingenerally,thebothmethodsusuallycometogetherinfabricationprocessofCNTbaseddevice,wheretop-downmethodsareusedtofabricatesupportingstructuressuchascontactingelectrodes,andbottom-upmethodsareusedtoassembleCNTsandlocalizeitontodesiredposition.2.3.1IntroductionofDEPThedielectrophoresisforceoriginatesfromPohl'stheoryandDEPforcesFDEPandtorquesTDEPcanbecalculated[79]asEq.(2.1)andEq.(2.2).AsshowninFigure2.4,thenanoparticleisdispersedinmediumandanelectricalforcewillbeappliedonparticlewhenanACvoltageisON.FDEP=(pÑ)E(2.1)24TDEP=pE(2.2)InEq.(2.1)andEq.(2.2),pistheinduceddipolemomentofthenanoparticleandEisthenonuniformelectricappliedonelectrodes.TheDEPforcescanbesimpliasEq.(2.3)andEq.(2.4)ingeneral,wherenpisthevolumeoftheparticleandÑjEj2istheroot-mean-squareoftheappliedelectricFDEP=12npRe(fCM)ÑjEj2(2.3)fCM=epemep+2ep(2.4)Basedonthisequation,thedirectionoftheDEPforceisdeterminedbytherealpartoffCM.InEq.(2.4),emdenotesthecomplexelectricalpermittivityoftheliquidmediumandepisthecom-plexelectricalpermittivityofnanoparticle.fCMistheClausius-Mossottifactor,whichindicateswhetherthemediumortheparticleismorepolarizable.WhenfCMislarger0,itiscalledposi-tiveDEPforce,resultingtheparticlemovingtowardsmicroelectrodes(highelectricregion).WhenfCMislessthan0,itiscallednegativeDEPforce,resultingtheparticleismovingawayfromhighelectricregion.TheelectrohydrodynamicsofCNTsisnotonlyrelatedtoDEPforce,butalsobytheeffectoftheexertedontheparticle.TheviscousdragforceisproportionaltotherelativevelocityforaprolateellipsoidalCNT.ThevelocitydynamicsinamediumenvironmentcanbeexpressedbyNewton'ssecondlawforaparticlewithmassm,asshowninEq.(2.5).ItonlyconsidersDEPforce,viscousforceandrelativevelocity(uv).TheconstantfshowninEq.(2.6),isthetranslationfrictionfactorwhichdependsonsize,shape,andviscosityh.RecallingPerrinfrictionfactorsandfurtherdevelopedhydrodynamicapproachesbyHardingsandSmall[80][81],thefrictionfactormoving25atrandomisgeneralizedshowninEq.(2.7).mdvdt=FDEP+f(uv)(2.5)hfi=3phlln(l=r)(2.6)v=(FDEPf+u)(1e(f=m)t)(2.7)Figure2.4Illustrationofthedielectrophoreticmanipulation.2.3.2AssemblyMethodandSystemDEPassemblyreferstothemanipulationofmicro/nanoparticlesusingdielectrophoreticforce,whichexertsonpolarizeddielectricparticles[82],whentheparticlespresentinanon-uniformelectricwithdistinctdielectricconstantandpolarizabilitytosurroundingmedium.AsshowninFigure2.5,thenon-uniformelectricalisgeneratedbetweensymmetricelectrodeswithroundshape,(SEMimageshowninFigure2.7)andcarbonnanotubeshavedifferentdielectricconstanttosurroundingmedium(Ethanol).ThereisaDEPforceappliedtoCNTswhenturningonexternalACvoltage,whichwillbridgeCNTbetweengap[83].Inthisnovelsystem,aniso-26lationmethodwasproposedtomeasureimpedancechangeswithinDEPsystemloop.AsshowninFigure2.6,acustomizedtransformerstructurewasusedtoreaddi=dtvaluetodetectcurrentchanges.Whenanewnanotubeisbridged,asuddensmallchangeofimpedanceintroducesatremendouslargedi=dt,whichwillbeonbothsidesoftransformer.TheintegratedweaksignaldetectionmodulewilleasilyreadcurrentchangeandsendafeedbacktoturnACoff.Basedonthismethod,thenumberofCNTstrappedbyDEPisquantitativelyComparedtothemethodofreadingelectricalcurrentdirectly,theproposedindirectapproachresponsesmuchfastersinceitdoesnotreadabsoluteweakcurrentvalueinDEPsystem.Figure2.5DEPforceonCNTinanon-uniformelectrical(sideview).Figure2.6Real-timemonitoringDEPsystem.RedrowshowscurrentloopwhenCNTisbridgedbetweengap(yellow).ThesystemwillshutdownACsourcethroughfeedbackwhenimpedancechanges.272.3.3QuantitativelyCNTDepositionandDeviceFabricationInDEPprocess,theAuelectrodeswerefabricatedusingElectronBeamLithography(EBL)andlift-offtechniquetoget1mmwidthand1mmgaponsubstrate.TheCNTpowders(Buckyusa)weredispersedinliquidmedium(UN1170EthylAlcohol)for1.5hourultrasound.Inthedeposi-tionprocess,adropletoftheCNTsuspensionwillbedroppedontosubstrateandanACvoltageof1.5Vpeak-to-peakwithfrequencyof1kHzisapplied.WhenACvoltageisON,theweaksignaldetectionwillalsostarttoreadcurrentchange.Ifthepeakislargerthanthreshold,theACvoltagewillbeturnedoffautomatically.ThereisacriticaltreatmentforCNTafterACvoltageoff.SincethebridgedCNTisstillinalcoholmedium,thevolatilealcoholwillremoveCNToutfromdesiredpositionduetosurfacetensionwhenthemediumevaporates.TheextratreatmentforCNTistomergethesampleintoDIwaterbeforealcoholdries.AlthoughCNTisdissolvedintoalcohol,thesurfaceofCNTishydrophobic.WhenthesampleisimmersedinDIwater,theresidualmedium(Alcohol)willbediluted.Afteroneminute,theDIwaterisisotropictothehydrophobicCNTsothatthealignedCNTwillstayontheoriginalpositionwhenitistakenoutofDIwater.Therearetwogroupsofexperimentsdiscussed.Figure2.7andFigure2.8showtheSEMimageofMWCNTandSWCNTalignedbetweengapsusingproposedreal-timemonitoring.TheweaksignaldetectionsystemwillcountpulsegeneratedbynumberofCNTsaligned.Theresultsshowone,two,threemultiplewallCNTsdepositioninFigure2.7andnanotubescanbepreciselylocated.Inthisexperimentalsetup,theCNTdepositionyieldisveryhighwhichreach100%(20/20)forsingleCNTdeposition.ThetwoCNTsalignmentexperimenthas90%(18/20)deviceyieldratio.Figure2.8showssinglewallcarbonnanotubedepositionfromquantitativelycontrol,whichincludesonenanotubeandaCNTdepositioninlocalizedarea.Althoughthesingle28Figure2.7SEMimageofmultiwallcarbonnanotubes.ThereisonlyoneCNT(MC1)ontopdevice,twoCNTs(MC2andMC3)onmiddledeviceandthreeCNTsonbottomdevice(MC4,MC5andMC6).wallCNTdepositionisalittleloweryield(15/20),itispossibletoimprovebyreducingsolutionconcentration.ThethicknessofCNTiscontrollable,asshowninFigure2.8bottom.2.4SensorReliabilityandResponseEnhancementThenanomaterialsareeitherdiameterorthicknessbetween1and1000nm[71].Duetosuchsmalldimension,notonlythematerialpropertieschange,butalsothesensingstructureandmech-anismsaredistinctfromconventionalbuckmaterials.Ontheotherhand,nanostructuressuchasnanotube,nanorods,nanobeltsandnanopossesshighsurfacetovolumeratio,largepene-trationdepthandfastchargediffusionrate,whicharesensitivetogassuchasH2,CO,NO2andvolatileorganiccompounds.Meanwhile,electricalchargestrappedunderCNTwillalsochange29Figure2.8SEMimageofsinglewallcarbonnanotube.TopissingleCNT(SC1)bridged.ThebottomdeviceshowssinglewallCNTusingrealtimeDEPdeposition.deviceperformanceinnanoelectronics[84].ItisexperimentallyprovedthattheonedimensionalgeometryofCNTsmakesthemhighlysensitivetotheirelectrostaticandelectrochemicalenvi-ronment[67].In[67],italsodemonstratesthatanelectrochemicalchargetransferreactionistheunderlyingphenomenongoverningthesuppressionofelectronconductioninCNTsdevices.Fromthesepoints,thedesignofCNTbasedinfrareddetectorwillbemorecomplicatedthangeneralbulksemiconductormaterials.2.4.1SubstrateEffectandPackagingonNanoSensorSincetheinterfacial/isolationlayeronsubstrate(showninFigure2.3)haseffectonIRphotoresponsebytheirelectrostaticandelectrochemicalenvironment,therearefourdistinctisolationlayersdepositedinexperimentsrespectively,includingSiO2(300nm),Si3N4(180nm),quartz(500mm),parylene-C(4.5mm),polyimide(10mm)polymer.SiO2andSi3N4aregrownonsiliconfromUniversityWafer(http://www:universitywafer:com/).Thesubstrateisptype(100),0.01-0.02ohm-cm.Thestabilityandreliabilityaretwocriticalparameterstocharacterizenanoelectronicsdevice.30ItwasfoundthatthestabilityoftheCNTdevicewasaffectedbyoxygencontamination.Theexposureofsingle-walledCNT(SWCNT)samplesanddevicestooxygenappearedtohaveastrongontheirelectronictransportpropertiesasreportedpreviously[85].InordertogetastableIRresponseofCNTsensor,thepackagelayerusingparylene-CthinwasdepositedontopofCNT-metalcontactsoasforoxygenbarrier.Theparylene-CisdepositedbyPDS2010parylenecoater.ThethicknessofparylenewasacriticalparameterinthedevicebecausetoothinlayerwillnotisolateCNTfromsurroundingenvironmentandtoothicklayerwillhavemuchabsorptionofIRirradiance.Inexperiments,1mmparylenewascoatedonCNTIRsensorusingPDS2000parylenecoating.Inordertocomparedeviceperformance,allmeasurementswereperformedusingthesameinfraredsource(100mW830nm),andalldatewerecollectedbyAgilent4156Cprecisesemicon-ductorparameteranalyserintheroomtemperature(25C).Figure2.9showsthedarkcurrentinveIRsensorswithdifferentinterfaciallayer.Itwaslargerthan1.75nAonSiO2(asSiO2)interfaciallayerwhileitwasaround0.6nAonquartz(asQuartz).However,thedarkcurrentwasreducedtolessthan0.3nAonparylene-C(asParylene),Si3N4(asSiN)andpolyimide(asPolyimide).Thepolymercanisolateoxygen-CNTcontactundertheSchottkybarrier,especiallythethickerpoly-imidewillreducethedarkcurrentto0.1nAlevel.Meanwhile,thelinearitymeasurementswereshowninFigure2.10.FivedifferentsensorshaveverygoodlinearitywhenIRirradiancepowerlinearlyincreases.Theresultsalsoshowthatthesensorwithpolyimideinterfaciallayerhaslargestresponseandthephotocurrentisaround170nAat3000mW=cm2IRirradiation.TheSiO2layerhastheworstresponseataround30nAphotocurrent.InCNTbasednanoelectronicdevice,thereisnoexactfunctionforelectrontransportalthoughgreenfunctionsarewidelyusedtodescribeandsimulatecurrent.IntheproposedCNTbasedIRsensorstructure,theinterfaciallayerunderCNT-metalcontacthaseffectsonelectron31Figure2.9DarkcurrentmeasurementresultsonCNTIRdetector.Figure2.10LinearitymeasurementresultsonCNTIRdetector.32transport.ThedifferentinterfaciallayerunderCNT-metalcanchangeoxygendopinginCNT[67].ItisreportedthattheCNTsworksashighptypesemiconductorwhenfullyexposingtooxygen.Whentheinterfaciallayerischanged,thechemicalnatureofCNTwillbedifferentsoastomakepdopingchanges0.Intheexperimentalresults,Si3N4interfaciallayerinducedhigherphotocurrentthanSi02.WhenSi02waschangedtoSi3N4,therewasnooxygenunderCNTsothatthecontactbarrierwashigh.Paryleneandpolyimideisalsooxygenfree.WhenCNTwasdepositedonparyleneorpolyimide,theelectrochemicalchargewaslessthanonSi02,sothattheCNTwouldbeinlowptype.Meanwhile,itincreasedthebarrierheightresultingahigherphotocurrent.2.4.2ExtrinsicSurfaceStateEffectInnanoscalematerial,surfacestatecouldalsointroduceextraelectronenergystateinCNTbandgap.ThesurfacechargesbetweenCNTandmetal,CNTandsubstratearecriticalinthedevice,whicharecloselyrelatedtoparasiticcapacitance.Thecapacitancebetweentwoelectrodesincreasesdramaticallywhengapdecreases.Thiscouldcausethesensorbehaviortodepartfromexpectedsensingperformance.AsshowninFigure2.11,thereareatleastsixparasiticcapacitorsinthisdevice.CsistheinternalcapacitancewithinCNT-metalSchottkybarrier,whichmeasuresthebuild-inpotentialinCNT-metalcontact.InSchottkydiode,thedepletionregionisaninsulatorthatseparatesthemetallayeranddopedsemiconductorlayer,formingaparallelplatecapacitorCs.Thethicknessofdepletionlayercanbemodulatedbythemagnitudeofexternallyappliedvoltage.C1andC2aresubstratecapacitors(˝Cs).CsrandCslareparasiticcapacitorsbetweenelectrodeandsubstrate,whichissourcetodeterioratesensorresponse.FromcapacitanceequationEq.(2.8),theparasiticcapacitancedependsonthegapsizebetweentwoelectrodesandisolationmaterialsunderneath.AlthoughC1andC2islargerthanthanCi,itiscouplingtogroundandhasnoeffect33Figure2.11ParasiticcapacitancemodelofCNTmetalSchottkybarrier.onsensorresponse.eisthedielectricconstantofsilicondioxideanddrepresentsthegapsize.Whengapsizedecreases,theCsincreasesdramaticallyfornanoscalesensor.Thislargecapacitorstoresmorechargesbetweentwoelectrodesandchangeselectricalpotentialdistributionalongcar-bonnanotube.Moreover,itmayreduceSchottkybarrierheighttomakeIRdetectorperformanceworse.IntheproposedCNTIRdevice,alow-kmaterialwasdepositedasinterfaciallayertore-ducechargedistributionundercarbonnanotube.OntopofSiO2layer(150nm)therewas10mmpolyimidespinedontopasinterfaciallayer.Thepolyimidethicknessandsurfaceisthekeyprocess.Inthisresearch,thepolyimide(HDMicroSystems,Inc.PI-2555)wasspinedtwiceonsiliconbasedwaferwiththespeedof2000rpm.Afterspinprocess,itwasputinovenandcuredthepolyimideat300Cfor2hourwith5Cperminutestartingfromroomtemperature.Theresultingpolyimidethicknessisaround10mm.Ci=eSd(2.8)Thecapacitancemeasureschargestorageabilitybetweentwoisolationplates.Itisobviousthatthechargestorageundersensorwillchangesensingperformance.AsshowninFigure2.12,whenmetalatomsaredepositedonisolationlayer,thereispositivechargelayerformedonsub-strate.BasedonthismodelandFigure2.11,parasiticcapacitorsfromCiandCsrorCslwillaffect34Figure2.12Surfacechargestorageonsubstrate.chargeredistributioninsubstrate.Meanwhile,electronworkfunctiondifferencebetweenmetalandsubstratematerialwillalsocontributeonchargedistribution.TheelectronworkfunctionofAuisaround5.3eV[86],whiletheenergybandgapofsilicondioxideis9eV,thepolyimideis4.32eVfortwolayers[87].Thisdifferencewillgeneratechargeaccumulationonthesurface,whichcontributestosurfacevoltage.Thehugerdifferenceformslargerdynamicsurfacecapaci-tance.Inordertomakeinfraredsensorstableandhighresponsivity,thepolyimidecontributeslesschargestoredonCslsincetheworkfunctionofAuandpolyimidelayerislessthanothermaterialsdiscussed.2.4.3SensorResponseEnhancementAlthoughtheIRsensorresponsecanbeoptimizedbydevicestructure,thefactorofCNT-basedIRsensorsisstilllimitedbylowincomingelectricattheirnanometerscalesensingarea.ThephotoresponsesofCNT-basedsensorsarerelativelylow.ItismainlybecausethedetectionmethodologybasedontheCNT/metalschottkyjunctionsuffersfromlimitedsensingareaandweakopticalabsorption.TherearemanywaysproposedtoenhancetheperformanceoftheCNT-basedIRsensors,ofwhichthepromisingapproachisuseopticalantennastoenhancethelocalelectric[88].Helicalantennashaveahighgainoverabroadbandoffrequencycharacteristics.Theradiationalongthehelixaxisisfoundtobethestrongestwhenthecircumferenceofthehelixisoftheorderofonewavelength.ToimprovethefactorofCNT-basedIRsensor,thethree-35Figure2.13CNT-basedIRsensorresponseenhancementbyhelicalantenna.Figure2.14I-VcurveofCNTIRSensor.a)deviceA;b)deviceB.dimensionalhelicalstructureswithmicroandnano-featuresareneeded.Inthisresearch,thehelicalnanobelt(HNB)structurewasfabricatedfromtheInGaAs/GaAsbyatop-downfabricationprocessinwhichastrainednanometer-thickheteroepitaxialbilayercurleduptoformthree-dimensional(3D)helicalstructurewithnanoscalefeatures[89].ItservedastheopticalantennastoimprovethefactoroftheCNT-basedIRsensors,whichisschematicallyillustratedinFigure2.13.Theelectricalandphotoresponsepropertiesarebothtestedanddiscussedtofullycharacterizetheperformanceoftheintegrateddetector.TheresultsofelectricalpropertytestsforbothdetectorsareshowninFigure2.13andFigure2.14,withtheredlineforthedetectorafterintegrationandtheblacklineforthebareone.Itcanbereadilyfoundthataftertheintegration,thedetectorismoresensitivetothechangeoftheelectricalsignal,andthereforetheHNBantennaisofabetterperformancefortheIRsensingfromtheperspectiveofelectricalproperties.Severalfactorscouldcontributetothiseffect.Firstly,theHNBworksasachargedparticle,whichwillgenerateasmallelectricaleldaroundtheCNT-metalcontactandthereforechangestheI-Vrelationship.Onthe36otherhand,duetothemechanicalassemblyoftheHNBantenna,thiscanalsobeattributedtothechangeofthepositionoftheCNTand/orthecontactbetweentheCNTandelectrode.2.5NanoscaleIRSensorCharacterization2.5.1SensorsandMeasurementMethodInCNTbasedinfraredphotodetectors,themostbasicstructureisCNT-metalschottkydiode,inwhichtheCNTisalignedbetweentwoelectrodesbyquantitativelycontrolleddielectrophoretic(DEP)assembly[90].AsshowninFigure2.15(top),theSEMimageshowsAu-CNT-Austruc-tureanditsdimensiononSiO2,wheretheCNTlengthisaround6mm.Therearethreeareasfromlefttoright(L,MandR):leftAu-CNT(1mm),CNTconnection(4mm)andrightCNT-Au(1mm).InthisAu-CNT-Ausymmetrystructure,bothphotoconductanceandphotovoltaiceffecthavepossibilitytoleadphotocurrentindeviceandthedominateeffectdependsontheCNTareaandCNT-metalcontacttype,showninTable2.1.Atleftandrightside,theAu-CNT/CNT-Aucon-tactcanintroducebothphotoconductanceandphotovoltaiceffectforIRresponsewhilethecenterCNT(Marea)onlygeneratephotoconductance.Inordertocertifytheforemostphotoresponsesource,areliabletestingsystemisrequiredtomeasureCNTIRsensorresponse.Thecharacteri-zationofnanoscalephotodetectorisahugechallengeduetoopticsdiffractionlimit[91]andthecommercializedinfraredlaserbeamspotsizecanonlynarrowto10mmscalewhichisthousandtimesofsinglewallCNTdiameter[61].InFigure2.15(bottom),itshowstherelativesizebe-tweenlaserbeam(WorldStarTechUH5-100G-808,100mW808nmsinglemodelasermodule)andCNTafterfocus,inwhichthebrightwhiteovalshapeshowsthelaserbeamandthecenterblackdotrepresentstheCNTunderlaser.TheCNTlengthisaroundtenthoflaserbeamdiame-ter.ItishardlytoknowthelocalizationofCNTandlaserbeaminthison.Therefore,37Table2.1Au-CNT-Austructureanditsphotoresponse.SideNamePossiblePhotoresponseAreaLPhotoconductance,PhotovoltaicAreaMPhotoconductanceAreaRPhotoconductance,Photovoltaicin[31][92][34],itjustshowedthemaximumphotocurrentwithestimatedinputirradianceenergy.Figure2.15Top:SEMimageofAu-CNT-Austructure.Bottom:TherelativesizebetweenCNTdetectorandIRlaserbeamspot.Intheproposedtestingbench(Figure2.16(a)andFigure2.16(b)),adigitalmicroscope(KeyenceVHX-600)andpreciseveaxissubstage(KleindiekEucentricFiveAxisTable)areusedtolocal-izeCNTphotodetector.ThelongworkingdistanceofdigitalmicroscopeleavesthespaceforIRirradiationondetector.Bothmicroscopeandlaserarelockedonnon-movablepositionandthemicroscopemonitorstheheight(zdirection)andposition(xandydirection)ofCNTdetector.ThenanoscaleCNTdeviceismovedonprecisesubstageandachargeintegrationreadoutcircuitmeasuresthecurrentwinginit.TheinsetatleftcornerofFigure2.16(a)showstherelative38Table2.2EucentricveaxistableItemsDimensionsL:72mm-W:50mm-H:44mmTravel(xandy)10mmTravel(z)3mmTravel(R)360degTravel(T)90degAbsoluteaccuracyT<0.2degRepeatablityT<0.03degLinearresolution<0.5nmRotationalresolution<6x106deg(107rad)Speedupto1mm/sResolution<0.5nmpositionofdetectorandlaserspotonstage.Figure2.15(b)ismorepreciseaboutthedimension.Figure2.16(b)istheexperimentalsetupusinglaser,substageanddigitalmicroscope.Table2.2liststheofeucentricveaxis(x,y,z,RandT),inwhichtheresolution,speedandtravelrangeisaccurateenoughforrepositioningdevice.Inordertoleveragethedetectorinhorizontal,therearefourmarkers(perpendicular`L'shapeanditsmirrorimage)designedonsubstrate,asshowninFigure2.16(c),M1-M4.Duringcalibra-tionprocess,thedetectorismovedbystagecontrollertofocusmarkerrespectively.Thisprocessistillfourmarkersareinfocuswhenitismovedtothecenterofmicroscopewithoutchangingzdirection.Inthemeasurement,thecenterpointofdigitalmicroscope(viamicroscopedisplayscreen)shouldkeepfocuswhenthedetectorismovingwiththesubstageonx-yplane.Themeasurementtrajectoryisaround1mmstepsize,asshowninFigure2.16(d)andthecenterofphotodetectorwillbemovedonthispathway.Themeasurementprocessincludesx-axisandy-axisscanning.InordertosetthecenterfocusedcorrespondingpointunderthesameIRirradiation,thefocuspoint/lineisalwaysinthecenterofdigitalmicroscopenomatterwherethedetectoris.AsshowninFigure2.17,therearethreepointsselectedfromdevice(A,BandC)andthreeimagingpoint(A0,B0andC0)formedonimagesensorafterobjectivelensandtubelens.The39clearfocusedpointsfrommicroscopeareinthesameheight.However,whenthedetectorplaneisnotperpendicularwithsymmetryaxisofmicroscope(pointA1pointC1lightpath),thereisonlyonenarrowline(pointB1)focusedasshowninFigure2.17right.AsinFigure2.16proposedmeasurementsystem,thedigitalmicroscopeisnotverticalondetectorplane,theremustbeonelinefocusedwhenmovingthephotodetectoronx-yplane.Bycontrollingthisfocuslineinthemiddleofmicroscopeimageonydirection,thedetectorwillbemovedonthesamehorizontallevel.Figure2.16a)Proposedtestingbenchusingdigitalmicroscope,laserandveaxissubstage.b)Hardwaresetup,insetissubstage.c)Fourpointscalibrationmarkerfordetector.d)Rasterscan-ning:experimentalmeasurementpathwayforcentroidofphotodetector.2.5.2ExperimentalResultsInthissection,CNTIRsensorresponseonx-axis,y-axis,biasvoltageandcontactlengtharediscussed.ThemeasurementisconductedafteraligningCNTphotodetectoralongx-axis,40Figure2.17Focusedandunfocusedlightraysondigitalmicroscope.inwhichthemaximumresponsecanreachto28nA,showninFigure2.18.Thephotocurrentdecreaseswhenmovingthedetectorupanddown(y>40mmandy<40mm).Thephotocurrent(Ip)relationonliney(=20,30,40,50,60mm)isorderedasEq.(2.9)andEq.(2.10)onitscorrespondingpoint.Thelinewithmaximumresponseiscloseonthecenterinydirection(y=40mm).Ip;y=40mm>Ip;y=30mm>Ip;y=20mm(2.9)Ip;y=40mm>Ip;y=50mm>Ip;y=60mm(2.10)InFigure2.18,therearethreeareas,includingpositiveresponsearea,negativeresponseareaandtheunknownareabetweenthesetwo.Thephotocurrentisonpositivedirection,negativedirectionandunstablerespectively.Inoverall,therearetwooppositedirectioncurrentsourcesinthedevicewhenIRirradiatesondetector,becauseitgeneratespositiveandnegativephotocurrentwithzerobiasonit.ThiscouldbeexplainedbytwoSchottkybarrierformedbyAu-CNT(left)andCNT-Au(right).Whenthedetectorisonleftsideoflaserspot,theCNT-AuSchottkydiodedominatestheresponse,whiletheAu-CNTSchottkydiodegeneratesmorephotocurrentonrightside.Themaximumresponsedoesnothappeninthecenterlinebutwithalittleoffsetonbothside.41AsshowninFigure2.18,itlocatesaround32mmatleftand55mmatrighthalf.Therefore,thephotoconductanceeffectcannotdominatethephotoresponsebecausethemaximumIRirradiationisonthecenterduetoitssinglemodegaussianbeam.Themaximumphotocurrentshouldbeoncenterifphotoconductancecontributesthemost.Inthemeasurement,themaximumphotocurrentisgeneratedonthelargeslopeareaofgaussianlaserbeam,wherethetwodiodeswillstayindistinctareaswithlargeIRenergydifference.ThephotodetectoronhigherpowerIRirradiationwilloutputmorecurrentafterneutralizingsmallpartofchargeswithanotherside.Intheunknownareabetweenpositiveandnegativeresponse,thephotocurrentisnoisyandunstable.Itjumpsfrompositivesidetonegativesideorreversely.Thereasonisthatgaussianbeamwillproducenearlyuniformoutputincenterarea.Thetwocurrentinfacingphotodetectorswillbecanceledbyeachothersothatitisveryhardtogenerateastablephotocurrent.AsshowninFigure2.18,themoreunknownareahappensthesensorisclosertocenter,whichisconsistentwithgaussiandistribution.Figure2.19showsthephotoresponsealongydirection.Thephotocurrentispositivewhenthedetectorislocatedonleftside,whileitgeneratesnegativeresponseonrightside.Meanwhile,thephotocurrentdecreaseswhenthesensorgoesfurtherleft(fromx=33mmtox=30mm)orfurtherright(fromx=48mmtox=51mm).Allthephotoresonsearesymmetryalongy=40mmduetothatthedetectordirectionisperpendiculartomovingtrajectoryandthelaserbeamoutputissymmetryalongy=40mm.ThephotocurrentisproportionaltothedifferenceofIRirradiationbetweenAu-CNTcontact(left)andCNT-Au(right)contact.However,thephotocurrentcurveisnotfullysymmetryatx=32mmandx=33mmbecausetheprecisesubstagechangesthepositionat1mmstepsizebutwithmeasurementpositionerrors.Ifphotoconductancedominatesthephotoresponse,eachlineshouldbegaussiancurveastheresultthatthephotocurrentdirectlythelaseroutputdistributiononydirection.42Figure2.18Photocurrentmeasurementalongxdirectionwithdistincty.Ininfraredphotodetector,theofmeritincludesNETD,responsivityetc[93].Responsiv-ityisreferredtophotosensitivitywhichisrelatedtoquantumefy(thenumberofelectronsreleasedperincidentphoton).Whennoiseisnotamainconsideration,thephotosensitivitycanbecalculatedbytheoutput(voltageorcurrent)perwattofincidentenergy,showninEq.(2.11),whereR:Responsivity,[V/W],S:SignalOutput,[V]or[A],P:Incidentenergy,[W/cm2]andA:Detectoractivearea,[cm2].TheresponsivitycanbecalculatedbylocalizingCNTIRphotodetec-torduetothattheIRirradianceisgaussiandistributiondependentonx-yposition.Inthisresearch,theproposeddetectormaximumresponsivitycanreachto16.8mA/mW.R=SPA(2.11)43Figure2.19Photocurrentmeasurementalongydirectionwithdistinctx.Thenanodeviceisverysensitivetoexternalenvironmentalchanges,e.g.chemical[94],elec-tricalsignals[95].IntheCNTphotodetectorcharacterization,thebiasvoltageissweepsignalfrom-10mVto10mVasshowninFigure2.20andthedarkcurrentisfrom-160nAto160nA,whichisalmostlinearcurveduetosmallvoltagerange.However,thephotocurrent(maximumvalue)isalwayswithin16.51nAandthephotoconductancehasnoeffectontotalcurrentinsensor.SincetheSchottkybarrieriscriticalinCNTbasedphotodetector,themetalmaterialsandcon-Table2.3CNTmetalcontactlengthandthedirectionofoutputphotocurrent.LeftLength(mm)RightLength(mm)Photoresponse81.5Singledirection61.8bi-direction3.51.2Singledirection44Figure2.20Photoresponseanddarkcurrentondifferentbiasvoltage.tactareawillplayakeyroleinoutput.TheCuandAgmetalCNTcontactarealsocharacterized.AgroupoftenAu-CNT-Au,Ag-CNT-AgandCu-CNT-CuSchottkydiodebasedIRphotodetectorsaremeasuredwithIRirradiation(100mW808nm).AsshowninFigure2.21,theAu-CNTcontacthasmoreresponsethanAgandCuinaverageduetoitshighworkfunction(Au:5.1eV,Ag:4.26eV,Cu:4.7eV).Thecontactlengthisalsocharacterizedbydifferentsize,showninTable2.3.Theandthirddetectoronlyrespondtoonesideandthe6/1.8mmdetectorhaspositiveandnegativecurrent.The1.8mmsideonlygeneratesaround1nAscalephotocurrentbut6mmsideproduceabout40nAresponse(Allthemeasurementsarefrommaximumpoint).TheexperimentalresultsindicatethatphotovoltaicdominatesphotoresponseonCNT-metalSchottkydetectoralthoughthephotovoltaicvoltagecannotbesampledduetoitstinyvaluemergedinnoise.ThedetectorIRresponsearedependentonCNT-metalcontactsizeandmetalworkfunction.2.6ChapterSummaryThestabilityandreliabilityofCNTbasedIRsensorwereanalyzedandpresentedfullyinthisstudy.InthebyanalyzingtheparasiticcapacitanceinCNTmetalSchottkybarrier,anovelstructurewasproposed.Thesensoronlow-kpolyimideinterfaciallayerhasaround100nAphotocurrentat45Figure2.21PhotocurrentcomparisononAu-CNT,Cu-CNTandAg-CNT.830nm3000mW=cm2irradiation.Theparasiticcapacitancereducesthesensorresponsewhilein-creasingsubstratesurfacevoltage.Itdominatesnanoelectricaldeviceperformance,nanomaterialsensorswhenthedevicefeaturesizeshrinksintosub-microornanoscale.Secondly,anoveliso-lationelectricalfeedbacksystemwasintroducedintoDEPsystem.Bymeasuringtheimpedancechanges,thesystemcanquantitativelycountthenumberofcarbonnanotubesbridgedbetweentwoelectrodes.Theexperimentalresultsshowthatthesystemresponsespeedisfasttosinglewallandmulti-wallCNTs,althoughtheimpedancediffersmuch.Thissystemwillalsoapplicabletothinlayergraphenedepositioncontrolandothernanomaterialsdepositionandlocalization.Thirdly,therobusttestbenchusingdigitalmicroscopeandpreciseveaxissubstageisusedtomeasuredetectorphotoresponse.TherelativepositionbetweennanoscalesensorandIRbeamislocal-izedbymappingthephotocurrentonlaserspot.Thedistancebetweenphotodetectorandinfraredlaserlensisleveragedbydigitalmicroscope.Theexperimentalresultsshowphotovoltaicquan-tumeffectdominatesCNT-MetalSchottkybasedIRdetectorandthephotoresponseisdependentoncontactsizeandmetalmaterials.Ourproposedmeasurementmethodprovidesarobustandpreciseapproachtocharacterizesub-microandnanoscalephotodetectorwhichisimportantforsub-wavelengthscalephotodetectorcharacterization.46Chapter3SinglePixelInfraredCamera3.1PreviousWorkOverthepastfewyears,therehavebeenconsiderablebreakthroughsinthethermalimagermarketincludingthefactthatpriceshavedroppedconsiderably.Withtheadvancesintechnologyandmaterials,infraredcamerasarelybeingdesignedfortheenduser.Regardingtothecost,therewasnonewfully-featuredimagerlessthan$20;000USD.However,therearemanyfantasticchoicesoutthereforbuildingapplicationswithawiderangeoffeaturesinpricesrangingfrom$2;000to$9;000USD.Ascomplexassomesystemsmayseem,infraredcamerasarecomprisedofsomebasiccomponents:lens,detector,processingelectronics,display,controlsandpowersupply.Somefeaturessuchasthermalsensitivityanddetectorsizeareusefulinevaluatingperformance.ThermalSensitivity:ThisisthemostimportanttoevaluateIRcamera.Thether-malimagerisabletoresolvetemperaturedifferencesatleast0.1(100mK)orlower.Thesmallernumberindicatesthebetter(i.e.moresensitive)ofthesystem.Inoverall,ahandfulof40-50mK(0:040:05)systemisalsoavailabletoprovidefantasticimagequalityandclarity.Thelowersensitivitiesarecapableofdiscerningsmallertemperaturevariationstypicallyencounteredinmarginalinspectionconditions(whentheinsidetooutsidewallsurfacetemperaturedifferenceislow).Inotherwords,theadditionalcostofimprovedsensitivityisaninvestmentthatcanhaverealreturns.DetectorArraySize:Mostinfraredsensorarrayshavefewerpixelsthanvisible-lightcameras.47IRimagersavailableforthecivilianmarketarealongwayfromthe5-8megapixelvisualarrayswhichareusedtoseeingonmostsmartphonecameras.However,morepixelsgenerallymeansgreaterdetailandhigherresolutioninfraredcamerascanmeasuresmallertargetsfromfurtherawayandcreatesharperthermalimages,addinguptomorepreciseandreliablemeasurements.Excellentinfraredsystemsforcivilianapplicationarenowbeingmadewith120120(14,400),160120(19,200)and320240(76,800)focalplanearrays(FPAs).TheFPAssmallerthan120120,thoughattractive,don'tprovidesufspatialresolution.Oncontrary,theFPAslargerthan320240,suchas640480(307,200),produceanimpressiveimagebutcostmore.Inaddition,itmustbeawareofthedifferencebetweendetectorspatialresolutionanddisplayresolution.ImageDisplay:Ahigh-qualityLCDdisplayscreenisessentialtodiagnosinganimage.Theremustbethedisplayresolutionandsensorarrayresolution.Inproductdescription,somemanufacturersboastaboutahighresolutionLCDandhidetheirlowresolutiondetector.Forin-stance,LCDresolutionmayspecat640480,butiftheIRdetectorpixelresolutionisonly160120,or19,200pixels,thegreaterdisplayresolutionaccomplishesabsolutelynothing.Thequalityofthethermalimageanditsmeasurementdataarealwaysdeterminedbythedetectorres-olution.FrameRate:9Hzsystemshavebecomewidelyavailableandworkjustaswellas30Hzand60Hzsystems.However,thehigherframeratehasthebetterabilitytorenderandcapturemovingtargets.Lowerframeratesarelesstoleranttomovementandwillblurtheimageifcrossingascenetooquickly.Althoughthereisanimportantconsiderationforindustrialthermographerswhoareinspectingcertaintypesofrotatingequipment(motorshafts,bearingsorcouplings),itisfarlessofaconcerninbuildingapplicationswherethetargetsarestationary.Thethermalinfrared(IR)camerathatattachestoasmartphone(FLIROne/SeekThermal)is48nowavailable,whichbringsinfraredtechnologyintoconsumerelectronics.Thenovel$250SeekThermalinfraredcameraevaluatesitseffectivenessinhelpingwildlandlingeringsmolderingareasduringthemopupstageofsuppression.FLIRONEisanotherlightweighteasilyconnectanduseinsmartphone.Itexploresthenaturalworldwithnoadditionalcords,cases,devicesorscreens.TheIR-BlueisanaffordablethermalimagingaccessoryforiPhoneandAn-droiddevices,whichusesa64zone164pixelnon-contactinfraredsensorarraytoreadthetemperatureinviewingandconnectsusingbluetoothtoiPhoneorAndroiddevicetoshowthetemperaturereadingascolors.Thenovelinfraredcamerasaremostlyinresearchlab.AsinglepixelIRcamerawasproposedin[69],wherethecamerasystemusedasingleCNTphotodetectortocompressivelysamplethelinearprojectionoftheimageontobinaryrandompatterns.Byem-ployingcompressivesensingalgorithm,highresolutionimagecanbeachievedwithfewersamplesthanoriginalimagedimension.In2006,compressivesensingbasednewdigitalimage/videocameradirectlyacquiresrandomprojectionsofascenewithoutcollectingthepixels/voxels[96].Thecameraarchitectureemploysadigitalmicromirrorarray(DMD)toopticallysamplelinearprojectionsofthesceneontopseudorandombinarypatterns.Thekeyhallmarkisitsabilitytoobtainanimageorvideowithasingledetectionelement(fisinglepixelfl)whilemeasuringthescenefewertimesthanthenumberofpixels/voxels.Sincethecamerareliesonasinglephotodetector,itisalsoadaptedtoimageatwavelengthswhereconventionalCCDandCMOSimagersareblind.493.2SpatialLightModulatorbasedImager3.2.1CompressiveSensingTheconventionalNyquist-Shannonsamplingtheoremrequiresthesufsamplingrateatleasttwiceofsignalfrequencyinorderforfullyreconstructionguaranteed.Thecompressivesensing,anewcomputingparadigmdirectlysamplesthesignalincompressedformsothatthesamplingratecanbesreduced,whichhasattractedextremelyinterestinimaging[54][69],geophys-icaldataanalysis[97],controlandrobotics[98],communication[99],medicalimagingprocessingincludingMRI,CT[100].Incompressivesensing,thereisnoneedtodesignsensorswithhigherbandwidththanoriginalsignalstofollowandcapture[101].Itcanbeseenasasumofthelinearprojectionfromoriginalsignaltomeasurementmatrix.Thecompressivesensingisacombinedsamplingandcompressionprocess,whichisthemostefwaytosamplesignalsfromsingleprocessingpointofview.E.Candes,etc.gavethemathematicproofofusingrandommeasure-mentmatrixtorecovertheoriginalsignalbysolvingminimizationofthe`0and`1optimizationproblem[57].Givenanunknownsignalx(x2RN),compressivesensingtakesMtimeslinearmeasurementsfrommeasurementmatrixtooriginalsignalx,asshowninEq.(3.1).IfM=N(Nisthedimensionofunknownsignalx),thesignalxcouldbeeasilyreconstructedbysolvinglinearequationorelseitwouldbeanunderdeterminedquestion.However,theoriginalsignalcanalsobereconstructedwithlessmeasurements(M<<N)usingcompressivesensing,whereFisaMNmeasurementmatrix,whichtransformsthemeasurementasalinearprojectionfromoriginalsignaltomeasurementmatrix.yisthemeasurementresult.Theoptimalsolutionforxcouldbereconstructed[102]bysolvingtheminimizationofthe1-normoptimization.Morediscussionandcomprehensiveintroductiononcompressivesensingcanbeseenon[103][104].50y=Fx(3.1)ThecoreofcompressivesensingisthesolverofproblemofEq.(3.1).ItisNP-hardunfortu-nately.Iny=Fx.F(F2Rmn)isthemeasurementmatrix(m<n).InsteadofsolvingtheNPharddirectly,Donoho[104]provesthatifxissparseandFisundersomeconditions,suchasthenullspaceproperty[105],theincoherencecondition[106]andrestrictedisometryproperty[55],theEq.(3.1)willbeequivalenttoEq.(3.2).Thenatureimage(x)sparsitywillbediscussedinChapter5.Ÿx=argminkxk1s.t.Fx=y(3.2)Alotofalgorithmshavebeenproposedtosolvethis`1problem,includingOrthogonalMatch-ingPursuit(OMP)[107],IterativeShrinkage-Thresholding(IST)[108],CoSaMP[109],subspacepursuit[110],AcceleratedProximalGradient(APG)[111],andAlternatingDirectionMethod(ADM)[112]anditslinearizedversion(LADM)[113].3.2.2SinglePixelImagerThestructureoftheIRsensingsystemisbuiltuponthecompressivesensingtheoryintroducedintheprevioussection.Figure3.1depictsthesystemsetupusingasinglepixelCNTphotodetectorasimagesensor,inwhichIRimagesaredirectedontoaDMDthroughasetoflenses.TheIRimagesontheDMDrepresentstheoriginalsignalxinEq.(3.1)(x2Rn,nisthenumberofimagepixels).TheDMDgeneratespatternsaccordingtoameasurementmatrixF(F2Rmn)soastocompresstheIRimages.EachpatternontheDMDiscomprisedbynpixels,thustherearem51differentpatternsformthemeasurementmatrixF.ThecompressedIRimagesareandfocusedtoaCNTphotodetector.TheIRsignalarrivingattheCNTphotodetectorrepresentsthelinearprojectionoftheimageontothemeasurementmatrix,whichcouldbeconsideredastheinnerproductofIRimagexandeachrowvectorinthemeasurementmatrixF.Thephotocurrentisrecordedbyafastreadoutsystemintegratingwithachargeintegrator,anAnalog-to-DigitalConverter(ADC),andaDataAcquisition(DAQ)card.TheamplitudeofphotocurrentrepresentsthevalueofyinEq.(3.1)(y2Rm,misthenumberofmeasurements).Itshouldbenotedthatcompressivesensingsetsm<n,therefore,theIRimagexiscompressivelysampledintothey.BasedonthemeasurementresultsyandthedesignedmeasurementmatrixF,theoriginalsignalxcanberecoveredusinganimagereconstructionalgorithm.3.3WeakSignalReadoutMethodThereadoutcircuitisoneofthekeycomponentsinimagingsystem.Withthelimitationofnanomaterialelectrontransport,thephotocurrentisfrompAtonAscaleinCNTinfraredsen-sor[32][33].Moreimportantly,itisbiasdependent.InordertoreadphotocurrentinCNTinfraredsensor,alownoise,highgainreadoutcircuitisneeded,inwhichafullycurrentreadoutsystemincludesacurrenttovoltagemodule,andaDigitalSignalProcessingmodule(DSP).3.3.1CurrenttoVoltageConversionMethodTherearetwoclassesofCNTphotodetectorsignalmonitoring,voltageandcurrentbased.Theformerrequiresgreatvoltagedifference(atleastmVscale)onsensorwhenlightirradiatesonit,especiallyforphotovoltaicdevice.However,theIRirradiationisnonlineartovoltagegeneratedinquantumeffectinfraredsensorbutlinearproportionaltophotocurrent.Meanwhile,inCNT52Figure3.1SystemsetupofcompressivesensingbasedimagingsystemusingaCNTphotodetector,responseenhancedbyphotoniccavity.53Figure3.2SchematicofRtypeIVconverter.detectorexperiments[114],CNTphotovoltaiceffectisnotassiliconphotodiode.ItgeneratesnAscalephotocurrentwhilethevoltagedoesn'tchangemuch.Therefore,acurrenttovoltageconversionwillbedesignedforCNTphotocurrentmeasurement.Thecurrentmonitoringrequiresthereadoutcircuitpresentzeroinputimpedancetothedetector,andabsorbthedetector'scurrentwithoutproducingabiasvoltage.Sincethephotocurrenthighlydependsontheappliedbias,itisdesiredtousezerobiasforcurrentmonitoringonCNTdetector.3.3.2ResistorbasedCurrentReadoutMethodInordertoconvertpA=nAphotocurrenttovoltage,acurrenttovoltage(transimpedancethemostimportantmoduleinreadoutsystem.Thebasiccurrenttovoltageconverter(IVconverter)istousephotocurrent(Ip)multipliedbyaresistor.Thesecircuits,showninFigure3.2,needverylarge(gigaohms)andpreciseresistortodetectpAcurrent[115].Thethermalnoiseofthelargeresistorwillalsolimittheresolutionofthecircuit,althoughthedeepnegativefeedback(OPA1)setCNTdetectoratzerobiasinthiscondition.3.3.3CapacitorBasedCurrentReadoutMethodCapacitiveTrans-Impedance(CTIA)alsoworksasacurrenttovoltageconverteranditcanbedesignedtohavehighchargetovoltageconversionratiowithlownoise.Figure3.3is54Figure3.3SchematicofCtypeIVconverter.thebasiccircuitofCTIAusingswitchcapacitorasadjustresistor,inwhichtherearetwoworkingphases.Inphase1,switchSisON,OPA2isresettheoutputaszero.Inphase2,switchSisOFFinToffandthephotocurrent(Ip)willchargenegativenode(-)ofOPA2toredistributeit,whereQ=IpToff,Vo2=Q=C0,(ToffrepresentsswitchSofftime).Inthisdesign,thecircuitcandetectultralowcurrentbasedonthenoiselevelininput.ItalsoachievesalargePowerSupplyRejectionRatio(PSRR),highopenloopgainandlargedynamicrange.TheCTIAgainisgivenbyG=Q=C0.3.4ROICStructureforCNTIRSensor3.4.1ZerobiasReadoutCircuitsAsdiscussedinprevioussection,therearetworequirementsonreadoutcircuits.Oneisthehighlyresolutionstopicoamperescaleandanotherisnobiasorbiasmodulatedonthedetector.InordertoreducenoiseandreadsuchlowcurrentfromMWCNTdetector,aCTIA,showninFigure3.3wasdesignedinthisIRreadoutsystem.SincetheoffsetofinputgeneratesaseriousbiasonCNTIRsensorduetoinputasymmetry,intheproposeddesign,adummyCNTIRsensorwillbeconnectedonpositiveinputofOPA,asshowninFigure3.4,suchthattheinputimpedancematchwillbein55Figure3.4Zerobiasreadoutcircuit.sameorder.3.4.2HighGainCurrenttoCurrentConverterInordertogethighspeedCNTbasedIRsensorreadoutsystem,thecurrentneedsbeforeconvertingtovoltage.Inthispart,aproposedcurrentreadoutmethodforcarbonnanotubeinfraredsensorwasdescribedherein,showninFigure3.5.Therearefourelectrodes,inwhichleftpartisCNTsensor,whilerightpartisconventionalBipolarJunctionTransistor(BJT)structure.Inthiscon-theCNTsensorisconnectedtothebaseofBJT,sothatthephotocurrentwillwtothebasenodeunderconditionofinfraredirradiation.ThebasecurrentcouldbebyBJTworkingprinciple[116].3.4.3HighSpeedReadoutInhighresolutionADC,delta-sigma(DS,orsigma-delta,SD)modulationisamethodforencodinghighresolutionsignalsintolowerresolutionusingpulse-densitymodulation.Ahighperformance,24-bitSDanalog-to-digitalconverterwasusedinexperiments.Itcombinedwideinputbandwidth56Figure3.5CurrenttocurrentconverterforCNTIRsensor.andhighspeedSDconversionwithaperformanceof106dBSNRat625kSPS,makingitidealforhighspeeddataacquisition.Widedynamicrange,combiningwithreducedanti-aliasingrequirements,thedesignprocess.Inaddition,thedeviceoffersprogrammabledecimationratesanddigitalFIR.ItisidealforapplicationsdemandinghighSNRwithoutacomplexfrontendsignalprocessingdesign.3.5HardwareExperimentalPerformanceandApplications3.5.1ReadoutSystemTestingInthissection,thehardwaresetupandsomeexperimentalresultsarepresented.ForthesakeofreducingthenoiseafterCTIA,aLowPassFilter(LPF)wasintroducedtolimitthebandwidthandoptimizethethermalnoisecontributiononCTIA.Meanwhile,itweakensallhighfrequencynoise,thepowerfrequencynoiseincluded.Figure3.6istheexperimentaldiagramofreadoutsystem.Basedonthisreadoutsystem,agroupofexperimentsweredesignedtoverifythereadoutperformance.AsshowninFigure3.7,itmeasuresthelinearityandstabilityofROIC,wherex-axisiscorrespondingtolaserintensity.ThedigitaloutputatT=1hkeepscloseasvalueatT=57Figure3.6Diagramofreadoutsystem.Figure3.7ReadoutlinearitytestonCNTbasedIRdetector.0withlinearresponse.Meanwhile,thereadoutoutputwasalsocomparedwithAgilent4155CSemiconductorAnalyzer(refertoCurveTracer)measurementinFigure3.8.TheROICoutputfollowsCurveTracerverywell.TheresultsshowthatthereadoutsystemcanreachpAresolutionwithhighstabilityperformance.3.5.2ReadoutApplicationsIntheCNTbasedsinglepixelIRcamera,itintegratedthepreciselyweaksignalreadoutsystemtoacamera.ThehardwaresetupwasshowninFigure3.9,whereIRirradiationwascontrolledbyDigitalMicromirrorDevice(DMD).Basedonthehardwaresetup,threeexperimentswererealized,showninFigure3.10.Arectanglebarwasmovedfromtoptobottomandtherecoveryimagecouldfollowtargetimageverywell.Invisually,therearetwofeaturesintherestoredimage,noisyandpartialrecoveryimage.Thenoiseoriginatesfromsensorandsamplingprocess.58Figure3.8Readoutcomparisonbetweenproposedsystemandsemiconductorparameteranalyzer(Agilent4155c).Meanwhile,theresultsalsoshowthatthelowerleftcornerisdestructive(partialimage)duetosuchtinysensingarea.Fromtheseresults,itisfoundthatthereadoutsystemworkperfectlyinCNTbasedIRsingleimagingsystem.3.6ChapterSummaryNano-photodetectorshavedemonstratedpromisingperformancetodetectIRsignals,especiallyCNTphotodetectors.However,asmallabsorptionareaandthediftofabricatealargescalephotodetectorarrayimpedeitsapplicationinimagingsystems.Inordertoovercometheseproblems,acompressivesensingbasedIRcamerasystemwasdeveloped.Firstly,theultrahighspeed,highresolutionreadoutsystemwaspresentedinthisstudy.ItcouldtestthecurrenttosubnAinCNTbasedIRdetectorwithlownoisesoastointegrateitinCNTbasedsinglepixelIRimagingsystem.Theexperimentalresultsshowthatthenumberofmeasurementsrequiredtorecovertheimagescanbemuchfewerthanthepixelnumberoforiginalimages.Thiscamerasystemwascapableofobservingthedynamicmovementofalaserspotinnearinfraredwavelength.Ingeneral,thecompressivesensingmethodmightmakethenanosensorbasedinfraredcameraachievable.59Figure3.9HardwaresetupofsinglepixelIRimagingsystem.60Figure3.10ImagesrecoverybasedonsingleCNTdetector.61Chapter4LightFieldImaging4.1PreviousWorkTheconventionalphotographsonlyrecordthesumtotaloflightraysforeachpointonimageplanesothattheytelllittleabouttheamountoflighttravelingalongindividualrays.Thefocusandlensaberrationproblemshavechallengedphotographerssincetheverybeginning,therefore,lightphotographywasproposedtosolvetheseproblems.Ingeometricoptics,thefundamentalcarrieroflightisarayanditisfocusedonincoherentlightwithobjectsfarlargerthanthewavelength.Theradianceisrepresentedtomeasuretheamountoflightwhichisinterpretedasasince1846[117].In1936,Gershunintroducedthislightterminology,thesystematicalphys-icaltheoryoflight[118],whichthelightasafunctionofpositionanddirectioninregionsofspacefreeofoccluders.Theplenopticfunctionmodeledthelightusingsevenparameters,showninEq.(4.1)ingeneral[59].P=P(x;y;l;t;Vx;Vy;Vz)(4.1)P=P(x;y;u;v)(4.2)Theraysinspacecanbeparameterizedbythreecoordinatesx,y,andzandtwoanglesqandfforanystationary,quasimonochromaticlightshowninFigure4.2.The5Dradiancefunctiondescribestheappearanceofscenethroughalllightrays(2D)emittedfrom3Dpoint.Itis62Figure4.1Concaveobjectradiance(left)andconvexobjectradiance(right).Figure4.2Parameterizinglightrayin3Dspacebyposition(x,y,z)anddirection(q,f).ave-dimensionalfunction,thatisafunctionoverave-dimensionalmanifoldequivalenttotheproductof3Deuclideanspaceandthe2-sphere.The5Dplenopticfunctionisnotpracticalsinceitisimpossibletocaptureviewsbypointtopoint.However,itiscanbereducedtofourdimensionsbytwoassumptions.Firstly,theregionofinterestisrestrictedtolocationsoutsideofconvexhulloftheobjectbecausethelightleavingfromonepointmayendbyanotherpointonaconcaveobject,asshowninFigure4.1.Secondly,itisreducedasfourparameterswiththeassumptionofrayspassingthroughfreespaceregionsandfreeofoccluders,suchasopaqueobjects,andscatteringmedia,suchasfog.Undertheseassumption,thelighttravelingisconstantalongitspathway,eliminatingonedimensionofvariation[60].Basedon4Dlightmodel,therearemanylightphotographyimplementationspro-posedinliteratures.Recently,lenslet-basedlightsystemhasbeenintegratedintodigitalcam-eras[119].Lightmodulatinginmask-basedsystemshaveevolvedtobemorelightefthan63pinholearrays[120].Nevertheless,bothapproachesceimageresolutionbecausethenumberofsensorpixelsistheupperlimitofthenumberoflightrayscaptured.Toavoidthislimitation,alternativedesignhasbeenproposedwhichfavorsspatialresolutionoverangularresolution[121].Inordertofullypreserveimageresolution,currentoptionsincludecameraarraysortakingmul-tiplephotographsbyasinglecamera[122].Inoverall,thetimesequentialapproachesarelimitedtostaticscenessincehighspeedcameraarraysarecostlyandbulky.Theideaofcompressivelightacquisitionitselfisnotneweither.In[123],itsimulatesacompressivecameraarrayforlightsensing.Recently,researchershavestartedtoexplorecompressivelightacquisitionus-ingasinglecamera,suchasopticalcodingstrategiesincludingcodedaperturesandcodedlenslets.Itisacombinationofcodedmask,apertureandrandommirrorUnfortunately,theyrequiremultipleimagestoberecordedsuchthattheycan'tcapturedynamicscenes,thoughtheysucceedinreducingthenumberofshotscomparedtononcompressivecounterparts.Lightcameraisnotonlyactiveinresearchlab,itisalsoverypopularinbusinessprospects.LytrowasfoundedbyStanforduniversitycomputergraphicslaboratoryalumnusRenNgtocom-mercializethelightcamera.Theydevelopedaconsumerlightdigitalcameracapableofcapturingimagesusingaplenoptictechnique.Raytrixisanotherlightcameracompanywhichhassoldseveralmodelsofplenopticcameraforindustrialandapplicationssince2010,withresolutionsstartingfromonemegapixel.PelicanImaginghasthinmulti-cameraarraysystemsintendingforconsumerelectronics,whichusefrom4to16closelyspacedmicro-camerasinsteadofamicro-lensarrayimagesensor.Meanwhile,NokiahasinvestedinPelicanImagingtoproduceaplenopticcamerasystemwith16-lensarraycameratobeimplementedinNokiasmartphones.TheAdobelightcameraisaprototype100-megapixelwhichtakesathree-dimensionalphotoofthesceneinfocususing19uniquelylenses.Eachlenswilltakea5.2-megapixelphotooftheentirescene.TheCAFADIScameraisanotherplenopticcameradevelopedbythe64UniversityofLaLaguna(Spain).CAFADISstands(inSpanish)forphase-distancecamera.Sinceitworksfordistanceandopticalwavefrontestimation,theproductproducesseveralimagesre-focusedatdifferentdistances,depthmapping,all-in-focusimagesandstereopairsfromasingleshot.Asimilaropticaldesigncanbeusedinadaptiveopticsinastrophysicssoastocorrecttheaberrationscausedbyatmosphericturbulenceintelescopeimages.Inordertoacceleratethesetasks,differentalgorithmsrunningonGPUandFPGA,operateontherawimagescapturedbythecamera.MitsubishiElectricResearchLaboratories's(MERL)lightcameraisbasedontheprincipleofopticalheterodyningandusesaprinted(mask)placedclosetothesensor.Anyhand-heldcameracanbeconvertedintoalightcamerausingthistechnologybysimplyin-sertingalow-costontopofthesensor.Thismask-baseddesignavoidstheproblemoflossofresolution,sinceahigh-resolutionphotocanbegeneratedforthefocusedpartsofthescene.However,thecoreofalllightcamerasinlaborcommercializedsystemarelargevisiblesensorarray.TheseapproachesarenotapplicabletoIRspectruminoverall.AnovelcompressivesamplingapproachwillbeproposedinthisresearchasitisnecessaryforIRcameratocapturemoreinformationthansimple2Dcamera.Inthisthesis,thelightimagingisanarrowlynameoflightomittedtimeandwavelength(orstationary,quasimonochromaticlight),asshowninEq.(4.2),alsoreferredtoplenopticcamera[124]orsyntheticaperturecamera[125].4.24DLightFieldModel4.2.1LightFieldModelinLensThe4Dlightasradiancealongraysinemptyspace,wasyproposedinformofthelight[126]andthelumigraph[127].Inthismodel,theobserverandthescenecanbeseparatedbyasurfacesothattheradianceisrepresentedbyafunctionoflightrayspassing65throughit.Themodelisnamedtwoplaneparameterization,asshowninFigure4.3,wheretheplane(s;t)and(u;v)settherelativecoordinatesforlightraypassingthrough.The(u;v)-planeistheviewpointplaneinwhichallcamerafocalpointsareplacedonregulargridpointswhilethe(s;t)-planeisthefocalplane.Inordertocreatesucha4Dplenopticmodelforrealscenes,alargenumberofviewsaretaken.Bythisway,theyarediscretesamplingoftheplenopticfunction.Thiscouldbeenhancedascontinuouslightbyassumptionoftherealobjectstobelambertian.Anypointoftheobjecthastheisotropicradianceinallpossibledirectionssothatthelightrayscanbefullyrepresentedbyinterpolatingalgorithms.AsshowninFigure4.3,theradiancecanberepresentedbyf(s;t;u;v)foranyrayspassingbetween(s;t)and(u;v).Thepairofpointsbetweenstplaneanduvplanecorrespondonlytooneray.Thedisadvantageoftwoplaneparameterizationisthatthemethodcannotrepresentraysparalleltothetwoparallelplanes.However,asshowninFigure4.4intherealcamerasamplingsystem,thelensplaneandsensorareassignedasuvandstplanedistinctly,becauseeveryraythatcontributestoaphotographymustpassthroughthelensandterminatessomewhereontheItalsoexplainsthecameralimitslightrayswithinAngleOfView(AOV).InFigure4.5,f(s0;u0),f(s0;u1),f(s0;u2)showthelightrayspassingthroughu0,u1andu2.Theu((u;v))isthedirectionalaxisbecausetheuinterceptingonthelensdeterminesthedirectionatwhichtherayswillbecollectedbysensors.Thes((s;t))referstothespatialaxis.Ingeneral,itcanbemappedtoapointontheray-spacediagramonFigure4.5foranyraysindiagramFigure4.4.Inconventionalcamera,thefocuswillbeadjustedbychangingthedistancebetweentheandthelens.Therayspacewillalsobechangedasthecamerafocusonthedifferentdepth.AsshowninFigure4.6(top),whenmovingthesensorarrayplaneclosertothelens,theraysfromselectedpointwillcorrespondtoalinewithpositiveslope,showninFigure4.7(left)andallraysfromsamepointmustbeonthispositiveslopelineinray-spacediagram.Onthecontrary,it66Figure4.3TwoplaneparameterizationforlightFigure4.4TwoplaneparameterizationinSLRcamera.67Figure4.5TwoplaneparameterizationinCartesiancoordinates.68Figure4.6Lightraydiagramofcamera(unfocused).willbeanegativeslopelinewhenmovingthefurtherfromthelens,asshowninFigure4.7(right).Theslopeontheray-spacedependsontheseparationbetweentherealimageplaneandplane[60].4.2.2LightFieldModelinMirrorInordertogetfullylightinformationofscene,thereisatleast4Dspacetosupportallvaluesasf(s;t;u;v).IntheDMDbasedlightsampling,italsorequirestwoplaneparametersto69Figure4.7RaysinCartesiancoordinates(unfocused).characterizelightrays.Inthemirroritisaduplicationofanobjectthatappearsidenticalbutmirrored.Ingeometry,themirrorimageofanobjectortwo-dimensionalisavirtualimageformedbyspecularItisthesamesizeastheoriginalobjectexcepttheobjectorhassymmetry(alsoknownasaP-symmetry).inamirroralsoresultsinachangeofchirality,morefromaright-handedtoaleft-handedcoordinatesystem(orviceversa).Asaconsequence,ifonelooksinamirrorandletstwoaxes(up-downandfront-back)coincidewiththoseinthemirror,thiswillgiveareversalofthethirdaxis(left-right).Themirrorimageappearstobethree-dimensionaliftheobservermovesorusesbinocularvision.Althoughthemirrorchangesthelightdistributioninthehalfspaceinfrontandbehindit,themirrorimagedoesnotviolatetheconservationofenergyasthemirrorsimplyredirectsthelight.Intermsofthelightdistribution,theDMDonlychangesthedirectionoflightsstoppedonitssurfacesothatthemirrorplanecouldbeoneinternalplaneoflightmodeling,asshowninFigure4.8.ThesecondplanewouldbeselectedinfrontofDMDanditisthemask(aperture)plane.ThroughthemaskandDMDplane,alllightrayspassingwithinthesetwoobjectscanbemodeled.70Figure4.8TwoplaneparameterizationinDMDbasedimagingsystem.4.3MaskbasedSinglePixelLightFieldSensing4.3.1OpticsandSystemDesignAsshowninFigure4.9,thebasicsinglepixelcameraiscomposedofmainlens,digitalmicromir-rordevice(DMD,workingasspatiallightmodulator),secondlensforconverginglighttosinglesensor,andIRsensor.Aftermainlens,thesceneordesiredobjectisprojectedontoDMDarraytoformavirtualimagebehind,showninFigure4.9inset.TheDMDcanbecontrolledindividuallyandDMDpatternsaredecidedbymeasurementmatrix.Throughmultiplepatterns,itwillgener-ateaserialofmeasurements,inwhicheachmirrorcorrespondstopartofthedesiredobjectandcontributestoonepixelinrecoveryimage.Thekeyopticalcomponentinthislightimagingsystemisapertureonmask,whichcontrolsthelightraydirections.AsshowninFigure4.10,mul-tipleangularimagesarecapturedinsinglepixelcamerainordertogetalldirectionalrays.TherearetwoapertureholesofA1andA2,whichgeneratetwodistinctrecoveryimagedi1anddi2inimagespace.Theyrepresenttwodistinctangularinformation(groupoflightrays)fromscene.Thewholelightrayscanberecordedthroughmultipleaperturepositions.71Figure4.9Schematicdiagramofsinglepixellightsensing.Figure4.10Distinctangularimagefromtwoaperture.72Figure4.11Syntheticapertureimaging.Thefocusplaneisbecomingfarawaytomainlensfromlefttoright.4.3.2ExperimentalPerformanceThemostpopularapplicationoflightcameraistostudythesyntheticaperturephotography,onwhichthefocusisadjustableusingasingleshot.Intheexperimentalsetup,therearetwoaircraftslocatedintwodistinctplanesalongthemainlens,oneclosetothelensandanotherisfurtherawayfromlens.Duringsampling,25angular(directional)imagesarereconstructed.Foreachangular,itwillsensedistinctraysfromsameobject.TheplaneandparallaxmethodwasusedtoshowrefocusofsinglepixelIRimaging,showninFigure4.11.Theleftshowsthatcameraworksinnearfocusonthebottomaircraftwhilethefaraircraftwillbecomeclearwhenthefocusplanemovesaway.Meanwhile,bothobjectswillhavekindofblurwhenfocusingbetweenthem.4.4DoubleCompressiveLightFieldSensingThelightsensingarecharacterizedbyspacialresolutionandangularresolutionthoughitistrade-offinconventionaldesign.Thisproblemcouldbesolvedbymultiplecamerasdesign.However,themorecamerasintroducemoredifopticsdesignandlargerspace.Incompressivelightsampling,itisobviousthatspatialsamplingissparsityassinglepixelcamera[128].Theangularinformationcouldalsobesparsitysincethedifferenceisrelativelysmallbetweenadjacentangularimages.Inthisresearch,doublecompressivelightsensingincludesthebasicspacial73andangularsensing.ThespatialcompressivesensingisdiscussedinChapter3.Thesecondcompressiveregimeisappliedforangularimagedifference.Bytakingtheangularimageasfullrecovery,thesystemonlysamplesthedifferencebetweensecondandangularimage.Inthefollowingrecoveryalgorithm,italsoonlyreconstructsthedifferenceandthewholesecondreconstructedimageiscombinedbythisdifferencewithangularimage.4.4.1ModelingofDoubleCompressiveLightFieldSensingThelightismodeledas4Dinformation,including2Dspacialimagesand2Dangularvalues.Thelightsamplingisdescribedas2Dimagesalong2Dangularvaluessothattheangularimagescouldbemodeledasseries,showninFigure4.12.Eachindividualangularimageisasinglesamplingandrecoveredbysinglepixelimageralgorithm.Alongtheangularaxis,thereisalsosomeredundantinsomeway.Thedifferencebetweentwoadjacentangularimagesarequitesparseinspatialdomain,asshowninFigure4.13,inwhichtheintensityisingrayscaleandthenonzerovalueschanges)areshowninFigure4.13(b).Basedonthedifferencesignal,anothercompressivesensing,referredtodoublecompressivesensing,canbeappliedonnextangularimagereconstruction.4.4.2RecoveryAlgorithmAsdiscussedinChapter3,`1minimizationismostpopularincompressivesensingduetoitis`0equivalenceundersomeconditions.However,TotalVariation(TV)regularizationmakestherecoveredimagequalitysharperandpreservestheedgesorboundariesmoreaccurately.Itnotonlyreconstructssparsesignalsorimagesbutalsosucceedswhenthegradientoftheunderlyingsignalorimageissparse.ThereisonlyalimitednumberofTVsolversavailable,includingSOCP[129],74Figure4.1255angularimagesofStanfordjellybeans.`1-Magic[57],TwIST[130]andRecPF[131].Inthisresearch,TVAL3[128]isusedfordoublecompressiverecovery.TheTVregularizationsolvestheproblemofEq.(4.3),whereŸxisestimatedvalue,x2Rst,Dxisthediscretegradientofx,Fisthemeasurementmatrix(randomBernoulli).yistheobservationofxviasomelinearmeasurements.Itcanbeeither1-norm(correspondingtotheanisotropicTV)or2-norm(correspondingtotheisotropicTV).Ÿx=argminkDxks.t.Fx=y(4.3)75Figure4.13Adjacentangularimagedifference.a)intensitydifference,b)changes(nonzerochanges)ofangularimage.InTVAL3algorithm,insteadofemployingtheaugmentedLagrangianmethodtominimizetheTVmodelEq.(4.3)directly,anequivalentvariantofEq.(4.4)isconsideredanditsunconstrainedcorrespondingaugmentedLagrangianfunctionisEq.(4.5).Supposethatx(l)andw(l)irespectivelydenotetheapproximateminimizersofEq.(4.5)atthe(l)iterationwhichreferstotheinnerit-erationwhilesolvingthesubproblem.Thealternatingdirectionmethodsolvestwosubproblemrespectively.76minwi;xåikwik;s.t.Fx=yandDix=wiforalli:(4.4)minwi;xjA(wi;x)=åi(kwikvTi(Dixwi)+bi2kDixwik22)lT(Fxy)+m2kFxyk22(4.5)4.4.3ExperimentswithDoubleCompressiveSensingInthissection,thedoublecompressivesensingnumericalresultsareanalyzed.Firstly,thebasiccompressivesensingisappliedonangularimagerecovery.Thesamplingratiomustbegreaterthan30%soastogethighimagequality,asshowninFigure4.14(top),whilethebottomlineshowsrecoveryimagesusingdifferencerespectively.Thoughitishardlytodistinguishthedifferenceinvisually,itismuchclearinFigure4.15wheretwomethodsarecomparedusingsamemeasure-ments.Thedoublecompressivesensingcanreducesamplingratiodownto9%withhighimagequalitywhilebasiccompressivesensingrequiresashighas25%.Figure4.14Angularimagerecoverycomparisonbetweenbasiccompressivesensinganddoublecompressivesensing.ThesecondsimulationisbasedonStanfordJellyBeans(httpThereare100(1010)imagesselectedandthecolumnischosenasreferenceimageineach77Figure4.15AngularimagerecoveryresidualerrorbyRMSEandPSNR.row.AsshowninFigure4.16,thereare10rows,5columns(50angularimages)recovered.Theestimationofnextangularimagecomesfromcurrentimageandthereconstructeddifference.Sincetheresidualerrorissuchsmallandimagespatialresolutionisonly9696,itishardtodiscoverdifferenceinvisual.Table4.1liststheRMSEandPSNRforeachimagecomparedtogroundtruth,whereangularIDrepresentstheimagerow/column(0102:01row02column).Inoverall,theresidualerrorisstillwithin30dBinveimagesaccordingtoPSNR.78Table4.1CharacterizingangularimagesaccumulationresidualerrorbyPSNRandRMSE.AngularID0102010301040105PSNR(dB)40.236734.707432.563431.2847RMSE1.70422.87353.25243.5392AngularID0202020302040205PSNR(dB)42.401735.251932.916331.5164RMSE1.30742.74023.18393.5185AngularID0302030303040305PSNR(dB)40.53534.358732.73831.1188RMSE1.58262.95463.2073.5282AngularID0402040304040405PSNR(dB)40.556934.642232.66531.3414RMSE1.6962.95513.29563.5862AngularID0502050305040505PSNR(dB)41.974634.98833.118731.3126RMSE1.35782.83853.21093.6197AngularID0602060306040605PSNR(dB)41.239934.165232.2831.073RMSE1.52563.13133.50523.7688AngularID0702070307040705PSNR(dB)41.586434.489932.398230.3772RMSE1.42092.93793.36593.7864AngularID0802080308040805PSNR(dB)40.459934.546932.118330.7085RMSE1.53412.88663.32933.6208AngularID0902090309040905PSNR(dB)41.996234.498832.522130.8233RMSE1.35242.90753.30433.6575AngularID1002100310041005PSNR(dB)39.931534.122632.310230.9432RMSE1.77863.06973.45783.7069794.5ChaptersummaryInthischapter,thelightsensingisdiscussedandcomparedwithconventionalimaging.Asinglepixelinfraredcamerasystembasedlightsensingisproposedtocaptureinfraredlightrays.Bytakingmultipleangularimagesthroughcodedaperture,wecanusesingleCNTIRsensortoreconstructfull4Dlightfromlargeobject.ThesyntheticaperturephotographywillremovetheIRfocusproblemaway.ThisbringsbroadlyapplicationinvirtualrealityThedoublecompressivesensingreducessamplingrateusingtheredundantofangularimagedifference.Theexperimentalresultsshowthatthemoreangularimageswillachievehigherangularresolution,betterrefocusinsyntheticaperturephotographyforsinglepixelIRcameras..80Figure4.16Angularimagerecoveryfromdoublecompressivesensing,thecolumnisreferenceimageandtheotherfourarerestoreddependsonitsleft.81Chapter53-DImaging5.1PreviousWorkThreedimensionalinfraredcamerasarepotentiallyusedinautonomousvehicles,robotmanufac-turers,securityindustrialandvideogamemanufacturers.Alloftheseindustriesuseeithersomesortofinfraredimagingsystemordepthcalculation.Especiallywiththedevelopmentofautonomousvehicleandadvancedrobotindustry,thethreedimensionalinfraredcamerabecomesmoreandmoreimportant.Autonomousvehicles,includingselfdrivingcars,useseveraldifferentmethodsofunderstandingtheirsurroundings[132].Ingeneral,therearetwocurrenttechniquestoachieveathree-dimensionalimageofanenvironmentinmarket.Theisrangedistancecameraswhichusebruteforcecomputationtocalculatetheimagebasedonsinglepointscanningsystem[133].Itwillproducea2Dimageshowingthedistancetopointsinascenefromapoint.Therefore,theproblemwiththissystemisthatitisverydiftoobtainvideorateimag-inginadynamicenvironmentduetothecomputationintensityrequiredtocollectallthedataforasingleimage.Besides,theyareunabletodistinguishwhatsomethingtheydetectis.Anythingofaclosesize(adeeroraparkedcar)lookssimilarinthoseimagingsystems.Stereophotographyisanother3Dcamera,whichusesstereoscopicvisionsystem,attemptingtooverlaytheimagescollectedbythetwocamerastocreateathree-dimensionalimage.Itmakesthecamerasimulatehumanbinocularvision[134]tocapturethree-dimensionalscene.However,thistechniqueislessthanidealduetoproblemswithcameraplacementorinadvertentmovement,calibrationandother82dif[135].Inaddition,thesesystemsarelargeandexpensive.Duetothemanufacturinglimitationoflargeinfraredsensorarray,thereisverylittle3DIRcamerabasedonbinocularvision.Inthe3Dcameramarket,itdidhavesome3Dvisiblelightcameras,whichcomeintothreedifferenttimeofcameras,projectedtexturestereovisioncamerasandphaseshiftcameras.TimeofFlight(ToF)isoneofeffectivewaystomapthedistancefor3Dimaging.In[136][137],thesystemcombinedToFandstandardRGBcameratodoreal-time3Dsceneaug-mentationwithvirtualobjectssothatboth3Dgeometryandcolorinformationarematched.Fromonshelfmarket,timeofcamerasarethemostcommon,whicharebytheSR4000fromMesaImaging.It'slargelyusedinindustrialinspectionprocesses.However,itshouldbenotedthatthenear-IR(orvisible)lightToFcameraiswiththepriceapproximately$10;000.Pro-jectedtexturestereovisioncamerasareexamplesof3DvisiblecamerasandrepresentedbytheEnsensoN10.ThiscamerahashigherresolutionthanToFandiscapableofprovid-ingvideoimages.PhaseshiftcamerarequirestwoCMOSsensorarrays,whichincreasecalibrationdif.TheyarerepresentedbytheFinepixReal3Dcamera,acommerciallyavailablesystemcapableoftaking10megapixelimage.Inoverall,allthreecameras'aredesignedtoworkwithvisiblelightratherthaninfraredandarethereforeincapableofoperatinginthemid-IRspectrumorlongerwavelength.Thereareatleastthreefollowingfactorsconsideredwhenselectinga3Dcamera,includingoperatinginhighlydynamicenvironments,cameraresolutionandprice.Itisundoubtedlythattheinfraredisbetterthanvisiblecameraforhighlydynamicenvironments,becausetheinfraredsensingcanvisiblenoiseindaytimeandworksatnights.However,theinfrared3Dcam-eraneedshighperformanceIRsensor,whichintroducesadditionaldifsstemmingfromacoolingrequirement.Itrequirescryogeniccoolingtoreducethebackgroundnoiseespeciallyformiddlewavespectrum.Therefore,the3DIRcameraisnotportable.83Inthisthesis,thelowdimensionalmaterialbasedIRsensorandimagingmethodwereproposedtosolvetheseproblems.Intheproposal,acombinationofdistancemeasurementandstereocamerawillallowasystemtomeasure3DcoordinatesandphotorealistictextureatavideoframerateintheIRspectrum.Thecompressivesensingcombinestheadvantagesofthesingle-pointtechniquewiththeadvantagesofthestereoscopic,e.g.ithasasingle-pointdetectorsoitdoesnotrequirethedifcalibrationasthestereoscopictechnique.Besides,itismuchlessintensivecomputingthancurrentsingle-pointmethodasitusesacompressivesensing.Assuch,itispossibletocapturevideo-rate3Dimagesthatareunobtainablewiththecurrentgenerationofsingle-pointsystems.Acriticalrequirementincompressivesensingimagingisthatthedetectorhastoswitchmultipletimesinsteadofasingleshot.Especiallyforahighresolutionimage,adetectorisrequiredtoswitchtensofthousandstimestoobtainanimage.ThisrequirementhasmadeitextremelydiftoapplycompressivesensingontoIRimaging.However,thenanomaterialbasedinfraredphotodetectorprovidesthesolutiontotheseproblems.5.2TimeofFlight3DImaging5.2.1WorkingPrincipleTheToFmethodcomesfromdepthmeasurementtechniques.Itreferstothepassiveprocessofmeasuringthedepthofascenebyquantifyingthechangesthatanemittedlightsignalencounterswhenitbouncesbackfromobjectsinascene.AsshowninFigure5.1,a3Dcameraworksbyilluminatingthescenewithamodulatedinfraredlightsourceandobservesthedistancetoa3Dobjectbymeasuringtheabsolutetimeofwhichalightpulseneedstotravelfromasourceintothe3DsceneandbackafterSincethespeedoflightisconstantandknown,(c=3108m=s),thisisalsocalledphaseshiftmethodwhichmeasuresthephaseshiftbetweenthe84Figure5.13Dcameraoperationprinciple.illuminationandtheInordertoimprovemeasurementaccuracy,theilluminationistypicallyfromasolidstatelaseroraLEDoperatinginthenear-infraredrange(850nm)invisibletothehumaneyesandworksinshortbandwidth[138].Thereceiversensorisdesignedtorespondtothesamespectrumasirradiationwhichconvertsthephotonicenergytoelectricalcurrent.However,theambientlightalsocontributestoobjectsothatthedepthaccuracywillgoworse(lowersignaltonoiseratio)withhigherambientcomponent.5.2.2TimeofFlightModelingandApplicationThephaseshiftdeterminesthedistancefromdesiredobject.InordertodetectphaseshiftbetweentheilluminationsourceandthethelightsourceispulsedormodulatedbyaContinuous-Wave(CW)[138].Themodulationwouldbeasinusoidorsquarewave.Thelatterismorecommonbecauseitcanbeeasilyrealizedusingdigitalcircuits.Therearetwowaystorecordtime.Theistointegratephotoelectronsfromthelightandanotheristostartafastcounteratthedetectionofthewhichrequiresaultrafastphotodetector.AsshowninFigure5.2,thephasedifferenceiscalculatedbyEq.(5.1)fromtherelationbetweentwodifferentelectriccharge(left)orfourdifferentelectriccharge(right).Thephasecontrolsignalshave90degreedelayfromeachother.Therefore,thedistancecanbecalculatedbyEq.(5.2).85DDepth=c2Dj2pf(5.1)d=12cDt(Q2Q1+Q2)(5.2)Oncontrary,thecontinuouswavemodulationmethodretrievesphaseshiftbydemodulationofreceivedsignal,whichcrossescorrelationofreceivedsignalwithemittedsignal.Foranemittedsi-nusoidalsignalg(t)withmodulationfrequencyw,thereceivedsignalcouldbes(t)afterfrom3Dsurface,showninEq.(5.5)andEq.(5.6),(b:constantbias,a:amplitude,f:phaseshift).Thecrosscorrelationofg(t)*s(t)isc(t)inEq.(5.7).Bysamplingc(t)atfoursequentialinstantswith90degreephaseoffsettogetQi=c((i1)p=2);i=1;:::;4,thefourresultingelectricchargecanbeusedtoestimatethephasedifferenceDjasEq.(5.3),wherethedistanceinEq.(5.4),cisthespeedoflightinconstant.ItalsodirectlyobtainstheparametersaandbinEq.(5.8).Dj=arctan(Q3Q4Q1Q2)(5.3)d=c2fDj2p(5.4)g(t)=cos(wt)(5.5)s(t)=b+acos(wt+f)(5.6)c(t)=sg=Z+¥¥s(t)g(t+t)dt=a2cos(wt+f)+b(5.7)a=p(Q4Q2)2+(Q3Q1)22b=Q1+Q2+Q3+Q44(5.8)86Figure5.2Twomethods:pulsed(left)andcontinuouswave(right).InCW3Ddepthmeasurement,theamplitude(a)isafunctionoftheopticalpower.Theoffset(b)isafunctionoftheambientlightandresidualsystemoffset.Thehighamplitude,highmodulationfrequencyandhighmodulationcontrastwillincreaseaccuracywhilehighoffsetcanleadtosaturationandreduceaccuracy.Athighfrequency,themodulationcontrastwillattenu-ateduetothephysicalpropertyofthesilicon,whichputsapracticalupperlimitonthemodulationspeed.TheadvantagesofpulsedmodulationToFmethodisdirectmeasurement,wheretheirra-diationlightpulseenergyistensorderhigherthanbackgroundilluminationtoreducebackgroundnoise.Mostimportantly,thedirectionofilluminationandobservationarecollinear.Themostobviousdrawbackislongtimemeasurementduetoscanningpointbypoint.Mean-while,thelightpulseinaccuracycomesfromlightscatteringondistinctsurface.Fromahardwareperspective,itisdiftogenerateshortlightpulseswithfastrise/falltimeandlowrepetitionrate.Thecontinuouswavemodulationisapplicabletovarietyoflightsourcesduetonoshortandstrongpulserequired.Besides,thelongintegrationtimeintroducesmotionblurandlimitstheframerates.ThemeasurementaccuracyisdependentonthepoweroftheemittedIRsourceandthematerialoftargetsurface.875.3StereoVision3DImaging5.3.1WorkingPrincipleThestereovisionimagingcomesfromhumanbinocularvisionwhichrequirestwoormorelenseswithseparatesensor.Thebinocularvisionhasdistinctadvantagescomparedtosingleeye.Itgivesprecisedepthperceptionbytwoeyesandallowsacreaturetoseemoreof,orallof,anobjectbehindanobstacle[139].Withtwoeyes,therewouldbeoverlappingofvisionfromsamescene,whichintroducesaslightlydifferentviewpointbetweenit.Asaresult,binocularvisionprovidesdepthinformationbycalculation.Inhumaneyeimaging,thedifferencesisprovidedasbinoculardisparitytobrainsoastocalculatethedepthofvisualscene,includingthenatureofthestimulusandbrainprocess.Inbinocularvision,theretinaldisparity,ortheseparationbetweenobjectsasseenbyleftandrighteye,arekeyvaluestoevaluatedepth.AsshowninFigure5.3,thedistancebetweentwoeyesisalmostalways6.5cmforadult.Theobjectimageinlefteyeandrighteyeisslightlydifferent.Thisretinaldisparitywillprovidetherelativedepthofobjectduetonoreferraldistance.Itmightbearguedthatpeoplecouldestimatethedistancebyeyes.Thisisbecausethereisacalibrationinbrainwhenseeingtheobject.Sincethe3Dimageishighlydependentontheimagedifference,itisveryeasytounderstandthatthecloserobjectsgeneratethesmallerretinaldisparity[140].5.3.2StereoVisionModelingandApplicationStereovisiongenerallyusestwocamerasseparatedbyadistance,inaphysicalarrangementsimilartothehumaneyes.AsshowninFigure5.4,theleftshowsthesimplepinholecameramodelandthisisalsothebasisoflasertriangulationandradardistancemeasurement.Theaandbarecomputedbytwodistinctimages.Thedepthzcanbecalculated,asEq.(5.9).88Figure5.3Retinaldisparityfromeyes.Figure5.4Stereopsisdepththroughdisparitymeasurement(left)andstereovisionsys-tem(right).89z=x1tana+1tanb(5.9)Figure5.4rightshowsthediagramofastereovisionsetup.Thebothcamerasaremountedexactlyparalleltoeachother,andwiththeexactsamefocallength,wherex:Thebaseline,orthedistancebetweentwocameras.f:Thefocallengthofcamera.P:TherealworldpointbythecoordinatesX,YandZ.uL:TheprojectionofPinimageplanebyleftcamera.uR:TheprojectionofPinimageplanebyrightcamera.Themodellocatesin2Dcartesiancoordinates(xA,zA)wherethedistancebetweentwocamerasarex,thepointP(X,Y,Z)areatuLanduRinimageplanerespectively.Byacquiringtwodistinctimages,theX-coordinatesofthepointuLanduRcanbegiveninEq.(5.10).ThedisparityreferstothedistancebetweenuLanduR.ItisobviouslytocalculatedepthbyEq.(5.11),thoughanactualstereovisionset-upismorecomplex.Thesamefundamentalprinciplesstillapplyanyhow.uL=fX=ZuR=f(Xb)=Z(5.10)DDepth=fbuLuR=fb=z(5.11)90Themajorchallengeofstereovisionisthecorrespondencecalibrationwhichinvolvescom-plex,computationallyintensivealgorithmsforfeatureextractionandmatching.ComparedwithToFmethod,thestereovisionimplementationcostisverylow,asmostcommonoff-the-shelfcamerascanbeused.Thistechniqueisintuitivepresentationtohumanssothatbothhumanandmachinearelookingatthesameimages.Instereovision,theerrorofdepthresolutionisaquadraticfunctionofthedistance.However,thestereovisioninfrared3Dcameraisveryuncommonduetoprice,resolution,temperaturelimitation.Inthisthesis,acompressive3Dimagingsamplingwasproposedforinfraredstereocameradesign.5.4Compressive3DImaging5.4.1Sparsityin3-DImageThesinglepixelcamerasystemhasbeenintroducedbasedoncompressivesensingtoovercomethecurrentlimitationandchallengeinmanufacturinglargescalephotosensorarrays[54].Itismoreattractivebyusingalowdimensionalmaterialsbasedinfraredsensors.Compressive3DimagingwillbothhavetheadvantageofToFsensorandstereovisionmethod.Themostimportantissueistoredundantandduplicateinformationorsparsityin3Dimagesoasforcompressivesensing.Generally,itisprovedthatmostnaturaldigitalimagesaresparse[55].Forexample,thebackgroundofaimagemayhavemanypixelswiththesamecolorandtextureinformation.Muchofthisredundantinformationendsupbeingdiscardedduringthecompressionprocess,makingthesehighresolutioncamerasveryinefIncaseofanons-parsesignal,asparsityprocessisrequiredtotransformthenon-sparsesignalintoasparesignalbyusingsomespecialbasis,suchaswavelet,curveletandFourier[141],asshowninEq.(5.12),wheresisthesparserepresentationofthenon-sparsesignalinbasis,yistheobservation.91Figure5.5SparsityinDMDbased3Dsampling.x=Ysy=fYs(5.12)Indigitalmicromirrordevicebasedsinglepixelcamerasamplingsystem,thereisanotherre-dundantinformationduringsamplingexceptfortheimageitself.InChapter3,italreadydiscussedhowDMDworked.AsshowninFigure5.5,whentheobject3Dscene(x)locatesinfrontofDMD,therewillbetwodistinctimagesfromDMDfiONflstateandfiOFFflstate,x1andx2.Itcanbeseenasthetwoimagesofstereovision.Duringeachsamplingprocess,whentheDMDpatternisD,theC1andC2canbederivedasEq.(5.13).C2isdependentonC1foradesiredscenesothatthesamplingwouldberedundant.ByintroducingantherIRsensor,itwillgeneratesecondimageforstereovisionsystem.C1=DX1C2=(ID)X2(5.13)925.4.2Compressive3DSamplingTheprototype3Dcameraissiliconinfraredsensorbased.Ina2Dsinglepixelcamera,itwillconsistofagroupoflenses,DMDandphotodetector,alongwithsomeassociatedopticsandhard-ware.Therearetwoapproachesfordevelopinga3DcamerausingDMD,asshowninFigure5.6andFigure5.7.Inthedesign,thecamerawilldetectIRemissionfromtwodistinctangularmirrorsviatheDMDstatusfiONflandfiOFFfl.WhentheDMDisfiONfl,thelightwillbefocusedonsensorfiIflfromtheupperlens.WhentheDMDisfiOFFfl,theinfraredlightpassesthroughthelowerlensintosensorfiIIfl.ThedifferencebetweenDMDfiONflandfiOFFflstatusisanangularchangeof12degree.Thisway,lightincidenttosensorsfiIflandfiIIflwillprovidean-gularinformationabouttheobjectbeingimaged.Afterthelightiscapturedbythephotodetector,theinformationwstoareadoutcircuitandADC,whichwillconverteranalogphotocurrentintodigitalvalues.Theimagerecoveryisbasedonsampledvaluesandred-cyancolormodelisappliedtobuilduncalibratedstereoimage.Thereisanother3Dcameradesign,showninFigure5.7,namedasthemaskInthisopticsdesign,thereisonlyoneIRsensor,whileanelectricallycontrolledmaskisplacedbeforethedetectortogettwoangularimages.TheobjectwillbeprojectedontotheDMDplanethroughthemainlens.Whenthemaskopenstheupperwindow,onlythelightthatisincidenttoupperwindowwillbeletthroughtothedetector.Thisallowsthelighttoserveasoneofthetwoimagesrequiredtoobtaina3Dscene.Likewise,thesameprocesstakesplacewhenthelowerwindowopens,allowingthesystemtohavebothimagesnecessarytoreconstructa3Dimagefallingonasingledetector.Thefollowingreadoutcircuitsand3DmodelingwillbesameasFigure5.6.93Figure5.6Dualdetectors3Dimagingsystem.Figure5.7Maskbased3Dimagingsystem.945.4.3ExperimentswithPrototypeCameraTheexperimentalresultsdemonstratethegenerationofstereoscopicimagesusingthedesiredimagingsystem.Figure5.6depictstheillustrationofthe3Dimagingsystemwhichcaptures3Dscenes.TwophotodetectorsareemployedandtheirpositionsarealignedtothesignalfromDMD,sothatthereconstructedimageoneachdetectorrepresentsthedirectionalinforma-tion.Intheexperiment,aninfraredlightsource(830nm)producesilluminationonthetargetscene.Whenthesignalsareectedfromthedigitalmicromirrordevices,aseriesofphotocurrentsaregeneratedoneachdetectors,whichareacquiredbyahigh-speedreadoutcircuitry.Finally,imagesarereconstructedbasedonthecompressivesensingalgorithm.Asaresult,twoimagescanbeob-tainedfromtwodetectorssimultaneouslyandtheyareconsideredassub-apertureimagesfor3Dcomputation.Theresolutionoftherecoveredimagesis6464pixels,with30%samplingratio.Itisnotedthatthesetwosub-apertureimagesalsorepresenttwodifferentperspectiveviewsofthesourcesignal.Therefore,astereoscopicimagecanbecomputedandgeneratedasshowninFig-ure5.8.Therearetwonoisesources,includingsamplingnoiseandrecoveryerrorofcompressivesensing.5.5ChaptersummaryThedevelopmentofthesetwosystemssolvestheproblemsconfrontingthecurrentgenerationof3DIRsensors.Theseproblemsincludethecomputationtimeandcalibrationissuesassociatedwithcurrent3DimagesystemsandthecryogeniccoolingsystemrequiredbymostcurrentIRsensors.Thecompressivesensingalgorithmdevelopedwillbecapableofdecreasingthecomputationaltimerequiredforimagingtoapointwheretheimageitselfcanbeupdatedatavideoframerateof30Hz.Thealgorithmcanbeadjustedtoensurephotorealisticqualityofenvironmentsandenable95Figure5.83Dimagereconstructioninred/cyancolor.precisedepthcalculations.TheuseofCNTphotodetectoranditsassociatedsignalenhancementthataredevelopedinchapter2willeliminatetheneedforthecryogeniccoolingsystem.Toconclude,theproposed3Dimagingsystemisbasedontheintegrationofcompressivesensingandbinocularvision.Byintroducingtwophotodetectorsintothesystem,twosub-apertureimagescanbeobtainedfromtwodifferentperspectiveviews.Currentexperimentalresultsindicatedthatinfraredimagescanberecoveredbasedonthisprincipleandtheimagescanbeusedtocomputestereoscopicphotography.Itopensthedoortothe3Dimagingthroughinvestigatingthenewcameraarchitectureusingcompressivesensingandcomputational96Chapter6SuperResolutionImaging6.1PreviousWorkTheHighResolution(HR)imagingarerequiredandnecessaryformanyelectronicimagingappli-cationsbecauseHRimageoffersmoredetailsofobject,especiallyformedicalimagediagnosis,patternrecognitionincomputervision.Themostdirectlywaytoenhanceresolutionistoincreasethechipsize(togetherwithlenschanges)orreducethepixelsizetoincreasethenumberofpixelsperunitarea.Bothtechniquesaredependentonmicrofabricationmanufacturing.Theyarelim-itedbydevicefabrication,priceandsensorsensitivity.Thedisadvantageofformermethodistoincreasecapacitancewhichmakesthereadoutcircuitinlowchargetransferrate.Italsorequiresahighprecisionoptics[142].Forthelattersolution,reducingthepixelsizewillalsodecreasetheamountoflightcomingtoeachsensorsothattheshotnoisewilldegradetheimagequality.Theminimumactivepixelareaisaround40mm2fora0.35mmCMOSprocess[143].However,theresolutionalsocanbeenhancedusingsignalprocessing(referredtosuperresolution).Therecoveredhighresolutionimageisobtainedfrommultiplelowresolutionimagesobserved.AsshowninFigure6.1,theredpointsrepresentpixelcenterofreferenceLowResolution(LR)im-age.TheblueandgreenpointshowtwoLRimageswithsubpixelshiftforthesamescene.Thethreelowresolutionimagesaresubsampled(aliased)aswellasshiftedwithasubpixel.Inordertoincreaseresolution,theLRimagesmusthavesubpixelshiftsorelsethereisnonewinformationforimagereconstruction.Thiscouldbeobtainedfromonecamerawithseveralcapturesorfrom97Figure6.1Multipleimagessuperresolution.Figure6.2FourcausesofLRimageacquisition.multiplecameraslocatedatdifferentpositions.Inconventionalcamera(visible,infraredSLRcam-era),thereareatleastfourfactorsinducingLRimage,includingopticaldistortions(outoffocus,diffractionlimit,etc.),motionblurduetolimitedshutterspeed,noisethatoccurswithinthesensororduringsampling.AsshowninFigure6.2,therecordedimageusuallysuffersfromdistortion,blur,noise,andaliasingeffects.TherearetwomaincategoriesofSuperResolution(SR)method,includingmultipleimages(frames)andsingleimagebased.AsshowninFigure6.1,themultiframeSRreconstructionhaslongesthistoryanditinvolvesthreesteps,imageregistration,interpolationandrestoration.The98mostdifistoestimatethemotionofLRinputframescorrespondingtoreferenceframe,referredtoimageregistration.Thetypicalmethodistolookupforinterestpointsinthelow-resolutionimageset,thenuserobustmethodstoestimatethepointcorrespondencesandcomputehomographiesbetweenimages.In[144],theiterativemethodwasusedtoestimateregistrationpa-rameters,shiftsandrotation.Theblockmatchingisalsoappliedtoregisterinputimagesin[145].TheBayesianmethodsearchedacontinuousspaceofshiftandrotationtogetherwithconventionalMAPreconstructionalgorithmtoestimatethehighresolution.Thedenseandsceneilluminationchangesarealsousedtoestimatemotionofeachpixelin[146],becausethephotometricwillbechangedaswellwhensceneilluminationchanges.Thezoomanddefocusapproachesaredis-cussedin[147]and[148]tobuilduptheconstraintsforrecoveringsuperresolutionimage,wheretheregistrationandestimationcouldbeinajointframework.In[149],thejointMAPestima-tionalgorithmscapturethedependencebetweenLRimageregistrationandHRimageestimationalthoughitmayintroduceovproblem.Allaboveapproachesareexplicitmotionestima-tionwhichiscriticaltoSRreconstruction.Besides,thefuzzymotionestimationbasedonblockmatchingisalsousedindenoisingalgorithmin[150].Comparedtomultipleimagesbasedsuper-resolution,singleimagebasedismoreapplicablesincethereisonlyonelow-resolutionimagerequired,especiallyforportableapplications.Themoststraightforwardwayofsingleimagesuperresolutionisinterpolation.Thenearestneighborinterpolationmodeltheunknownpointbyitsnearestneighborpoint[151].ForeachpointontheHRgrid,theclosestknownLRpixelisselectedasthevalueatthepointinHRgrid.Thedisadvan-tageisjaggyeffectontheHRimages.Bicubicinterpolationutilizedacubickerneltointerpolatebutcreatedblureffect[152].Theedgedirectedinterpolationmethodispresentedin[153]forsuperresolution.Itestimatesthelocalcovariancecoeffromasinglelowresolutionimage,andappliesthesamecoeftoreconstructhigh-resolution.Thecontourlettransformandwavelet99basedlinearinterpolationisproposedin[154].Thedirectionalanddatafusionwereusedtoedgeguidednonlinearinterpolationtopreservesharpedgesandreduceringartifactsin[155].Theanothersingleimagebasedsuperresolutionisstatisticsmethods.Itutilizesstatisticaledgedependencyinformationasinlowandhighresolutionimagesin[156].Thelearningbasedmethodoperatesbybuildingamodelfromexample,learnsfromandmakespredictionsondata.Itisapowerfultoolforimagesuperresolution,althoughitrequirestwolargetrainingdatasets.Inordertogetsuperresolution,aofimagegradientwasdescribedastheshapeandsharpnessinrecoveryalgorithm[157].In[158],itproposedamethodviaanexample-basedstrategywhichdividedthehigh-frequencypatchesofalow-resolutionimageintodifferentclasses.Thesignalsparsityandlearningmethodswerecombinedtogetherin[159].ThelowresolutionimageswererepresentedinsparsedomainbycomputingcorrespondingcoefandthehighresolutionimagesaregeneratedviathesecoefInimagereconstruction,itrequireshighcomputationaleffort,evenforproblemofmoderatesize,especiallyforrealtimeprocessing.Thesuperresolutionbecomesmoreattractiveinsinglepixelimagingsystemduetotimecomplexityofcompressivesensing.Manyefalgorithmshavebeendevelopedforsolvingthisminimization,includinglinearizedBregmanmethod,runtime10sfor4096dataset[160],edpointcontinuationmethod,6.2sfor4096dataset[161],Bregmaniterativemethod[162].Althoughafastcoordinatedescentmethodwasproposedin[163],of0.94sfor4096dataset,itwasspecializedforsparsitydata,howeveritmightnotapplicabletoimagerecovery.FourierdomaincomputationwasintroducedtoreplaceiterativelinearsolversinMRIreconstructionvia`1minimization,whichcost11.5storeconstructof256256imageinMatlabon1.2GHzlaptopcomputerin[164].Theproblemwasthattheruntimewouldincreaseto346sfor10241024uterusimage.AnotherapproachistogetfastrecoveryviahardwareimprovementonFPGA[165].Itin-100tegratesorthogonalmatchingpursuit(OMP)andapproximatemessagepassing(AMP)inFPGAtoget0.63msfor3232imageblockrecovery.Duetohardwarelimited,itisstillhardtodo10241024imagesize.TheTotalVariation(TV)normimagerestorationmakestherecoveredimagequalitysharperandmoreaccuratelybypreservingtheedgesorboundaries.Itworkswhenthegradientoftheimageissparse.MostimagescanbewellapproximatedbyTVnormsothattherearehugeTVminimizationmethodsproposed.Theweaknessofthesealgorithmsarestilleithertoosloworlessrobustcomparedwith`1minimizationalgorithm,especiallyforlargeimagesize[128].Inordertoproduceimagewithsuperresolution,highimagequalityinsinglepixelcamerasystem,itwillcostmorethan10sfor256256imagein1.4GHzdesktopusingTVL3algorithm.Moreover,theruntimewillincreasedramaticallywhenimagesizegoeshigh.Thecombinedsuperresolutionandcompressivesensingattractsmoreinterestingininfraredspectrumthanvisibleimage.ThehigherresolutionIRimagesmeansbetterabilitytospotsmalltargetsatlongerdistances.ItishardlytodirectlyreconstructhighresolutionIRimagefromcom-pressivesensing,suchthatthesignalprocessingapproachwouldbebestoptionforIRimagesuperresolution.6.2ObservationModelingThedigitalimagingsystemsuffersformhardwarelimitations,acquiringimageswithvariouskindsofdegradations,asshowninFigure6.2,includingtheopticaldistortion,themotionblurduetoaperturetime,thesensornoise.Finallytheframescapturedbythelowresolutionimagingsystemareblurred,aliasing,andnoisyversionoftheunderlyingtruescene.101Yk=DkBkMkX+nk;k=1;2;:::;KX=[x1;x2;:::;xN](6.1)AsshowninEq.(6.1),X=[x1;x2;:::;xN]denotesthedesiredhighresolutionimage,sampledaboveNyquistsamplingfrequency.YkisthekthLRimagewithsubpixelshiftfromreferenceLRimage(k=1).Allvectorsarerepresentedinlexicographicalorder,andifM=1,itissingleimagesuperresolutionobservationmodel.MkdescribesthemotioninformationforkthLRimage.BkistheblurmodelseffectandDkisthedownsamplingoperator.nkrepresentsthenoisemodel.ThelinearequationisinEq.(6.2).ThekthLRimageisdenotedasY(s;t);(s=1;2;:::N1;t=1;2;:::N2),N1;N2arehorizontalandverticaldirectionresolutionofLRimages.TheparametersL1;L2aredownsamplingfactorsforeachdirectionrespectively(N=L1N1L2N2).Inmathematically,themotionwarpmatrixsizeisL1N1L2N2L1N1L2N2,blurmatrixBkisL1N1L2N2L1N1L2N2,whilesubsamplingmatrixDkis(N1N2)2L1N1L2N2size.266666666664Y1Y2...YK377777777775=266666666664D1B1M1D2B2M2...DKBKMK377777777775X+266666666664n1n2...nK377777777775(6.2)AsshowninEq.(6.1)andEq.(6.2),themotionwarpmatrixrepresentstheglobalorlocaltranslationandrotation.ThiscanbeestimatedbythedifferencebetweenreferenceandparticularLRimages.ItisaninterpolationmethodwhenthefractionalunitofmotionisnotequaltotheHRsensorgrid.Theblurmatrixareveryhardwarerelated,whichdependsonopticalsystem,focus,diffractionlimit,aberration.Itisusuallyaknownvalue.ThedownsamplingmatrixDkmodelsthe102aliasinginLRimage.Thesethreematrixarecorrelatedtosuperresolutionprocessclosely.6.3MultipleImagesbasedSuperResolution6.3.1NonuniformInterpolationApproachCurrentlymostofthenon-uniforminterpolationbasedsuperresolutionimagereconstructionmeth-odsproposedintheliteratureconsistofthethreestages:registration,interpolationandrestoration.Asthecrucialsteptothesuccessofthesuperresolution,thedisplacementbetweentwopixelsiscalculatedasgiveninformation.Thesampledangularimagesareirregularlydata,lackofhighfrequencycomponentsandinpresenceofnoisefromtheopticalsystem.ThesimplestwaytointerpolateaHRimageisnearestneighborinterpolation.AllLRimagesareregisteredinitsposition,showninFigure6.1.Itinterpolatespixelbyitsnearestneighborpoint.ForeachpointontheHRgrid,theclosestknownLRpixelisselectedandusedasthevalueatthepointinHRgrid.Thishaslowercomplexitybutwithjaggyeffect.Themostapplicablenon-iterativealgorithmisgradientadaptiveinterpolation.Theinsertedpixelvalueisbythelocalgradientofpixels,whereallLRimagemotionswillbeestimatedbydistinctalgorithms.Accordingtothemotions,thelowresolutionimagesaremappedtotheuniformhighresolutiongrid.ThethreenearestpixelsaroundtheinterpolatedHRpixelwillbeselectedbyEuclideandistancebetweenthedesiredpixelanditsneighboringpixels.TheinterpolatedgrayvaluescanbecalculatedbyEq.(6.3),wheref(i;j)istheinterpolatedpoint,f(ik;jk)k=1,2,3areitsthreenearestpoints.W(ik;jk)isthegradientweightfunction,andmisapositivevalue,misapositivevaluecloseorsmallerthan1.f0x(ik;jk);f0y(ik;jk)areverticalderivativeandhorizontalderivative.Thegradient-basedadaptiveinterpolationtakesintoaccountoflocalgradient.Thesmallerthelocalgradientofapixelis,themorecontributionsitshouldhaveontheinterpolatedpixel[166].103f(i;j)=k=3åk=1W(ik;jk)S(ik;jk)f(ik;jk)k=3åk=1W(ik;jk)S(ik;jk)S(ik;jk)=(1Dx(ik;jk))(1Dy(ik;jk))Dx(ik;jk)=jikijDy(ik;jk)=jjkjjW(ik;jk)=(mG(ik;jk)+1)mG(ik;jk)=jf0x(ik;jk)j+jf0y(ik;jk)j2q(f0x(ik;jk))2+(f0y(ik;jk))2(6.3)6.3.2FrequencyDomainApproachSincethemultipleframessuperresolutionapproachreconstructstheHRimagefromLRimages,theremustbearelationshipbetweenLRimagesanddesiredHRimage.Thefrequencydomainmethodutilizesthisrelationship,whichisbasedon:1)TheshiftingpropertyofFouriertransformshifttheorem.Fff(tt0)g(s)=ej2pst0F(s).2)ThealiasingrelationshipbetweentheContinuousFourierTransform(CFT)ofanoriginalHRimageandtheDiscreteFourierTransform(DFT)ofobservedLRimages.3)ThereisanassumptionthatanoriginalHRimageisbandlimited.ThetwoprinciplesarecriticaltofrequencydomainsuperresolutionbecauseitbuildstheconnectionbetweenthefouriertransformofreferenceHRanditssubpixelshiftorthetransfor-mationofHRanditsdownsamplingsignals.GivenareferencecontinuousHRimagex(t1;t2),thecorrespondingfouriertransformisX(w1;w2).WhenasubpixelLRimageistaken,itisdown-samplingofankthshiftedimagefromx(t1;t2),asxk(t1;t2)=x(t1+dk1;t2+dk2)inspacial104domain(t1;t2arerepresentinghighresolution).Bythefrequencytransformshifttheorem,thecontinuousfouriertransformofshiftedimageXk(w1;w2)canbederivedasEq.(6.4).Xk(w1;w2)=exp[j2p(dk1w1+dk2w2))]X(w1+w2)(6.4)AllkthshiftedimagescanbeexpressedasEq.(6.4)infrequencydomain.BasedoninverseFouriertransform,thekthLRimagewillbeEq.(6.5).BycomparingEq.(6.4)andEq.(6.5),thediscretefouriertransformofLRimagesandCFTofHRimageisconnectedandmodeledasY=FX,suchthattheHRimagesuperresolutionbecomesanlinearinverseproblem.rk(t1;t2)=1T1T2L11ån1=0L21ån2=0(2pT1(t1N1+n1);2pT2(t2N2+n2))(6.5)6.3.3RegularizedSRReconstructionApproachThenon-uniforminterpolationandfrequencydomainapproacharebothnon-iterativealgorithmforsuperresolution.However,theiterativesuperresolutionisalsoattractiveduetotheexcellentperformance.AsshowninEq.(6.1),theobservationmodelcouldbeasEq.(6.6).Thedeterministicregularizedapproachsolvethisproblembythepriorinformation,whichcanbeconvertedtooptimizationbychoosingaxtominimizethelagrangianinEq.(6.7).Forleastsquarepriorknowledge,itsuggeststhatmostofimagesarenaturallysmoothwithlimitedhighfrequencyvalues.InEq.(6.7),arepresentsthelagrangemultiplier,commonlyreferredtotheregularizationparameter.ThisiscriticalinreconstructingHRimagesincethelargervalueofawillgenerallyleadtoasmootherHRimage.TheleastsquareapproachworksforasmallnumberofLRimages105available(ortheproblemisunderdetermined)ortheobserveddatahastoomuchnoiseduetoregistrationerrorandnoisedsampling.Ontheotherhand,ifalargenumberofLRimagesareavailableandtheamountofnoiseislow,smallawillleadtoagoodsolution.TheHRrecoverycanbeconvertedtoaniterationproblemasEq.(6.8).Theconvexproblemisdifferentiablewithquadraticregularization,wherebrepresentstheconvergenceparameter.Yk=WkX+nk;k=1;2;:::;K(6.6)minpåk=1kykWkxk2+akCxk2(6.7)(påk=1WTkWk+aCTC)‹x=påk=1WTkyk‹xn+1=‹xn+b(påk=1WTk(ykWk‹xn)aCTC‹xn)(6.8)6.3.4MultipleFramesSamplinginSinglePixelCameraInconventionalmultipleframessuperresolution,theLRimagesaretakenbymultiplecamerasorsinglecameraindistinctposition/time.However,thecompressivemultipleframessamplingcanbeimplementedbysimpleopticsdesign.AsshowninFigure6.3,thebasicsinglepixelcameraiscomposedofonemainlens,digitalmicromirrordevice(workingasspatiallightmodulator),lensforconverginglighttosinglesensor,andIRsensor.TheimagingsystemworkingprocessisdiscussedinChapter3andChapter4.Inordertogethighresolutionimage,multiplepositionangularimagesarecapturedinsinglepixelcamera.AsshowninFigure6.4,whentheobject`S'locatesinfrontofmirror,therewillbeavirtualimagewithsymmetry.Whentherearetwoimagingsystems(eyes)capturingit106Figure6.3Schematicdiagramofsinglepixelcamera.fromdistinctdirections,eachmicromirrorwillcorrespondtodifferentareaontwoangularimages.Themicromirrorinsinglepixelcameraissimilarasonepixelinsensorarray.Ifallmicromirrorsarecontinuouslyasacompleteplane,thetwoseparateimagesareexactlysameonlywithimagetranslation(thearesamefordifferentangular),asshowninFigure6.4right.TherearetwopinholesofA1andA2whichgeneratetwodistinctrecoveryimagesdi1anddi2andtheimagedisplacementcanbecalculatedbyEq.(6.9).Dd=d0d0+daDx(6.9)AsshowninFigure6.5right,fourdifferentmirrors(indifferentcolor)willdifferentpartoflightraystosinglesensorthroughtwopinholes.Forexample,theleftendpointwillmaptodifferentmirrorthroughapertureA1andA2,whichcontributestodifferentimagepixels.Themoreangularimagesarecaptured,thehigherresolutionwillbeachieved,becausetherearemoresubdivisionsofobjectthanonepinholerecovery.Inthisresearch,16differentangularimagesaresampled,asshowninFigure6.5leftandeachaperturepinholesizeisclosetomirrorsizein0.5mm.Thereare16measurementscorrespondingto16angularimages,ofwhichthedifferencearewithinsubpixel.Inexperimentalsetup,anelectricalcontrolledaperturemaskisputbetween107Figure6.4TwoaperturesdesigninDMDimagingsystem.Figure6.5Multipleaperturesdesignforhighresolutionimaging.DMDandlenstocontrolsubpixeldifference.Bythisway,themultipleframessub-apertureimagecompressivesamplingisimplemented.108Figure6.6MeasurementcoveredinneighborhoodofX0point.6.3.5ExperimentswithPrototypeCameraDuetothelimitationofcomputationandaccuracyofcompressivesensingsystem,theacquiredimagesareinlowresolution.Inthisexperiment,thecontinuouslygradientbasedinterpolationmodelwasimplementedinsinglepixelIRcamerasystem.Theregistrationincompressivesens-ingcameraisthecrucialsteptothesuccessofthesuperresolutionimageconstruction.InDMDbasedsinglepixelcamerasystem,theregistrationiseasierthanSLRcameraandthedisplacementbetweentwopixelswithinanydifferentangleimagesiscalculatedasgiveninformationsothattheregistrationislinearandeasilyobtained.Inthisresearch,thecontinuousgradientbasedinterpo-lationalsonamedsurfaceinterpolationisintroducedasmodel.AsshowninFigure6.6,withinalocalneighborhoodcenteredatS0(x0;y0),theintensityvalueatpositionS(x+x0;y+y0),f(s;s0)isapproximatedbyapolynomialexpansionwithbasis,showninEq.(6.10).However,asshowninFigure6.6,themeasurementforeachpixelcoversalargeareainlowresolution.Themeasure-mentwillbeaintegrationonarea,showninEq.(6.11),wheretheobservationmatrixisI.InordertosolvefortheprojectioncoefPforpointS0,anextraconstraintneeded.Inthisresearch,leastsquaresestimationorL2minimizationwasappliedtogetuniquesolution.f(s;s0)=p0(s0)+p1(s0)x+p2(s0)y+p3(s0)x2+p4(s0)xy+p5(s0)y2+p6(s0)x2y+:::(6.10)109I=ZZf(s;s0)dxdy=c0+c1x+c2y+p0(s0)xy+p1(s0)x2y+p2(s0)xy2+p3(s0)x3y+p4(s0)x2y2+p5(s0)xy3+:::(6.11)Agroupofexperimentalresultswereshowntoverifytheimagesfromhighresolutionsinglepixelinfraredcamera.Figure6.7showsthehardwareofsystem,includingIRsource,object,mainlens,DMDanditscontroller,digitalmask.InthistheIRsourceirradiatesonwhitespartanobject,generatingpseudodiffuseonthesurface.TheobjectirradiationwillpassthroughmainlensandformavirtualobjectimagebehindDMD.Adigitalmaskanditspatternarefollowedforangularimagecapturing.AsshowninFigure6.8,therearethreegroupshighresolutionimage.Inleft,itis9696singleangularimage,andtherightimagesare192192(4),288288(9),384384(16)pixelsdistinctly.Itisobviousthatallhighresolutionimagesgettheedgeofinsideblackhole,whiletheoriginal9696imagehasblurinsideandontheedgebecausethehighresolutionimageshasnoisyremovalduringcalculation.Atthesametime,thesharpnessbecomesbetterthanoriginalLRimageandthedetailsinsidefootballbecomesclearinwhichtheimagegoesfromfiwhite-gray-blackflfromlefttoright.Meanwhilethetexturebecomesclear.6.4SingleImagesbasedSuperResolutionDuetothehardwarecomputationallimitation,thesinglepixelcamerahasrelativelylowrecoveryresolution.Althoughpreviousinterpolationmethodcouldenhanceimageresolutionbymultipleimages,itiscomplicatedwithhightimecost.Thesingleimagebasedsuperresolutionmethods,includinginterpolation-based,learning-based,statistics-basedandotherswerereported[167].In110Figure6.7Prototypehardwareofsuperresolutionsinglepixelcamera.Figure6.8Experimentalresults,fromtop:4,9,16.111thisresearch,asplinebasedreproducingkernelhilbertspaceandapproximativeheavisidefunctionaredeployedtomodelsmoothpartandedgecomponentofimagerespectively.Byadjustingtheheavisidefunctionparameters,theproposedmodelwillenhancedistinctpartofimages.Comparedtomultiplescenesorvideosequencessuperresolutionmethod,theproposedalgorithmisappli-cableonreal-timecameraorvideosystem.TheoverallsystemcangetsuperresolutionIRimagebasedonsingleIRphotodetector,makingIRcamerabettermeasurementaccuracyandobservingmoredetailsatlongdistance.6.4.1ObservationModelingThedigitalIRimageacquiringsystemishardwaredependent.Theoutputimagecomeswithvariouskindsofdegradationsfrommotionblur,aliasingandnoisyversionoftruescene[143].AsshowninEq.(6.12),X=[x1;x2;:::;xN]denotesthedesiredhighresolutionimageaboveNyquistsamplingfrequency,whereYisthesingleLRimageobserved,BistheblurmodelseffectandDisthedownsamplingoperator,nrepresentsthenoisemodel.ThisismodelofEq.(6.1).Y=DBX+n;X=[x1;x2;:::;xN](6.12)AsshowninFigure6.9,themultipleLRimagessuperresolutionwillreconstructeachHRgriddistinctlybyextrasampling.However,thesingleLRimagewillcombinethefeatureofHRimageintoone,sothatanextraconstraintwillbenecessaryforHRreconstruction.112Figure6.9Classicalmultipleimagesandsingleimagesuperresolution.6.4.2SplinebasedReproducingKernelHilbertSpaceandApproximativeHeavisideFunctionModelTheimageintensityf(x;y)isdividedbysmoothcomponentandedges,asf(x;y)=h(x;y)+g(x;y).Inthemodel,thesmoothpart(h(x;y))ismodeledbyfunctionsinareproducingkernelhilbertspace(RKHS)andeveryevaluationfunctionisbounded.Edgeisdescribedasstepfunctioning(x;y).Foranarbitraryrealvalueh(x),x2[0;1],itstaylorseriescanbeexpandedatx=0asEq.(6.13).h0(x)andh1(x)iscorrespondingtoandsecondpartofh(x),alsoreferredtolagrangeremainder.ThespanofEq.(6.14)willformaRKHS[167],H0=spanff1;f2;f3;:::;fmgandh0(x)=åmn=1dnfn(x),dn(n=1;2;:::;m)iscoefh(x)=m1ån=0xnn!h(n)(0)+Z10(xt)m1(m1)!h(m)(t)dt=h0(x)+h1(x)(6.13)fn(x)=xn1(n1)!n=1;2;3;:::;m(6.14)Theremainderh1(x)canalsobeexpandedonaRKHS[167].LetGm(x;u)=(xu)m1(m1)!andtheboundaryconditionshn1(0)=0;n=0;1;:::;m1.H1isaHilbertspaceon[0;1]withreproducing113kernelK(s;t)=R10Gm(t;u)Gm(s;u)du[168].Thefunctionh1canbeexpandedonthebasisofH1,h1(x)=åni=1ciK(si;x).ThegeneralizedtaylorseriesexpansionisdescribedasEq.(6.16).Inmatrixformat,~h=Td+Sc,whereTisnmmatrix,Sisannmatrix,d=(d1;d2;:::;dm)0,c=(c1;c2;:::;cn)0.Thenoiseobservation~g=~h+hcanbesmoothingbyminimizingEq.(6.17).h1(x)=Z10(xu)m1(m1)!h(m)(u)du=Z10(xu)m1(m1)!dh(m1)(u)=Gm(x;1)h(m1)(1)Gm1h(m2)(1)+Gm2(x;1)h(m3)(1)+::+G1(x;1)h(0)(1)(6.15)h(x)=måv=1dvfv(x)+nåi=1cixi(x)(6.16)minc;d1nk~gTdSck2+lc0Sc(6.17)Aspreviousdiscussion,theRKHSbasedmodelmakestheimagesmooth.However,theim-agecontrastisedgedependentandfunctions,especiallyfornarrowbandcooledIRcamera.Theheavisidestepfunctionincludesasingularpointatx=0,whichdescribestheedgeonimagesignalperfectly.Intheproposedmodel,anapproximatedheavisidefunction,Eq.(6.18)isapplied,wherexcontrolsthesharpnessofsignal.ThexisselecteddistinctlyforveIRimage,narrowbandandbroadbandIRimagedependingonthethermalimagecontrastre-quirement.Thetotalimageintensityfunctionisthesummationofedgesandsmoothspart,as~f=~h+~g=Td+Sc+Yb,where~gisedgemodel,Yisedgematrixandbisedgecoef114y(x)=12+1parctan(xx)(6.18)6.4.3IterativeReconstructionAlgorithmBasedonproposedmodel,thesuperresolutionimagerestorationistheminimizationproblemasEq.(6.19),wheredesiredH=Thd+Shc+YhbisthetargetsuperresolutionoutputandtheequivalentprobleminEq.(6.20).TheEq.(6.20)`1problemcanbeeasilyconvertedtoitsaugmentedlagrangianfunctionasEq.(6.21),a,r;b2R.InordertosolveEq.(6.21),itisdividedbytwosubproblemsastheuproblemand(d;c;b)problem.TheusubproblemEq.(6.22)hasthesolutionofEq.(6.23)[169].min1nkLDB(Thd+Shc+Yhbk2+lc0Sc+akbk1(6.19)min1nkL(Tld+Slc+Ylbk2+lc0Slc+akuk1s:t:;u=b(6.20)L(d;c;b;u)=1nkL(Tld+Slc+Ylbk2+lc0Slc+akuk1+r2kub+bk2(6.21)minakuk1+r2kub+bk2(6.22)ui=maxfkbibik;argbibikbibik(6.23)The(d;c;b)subproblemismorecomplicatedthanusubproblem.TheEq.(6.24)subprob-lemcanbeemployedbyleastsquaremethod[167]sothatd;c;bcanbeobtained.Themain115Table6.1RKHSbasedsingleimagesuperresolutionalgorithm.Algorithmfori=1:na.Computetheu(i)byEq.(6.23).b.Updatethecoef(d(i);c(i);b(i))=argmin1nkL(k)(Tld+Slc+Flb)k2+lc0Slc+akbk1c.Calculatethehigh-resolutionimage:H(k)=Thd(k)+Shc(k)+Fhb(k).d.DownsamplingtoŸL=DBH(k).e.Residualupdate:L(i+1)=L(k)ŸLenditerationisshowninTable6.1.mind;c;b1nkL(Tld+Slc+Ylbk2+lc0Slc+r2kub+bk2(6.24)6.4.4SimulationsandExperimentsInthissection,therearethreegroupsofnumericalexperimentsdiscussed:veIRimages,narrowbandcooledandbroadbanduncooledIRemissiveimages.ThegroundtruthimageisfromIRcamera.Inorderforcomparisonandalgorithmevaluation,alowresolutionimageisgeneratedfromdownsamplingofHRgroundtruthatscaleof2or3.AllexperimentsareruninMATLAB(R2010a)onalaptopof3.25GbRAMandIntel(R)Core(TM)i3-2370MCPU:2.40GHz.TheveIRsensorsworksimilarasSLRcamerabutdifferentwavelength,whichmani-feststheobjectshapeandmaterialTheimagedetail(highfrequencycomponents,theedges)andbase(lowfrequency,smoothvariations)areequivalenttoeachother,sothatamediumedgepartischosenformodelingnearIRimage.AsshowninFigure6.10,thestandardbicubicandnearestneighborapproachesarecompared116Figure6.10NearIRimageofbuilding.a)bicubicmethod;b)nearestneighbor;c)proposedmethod;d)groundtruthimage.toproposedmethod(Figure6.10(c)).Fromqualitativeimages,itisobviouslythatthebicubicmethodsmoothsthetargettoomuchandgeneratesbluratedge,whilethejaggyeffecthappensonnearestneighbor.Thesmallarcdoorshapeandtriangularedgeinselectedgreenboxareaareveryclosetogroundtrueimage(Figure6.10(d)).Inquantitativeviewpoint,theRMSEvalueisbicubic:6.8170,nearestneighbor:8.7848andproposedmodel:6.0488atscaleof2.Theimagespacialresolutionincreases4timeswith2.37%normalizedRMSEerror.ThecooledemissiveIRimageshaveultrahighsensitivityandbackgroundsothatitout-putsahighcontrastwithgreatsharpnessimages.However,thesmoothpartwilldominatesuperresolutionresultsincetheedgecanbesampledinlowresolutionimage.TheproposedmodeliswithasmalleredgecomponentcomparedtoveIRimagesuperresolution.Fig-ure6.11andFigure6.12showtwocooledemissiveIRimagesandthreesuperresolutioncompar-117Figure6.11Handandheadinfraredimagefromsuperlattice(SLS)cooledFPA.a)bicubicmethod;b)nearestneighbor;c)proposedmethod;d)groundtruthimage.Figure6.12Handprintthermalimagefromcooledthermalcamera.a)bicubicmethod;b)nearestneighbor;c)proposedmethod;d)groundtruthimage.ison.ThenearestneighborisworstbutallotherthreeimageslookverysimilarinFigure6.11.The118RMSEofbicubicandproposedapproachhavebothrelativesmallvaluesof2.8092and2.3202respectively.TheweakeredgecomponentwillimprovebetteroutputonimageswhichhashighercontrastinLRformat.Figure6.12comparesthesuperresolutionreconstructionofhandprintthermalimageincolorstyle,inwhichthedifferentcolorshowsthetemperaturedistribution.Theproposedrecoveryisbetweenbicubicmethodandgroundtruthimage.Theselectedareacolororderispurple,blue,green,yellowandredfromoutsidetoinside.Thedownsamplingisatscaleof2andtheRMSEis3.7619(bicubic)and3.5696(proposed).Figure6.13Theuncooledthermalimagesuperresolutioncomparison.Figure6.14HandinfraredimagefromuncooledthermalIRcamera,a)bicubicmethod;b)proposedmethod;c)groundtruthimage.TheuncooledIRimagesarenotasgoodascooledsensors.Theimagethermalcontrastis119lowerthancooledimagessothattheedgewouldbelostinlowresolutionsampling.Intheproposedmodel,ahighedgecomponentisappliedonthisimagerestoration.Figure6.13showsonemedicalIRimage(top)andindoorIRimage(bottom)wheretheleftsmallimageisdownsamplingatscaleof3fromthemostright.Thelargesizeimageinthemiddleisfromproposedmodelwhiletheleftisfrombicubicmodel.Fromviewingimagingpoint,theneckpartintopimageisbecomingclearcomparedtoleftHRimage.Althoughitisveryhardtodistinguishfromeachotherduetothelargerawimage(top:1200900,bottom:384288),theTable6.2showstheRMSE.Theworstnormalizedroot-mean-square-erroris2.35%inallresults.Table6.2RMSEvalueofmedicalIRimageandindoorIRimage.WavelengthBicubicProposedmethodMedicalIR6.8176.004IndoorIR5.6365.479MostofuncooledthermalIRcamerahasverylowresolution.Thelowlevelcivilthermalim-agerfeaturesaround6080pixelsandenhancesthespacialresolutiontomultipletimesdigitally.Theedgeinformationislostasaresultofsuchlargeenlargement,asshowninFigure6.14(c),theoutputthermalimageisblur.However,ahighedgemodelisintroducedtostrengthenimagedetailsinFigure6.14(b).Theedgecanbeclearlyseeninthereconstructimage.6.5ChaptersummaryThesignalprocessingbasedsuperresolutionmethodisdiscussedinthischapterwhichenlargestheimageresolutionbycomputing.ThemultipleLRimagesandsingleLRimagewerebothdiscussed.Bytakingmultipleangularimages,arobusthighresolutionreconstructionalgorithmwasappliedtorecoverHRimage.Meanwhile,theadaptivesingleimagebasedsuperresolutionmodelwasalsodiscussedforIRimageenhancement.Theinfraredimageissplitbysmoothpartandedge120details,wherethe2DsplinebasedRKHSmodelisusedforimagebase(smoothcomponent)whileamodulatedheavisidefunctionisappliedforimageedge.Thesuperresolutioniscastedintoamodelbasedestimationproblem.Bycomputingthecoefoftheredundantbasisoflowresolutionimage,itwasappliedassameinhighresolutionimage.TheiterativeschemewasproposedtopreservemoreimagedetailsandtheexperimentalresultsshowthemodelisrobustandeasilyadaptivewithdifferentIRimage.Thecomputingisstillachallengingforthismethodduetoiterationinestimation.However,theparallelcomputingandhardwareacceleratingwillmaketheproposedapproachfastandeasilyintegratedintoconsumerelectronics.121Chapter7CompressiveVideoSensing7.1PreviousWorkInplenopticfunction,thelightirradiationistimedependentforadynamicscenesothatthevideosensingisnecessaryformovingvisualimageacquisition.Thevideocamerabringsbroadcastinglivelyandrealtimeimagesdirectlytoascreenforimmediateobservation.ItnotonlyservesforliveTVproduction,butalsoforsecurity,militaryandindustrialoperations.Therearenumerousde-signsandapplications,includingprofessionalvideocameras,camcorders,closedcircuittelevision(CCTV),webcams,phonecameraandsomespecialcamerasystems.Theanalogvideoproduc-tionattractedresearchattentionsince1970s,suchasvideotaperecorders(VTR)onmagnetictape,althoughthedigitalvideorecorderwasmostlypioneeredbycompany[170].Thedigitalvideoishighlycorrelatedwithbigdatabecauseitcomprisesaseriesoforthogonalframesataconstantrate.Foranexampleofoneminuteduration,framesize640480px,24bitscolordepth,andframerateof25fps,thevideosizewillbe1.38GB.Therefore,thevideocom-pressionisnecessaryforrecreatingthevideosignalsanditisusuallyintegratedinonechip[171]orsoftwarealgorithm[172].Inconventionalmethod,asshowninFigure7.1,thecompressionisaftersamplingprocess,ofwhichthemostappliedstandardisMPEG(MovingPictureExpertsGroup)compression.Theseparatedsamplingandcompressionmethodwillnotbeapplicabletononfocalplanesensorsimagingsystemassinglepixelcamera.Otherwise,itwillcostmuchtimeandmakesitveryslowframerate.Thecompressivesensingallowesustocombinethesignal122Figure7.1ConventionalNyquistShannonsignalsamplingandcompressivesampling.acquisitionandcompressionprocessesintoonestep[55],asshowninFigure7.1bottom.Althoughtheconventionalvideocameracanrecordmostofscenes,itstilldoesn'tworkwhenseeingmoreintemporaldomain.Foraspaceshuttlediscoverydeck,itis2.74gigapix-els[173]andabubbledynamicsresearchneeds500fpsvideomicroscopy[174].Moreimpor-tantly,thecommercializedhighperformancevideocameraisveryexpensiveasabasicmodel7,500fpsatmegapixel(1K1K)resolutionwith12bitcolordepth(fastcamSA5fromphotron)isaround$100,000(quotationfromtechimaging.com).Thereareatleasttwofactors,lightlimita-tionandreadoutbandwidth,toelevatethecost.AsshowninFigure7.2andEq.(7.1),theelectronsaccumulatedonsensorwillbereverseproportionaltoF-number,theratiooffocallengthtoaper-turesize(Fn#),butproportionaltoexposuretime(t),incidentillumination(Isrc),scenevity(R),quantumefy(q)andpixelsize(D2)[175].Thehighspeedvideocamerarequireslesst,lessD,buttheyaremutualrestraintinthecameradesign.Anotherlimitationissensorreadoutrate.Sincethereadouttimingwillincludeanalogtodigitalconversion,clearchargefromparallelregisterandshutterdelay,the1megapixel,1000fps,16bitcolorcamerarequires4GB=sreadout123Figure7.2LightilluminationinSLRcamera.circuitry.J=1015(Fn#)2tIsrcRqD2(7.1)Thecompressivevideosensingwasproposedtosolvethistradeoffandmadeinfraredspec-trumhighspeedvideocamerafeasible.In[96],asinglepixelcamerawassetupforvideosamplingwiththeassumptionofimagechangingslowlyacrossagroupofsnapshots.Thiswouldnotbeusedforrealtimevideostreamingwithoutincurringlatencyanddelay.Anotherpracticalsystem[176]isdevelopedbysplittingeachvideoframeintonon-overlappingblocksofequalsize.Com-pressivesamplingisthenperformedonsparseblocks,determinedbypredictingsparsitybasedonpreviousreferenceframeswhicharesampledconventionally.However,theremainingblocksarestillsampledfullysothatthecompressiveratioislow.Inordertofullyusesparsityofimagein[177],therearetwoframescapturedthepositionofthemovingobjectattwoinstances,bywhichthedifferenceframeisobtainedsubtracting.Itaccumulatesresidualerrorandmeansquareerrorincreaseswhenthedifferenceisnotsparse.Foraperiodicsignals,themodelbasedhighspeedsamplingwasproposedin[178],torecoverysignalwithtemporalsparsity.Meanwhile,anovelmeasurementmodelreducesthebilinearproblemtoasequencelinearconvexproblemfor124Figure7.3Sparsityinvideosignal.alineardynamicsystemin[179].However,thereishardlyanycontributionsoninfraredspectrumduetothesensorlimitation.Inthischapter,aninfraredhighspeedvideosensingwillbediscussed.7.2SparsityofVideoThevideocanbeseenasaseriesofimagesasshowninFigure7.3,wherethecoordinatespace(x;y;t)iscomposedasspatialdomain((x;y))andtemporaldomain(t).Foreachuncompressedframe,itisfullresolutionof2Dimage.Fortunately,itisprovedthatthenaturalimageissparseindomainsothespatialinformationcouldberedundantandintraframecompressionisproposedtodiscussthecorrelationandcompressionwithinaframe.ThebaselineisJPEGcom-pressionstandard.Theanotherredundantofvideoisbetweenlikeframes.Thecompression,referredtointerframecompression,willbaseonH.261standard.Theaudiocompressionisnotincludedinthisthesis.125Figure7.4Sub-samplingofimagecompression.7.2.1IntraframeCompressionInvideosequence,theindividualframecanbeconsideredassperateTheintraframecoding,byspatialredundancy,referstothefactthatthevariouslosslessandlossycompressiontechniquesareperformedrelativetoinformationthatiscontainedonlywithinthecurrentframe.Sub-samplingisthemostbasicofintraframecompressiontechnique,whichreducestheamountofdatabythrowingsomeofitaway.AsshowninFigure7.4,itsamplesthecolorcomponentofimageby2inbothdirections,inwhicheverysecondlineandcolumniscompletelyignoredsothatonlyquarterofpixelsaresampled.Itisverycommontosub-samplethecolordataandtheinterpolationprocessmakessubsampledimageappeartooriginalresolution.Ratherthanreducingthenumberofpixels,coarsequantization[180]methodaccomplishescompressionbyreducingthenumberofbitsusedtodescribeeachpixel.Thepixelisreassignedalowerbitscolordepththantheoriginalimagetosavestoragerequirementsforimages.Vectorquantization[181]isalsointroducedtoimagecompression,wheretheinputdatastreamisdividedasblocksandcodedusingpatternsinatable.Bycomparingtheblockbetweeninputdataandblockintable,theimagecouldbecompressivedescribed.Thewidelyusedintraframecompressionistransformcodingwhichtransformsanimagefromthespatialdomainintootherdomain(frequency,waveletetc).TheDiscreteCosineTransform(DCT)isthemostpopulartransforminimagecodingbecauseofthelowcomputationalcomplexity.AsshowninFigure7.5,the88gridisrepresentedbyDCTcoefinsteadof64original126Figure7.5DCTbasedtransformcodingimagecompression.values.ThecoefcanbecoarselyquantizedwithoutseriouslyaffectingthequalityoftheimagewhichresultsfromaninverseDCTofthequantizedcoefMostoflossyimagecodingtechniquesareextensionofthesebasicsbutonlyslightimplementationdetaildifferences.7.2.2InterframeCompressionInvideostreams,theadjacentframesarehighlycorrelatedandrelativelylittlechangeshappenfromoneframetothenextingeneral.Theinterframecompressioniscompressionmethodap-pliedtoasequenceofvideoframesratherthanasingleimage,whichexploitsthesimilaritiesbetweensuccessiveframes,knownastemporalredundancy,toreducethevolumeofdatarequiredtodescribethesequence.Inmostofinterframecompressiontechniques,thebasicconceptistoreusepartsofframesal-readyexist.Thenewframecomesfromthepreviousframeandestimation.Thesimplestinterframecompressionissub-samplingwhichoriginatesfromsub-intraframesampling.Itonlytransmitssomeofframes,e.g.everysecondframeandthedecoder(viewer)wouldberequiredtointerpolatethemissingframesatthereceivingend.Theinterframecompressionusingadjacentframeisconditionalreplenishment[182],alsonameddifferencecoding.Itbasesonthepremiseofsmall127Figure7.6Differencebetweenadjacentframe.differencebetweensuccessiveframes.AsshowninFigure7.6(160120px)andFigure7.7,morethan85%pixelshavenochanges.Inthismethod,eachframeofasequenceiscomparedwithitspredecessorandonlychangedpixelshavebeentransmitted.Thelosslesscompressionwillrequiretosampleeverychangedpixels.However,onlypixelsthatchangewillbeupdatedatthecostofintroducingsomeloss.Sincethedifferencecodingcompressionconsidersthewholeimageastargetobject,theblockbaseddifferencecodingwillbemoreefy.Duringthiscompression,theframesaredividedintonon-overlappingblocksandeachblockwillbecomparedwithitscounterpartinthepreviousframe.Theupdatesamplingisonlyforblockschanged.Thebasicdifferencecodingblocksizeissinglepixelwhiletheblockbaseddifferencecodingdependsonanarea.Duetothelargerblocksupdatesomepixelsunnecessarily,theblocksizewouldbeoptimizedbeforedesigningthepatternandalgorithm.Ingeneralperspective,theblockbaseddifferencecodingislimitedbyitsprerequisites.Whenthereisalotofmotionorthecameraitselfismoving,therearelotsofchangeshappenedineachframe.Tosolvethisproblem,itisnecessarytocompensatetheobject128Figure7.7Histogramplotofadjacentframedifference.motion.Theblockbasedmotioncompensationisapracticalandwidelyusedmethodtocompensatemovementsofblocksforfutureframe.Firstly,ithasframesegmentation,asshowninFigure7.8.Theframetobeapproximated(currentframe)isdividedintouniformnon-overlappingrectangularblocks.In[183],itclaimsthattheblockisrectangularshape(168pixels)basedonthefactthatmotionwithinimagesequencesismoreofteninthehorizontaldirectionthanthevertical.Forlargerblocksize,therewillbefewermotionvectorsbutmorecorrectiondatatotransmit[184].Ontheopposite,thesmallerblockswillresultinagreaternumberofmotionvectors,butverysensitivetonoise[185].Itisatrade-offbetweenminimizingthenumberofmotionvectorsandmaximizingthequalityofmatchingblocksinframesegmentation.Secondly,theblockmatchingisthemosttimeconsumingofencodingcompressionbecauseeachtargetblockofthecurrentframewillbecomparedwithapastframesoastoamatching/correspondingblock.Althoughthereisasearchareainblockingmatching,whichcouldbeonallofthepastframe,itisusuallyrestrictedtoasmallerareacenteredaroundthepositionofthetargetblockincurrentframe.If129Figure7.8Flowchartofmotioncompensationprocess.theblocksizeisbandthemaximumdisplacementsinthehorizontal,verticaldirectionsaredx,dyrespectively,thesearchareawillbeofsize(2dx+b)(2dy+b).TheExhaustiveSearch,BruteForceSearch,andFullSearchareproposedinresearchforblockmatching[183].Thematchingcriterionaffectsqualityofcompression,theruntimeofalgorithm.Thewholeprocessofthebestmatchisknownasmotionestimation.Thirdly,amotionvectorwillbecalculatedbasedonrelativedifferencebetweentargetblockandpastframe,whichisquantizedbyefarithmetic,adaptiveHuffmanandLewpel-Zivcoding[185].Ifthetargetblockandmatchingblockarefoundatthesamelocationintheirrespectiveframes,themotionvectorthatdescribestheirdifferenceisknownasazerovector.Inthedecompressionprocess,themotionvectorsandpastframewillbeusedreconstructcurrentframe.However,theblockmotioncompensationhasatleasttwolimitations.Itistheassumptionthatthetargetismovingparalleltocameraplaneandtheilluminationisuniformovertime.130Figure7.9Macroblock(4:2:0).7.2.3VideoCompressionThevideocompressioncanbeaslosslessandlossymethod.Theformercompressionretainstheoriginaldataandthealgorithmsavessomespacebyjustremovingimageareasthatusethesamecolor.Thelossycompressionmethodsremoveimageandsoundinformationthatisunlikelytobenoticedbytheviewerornotdifferentiatedbythehumanperception.Therefore,thequalityperceivedisstillthesame,butthevolumeisdramaticallydecreased.ThemostpopularcompressionschemeisMPEG-X(MovingPictureExpertsGroup)seriesandH.26xseries.Thecommonofthesevisualcodingstandardsaresimilarmacroblockbasedmotioncompensatedprediction,asshowninFigure7.9.Itiscorrespondingtoa1616pixelregionofaframe,including1616pixelsforYframe,88forCbandCrframes.The4:2:0chromasubsamplingwillhavefourY,oneCbandoneCr.InH.261standards,therearetwotypesofimagesframes:Intra-frames(I-frames)andInter-frames(P-frames),asshwoninFigure7.10.TheI-framesarecodedusingintraframecompression,whiletheP-framesarecodedbyaforwardpredictivecodingmethod.InMotioncompensation,theencoderwillsearchtheimagesurroundingmarchingblocktodiscoverwhereitcomesfrom.TheBruteForce,HierarchicalSearchareappliedtoencodemotions.Themotionvectordependingonthedifferencebetweencurrentmacroblockandbestmatchblockwillbeencoded.TheDCTcoefwillbequantizedforentropycoding.131Figure7.10H.261framesequence.MPEG-1isanotherinternationalstandardsemployingtwobasictechniques:blockbasedmo-tioncompensationfortemporalredundancyandtransformcoding,suchasDCTforspatialre-dundancy.Thevideosequenceisdividedintoagroupofpictures,asIntracodedFrame(I-frame),PredictiveFrame(P-frame)andBidirectionalpredictiveFrame(B-frame),asshowninFigure7.11.Forforwardprediction,themacroblockinP-frameisassignedabestmatchingblockfrompreviousI-frameorP-frame.However,thetargetmacroblockmaynothaveagoodmatchinginthepreviousframeduetounexpectedmovementsandocclusion,butitmightbebestmatchfrommacroblockinnextframe.InMPEG-1searchmatchingalgorithm,abackwardpredictionisalsoperformedinadditiontoforwardpredictionsothatthebestmatchingmacroblockisfromanextP-frameorB-frame.Consequently,eachmacroblockfromaB-framewillbetotwomotionvec-tors,onefromforwardandonefrombackwardprediction.Figure7.12showsthewholeMPGE-1encodingprocess.TheMPEG-Xseriesachievesahighcompressionratebystoringonlythechangesfromoneframetoanother.MPEG-1providesa352240resolutionat30fps.TheMPEG-2providesa720480and1280720resolutionsat60fpswithfullCD-qualityaudio.AlthoughtherearemanystandardsfromMPEG-1,theyallhavesimilarmotioncompressionwithlittledifferenceoncodingormotionestimation.132Figure7.11MPEG-1framesequence.Figure7.12MPEGcompression.133Figure7.13Spatialandtemporalresolutiontrade-offinvideostream.7.3CompressiveVideoSensing7.3.1IntroductionAsdiscussedinthischapter,thetemporalresolutionandspacialresolutionareatrade-offduetoEq.(7.1)inconventionalvideosampling.AsshowninFigure7.13left,theimageishighspacialresolutionbutlowtemporalresolutionandthespacialresolutionwillgoworsewhentemporalresolutionincrease,showninFigure7.13right.However,thereisnotradeoffbetweenspacialandtemporalresolutionincompressivevideosensing.Thespacialcompressivesensingcouldbeimplementedusingspaciallightmodulator,DMD(DigitalMicroDevice),LCOS-SLM(LiquidCrystalonSilicon-SpatialLightModulator),discussedinChapter3andChapter4.Thecombinedspatialandtemporalcompressivesensingwillbediscussedinthischapter.134Figure7.14Framedifferencesampling.7.3.2CombinedSparsitySamplingforVideoIntheproposeddesign,thetemporalredundancywillbethepriority.AsH.263standards,theimagesequenceisdividedbygroupsasthreeframesineach.Theframeisreference,whichonlysampleswithspacialredundancy,asshowninFigure7.14.Anothertwoframesfollowingreferenceframe,nameP-frame,willbesampledbytemporalredundancyandreconstructedbyreferenceframeandthedifferencerecovery.7.3.3Non-convexProblemSolverAlthough`1minimizationisfullyunderstoodandstablewithanumberoftheoreticalresults,itisnotthebestsparestsolutionsinspecialapplications,e.g.computedtomography,videostreamdifferenceinwhichtherecoveredsignalisverysparse,asshowninFigure7.6.Thenon-convexop-timization,alsoreferredto`pminimization,requiresweakerconditionsandenhancesthesparsityofrecoverysolutions.Thebasicnon-convexoptimizationis`p(0<p<1),between`1and`0problem[186][187].135Figure7.15showsthefeasiblesetofy=Fxat0;1=2;1and2norm.Thenorm-2hasabadintersectpointwhichisnotonx1orx2.Relativelyspeaking,the`1=2isclosertonon-convexproblem`0andthesmallerpwillrecovermoresparseresult.Moreimportantly,the`pminimizationproblemreconstructsresultsfromfewerlinearmeasurementsthan`1optimization[188].Invideostreams,thedifferencebetweentheadjacentframesareverysparse,whichneednonconvexpenaltytoimprovetherecoveryperformance.Figure7.15Countourmapsofdifferentpenaltiesandfeasiblesetofy=Fxatp=0;1=2;1;and2.Inorderforunderstanding,theboldfontxdenotesavectorandx=(x1;x2;:::;xn)2Rn.Thetruthvalueisx0andthevariableatitsithiterationisdenotedasx(i).Ÿxrepresentstheconvergedvalue.Ingeneral,therearethreecategoriesof`pproblemalgorithmsforEq.(7.2).IterativelyReweighted`1minimization(IRL1)isthemethodtosolveproblemEq.(7.2).Theweighted`1minimizationisdemonstratedasEq.(7.3).Candesetal.[189]suggeststheweightsasinverselyproportionaltothethegroundtruthvalue.Inimplementation,itissuggestedtochooseaccordingto136thecurrentiterates,asEq.(7.4).TheiterativelyreweightedminimizationproblemcanbeasEq.(7.5),byiterativelysolvingit.TherearealotofextendingalgorithmsbyIRL1andtheweightsaresuggestedasEq.(7.6)[187][190],wheree>0toavoidzerosunderdenominator.TheIRL1isanalyzedandprovedthattheiterateswillconvergetothesparsestsolutionwhenmeasurementsaresufŸx=argminkxkps.t.Fx=y(7.2)minx2Rnnåi=1wikxiks.t.y=Fx(7.3)wl+1i=1jxlij+e(7.4)xl+1=argminfnåi=1wlijxij;s.t.Fx=yg(7.5)wl+1i=1(jxlij+e)1p(7.6)Theparallelapproach,IterativelyReweightedLeastSquares(IRLS)wasproposedin[191]for`pminimization.ItisverysimilarasIRL1butcompletelydifferent,asEq.(7.7),wheretheweightsaresetbyEq.(7.8).AlthoughitisshownthatIRLSistheoreticallybetterthanIRL1,theconvergenceisstilluncertain.xl+1=argminfnåi=1wlix2i;s.t.Fx=yg(7.7)137wl+1i=((x(l)i)2+e)p21(7.8)Theiterativelythresholdingmethod[188]hasbeenestablishedforunconstrainedproblembyintroducingF(x;l)aspenaltyfunction.Eq.(7.9)canbesolvedbyalternatingminimizationofEq.(7.10),wherezlisanauxiliaryvariable.Bysimplyassigningz(l+1)=x(l+1),thesolutionwillbegivenbyEq.(7.11),whereQisreferredtothresholdingfunction.Thisalgorithmwillsearchalocalminimizationsequenceforlargescaleproblem,althoughitonlyappliesto`pproblematp=0,1/2,2/3and1.minx2Rn12kyFxk2+F(x;l)(7.9)x(l+1)=argminx2Rn12kx[(IFTF)z(l)+FTy]k2+F(x;l)(7.10)x(l+1)=Q((IFTF)zl+FTy;l)(7.11)7.3.4Non-convexSorted`1MethodInordertogetcloserto`0,anon-convexsorted`1isintroduced.Letthecoefwibeanonde-creasingsequenceofnonnegativenumbers,wherewn>:::>w2>w1>0.TheobjectivefunctionisasEq.(7.12),inwhichthehigherweightsareassignedoncomponentswithsmallerab-solutevalues.Thecontourmapofproposed`pisshowninFigure7.16.AnadditionalvariablePisintroducedinEq.(7.13)tosolvethenonconvexsorted`1problem,whereW=(w1;w2;:::;wn)T,(x)iistheithelementofvectorx.Asassumedtheabsolutevaluesarerankingindecreasingorderforgenerality,jx1j>jx2j>:::>jxnj.138Rw(x1;x2;:::;xn)=w1jx1j+w2jx2j+:::+wnjxnj(7.12)F(x;P)=nåi=1(PW)ijxij(7.13)Figure7.16Countourmapsofproposednonconvexsorted`1withM(M=4)values.GivenanyPRnn,if(PW)16=w1,let(PW)k=l1,andk>1.ByrowswitchingofP,let1strowexchangewithkthrowandobtainP(1)suchthat(P(1)W)1=w1,otherwiseP(1)=P.ItiseasytoderivetherelationofEq.(7.14),andF(x;P(1))6F(x;P).Foranarbitrarilyj>1andi=1;2;:::;j,itissimilarlytoP(j)suchthat(P(j))i=wi,andF(x;P(j))6F(x;P(j1))6:::6F(x;P).Afterntimesordering,P(n)W=InW,whereInisidentitymatrix,andF(x;In)6F(x;P).ItmeansminPF(x;P)=F(x;In)=Rw(x),showninEq.(7.12).F(x;P(1))F(x;P)=w1jx1j+(PW)1jxkj(PW)1jx1jw1jxkj=[w1(PW)1](jx1jjxkj)60(7.14)139Sincethenonconvexsorted`lhasthesameconvergenceasminimizationofF(x;P),theequiv-alentbasispursuitproblemEq.(7.3)willbecomeEq.(7.15),whereL(x)islagrangemultiplierofunconstrainedfunction.Itisproventhatbyagivenedx,whenxminimizesF(x;P),thexisalocalminimizerofF(x)in[188].InEq.(7.15),therearetwovariablestosolvetheproblem.Thealternatingminimizationprocedurewillbethebestoptionbecauseitiseasytosolveproblemwithanyoneofvariablesed.Inthisapproach,theoptimizationisdividedbyvariablePandx.TheproblemforxwithedPisformulatedintoaweighted`1minimization,suchasbasispursuit.TheupdatingofPistochangetheweights,referredtoiterativelyreweighted`1problem.minxnåi=1wijxij;s.t.y=FxorminxRl(x)+L(x)(7.15)Thenonconvexsorted`1canbedividedasdifferentlevelsbasedonthenumberofweights.Itbecomes2-Levelwhengivingtwodifferentweights,e.g.w1=w2=:::=wk=a1andwk+1=:::=wn=1.Itwillturnintoiterativesupportdetectionwhena1=0.TheproposedM-Levelsorted`1minimizationhasm(m>1)numberofweightsandthevalueisgeneratedbyEq.(7.16),wherercontrolsthedecreasingrateform1to0,Kdependsonsupportdetectiondiscussedlater.Theparametersa1andrarecloselyrelatedtosignalsparsityandthealgorithmnonconvexity.Thesmallera1is,themorenonconvexitybecomes.Inordertoconvergefast,itstartsfrom`1togetinitialvaluex0inthebeginningandincreasesthenonconvexitybyupdatingK.wi=8>><>>:1ifi>Ker(Ki)=Kotherwise(7.16)140Theweightsgeneratingruleiscriticalonconvergingspeed.Intheproposedmethod,KinEq.(7.16)isupdatedbyreliablesupportdetectionandtherearetwocategoriessignalsincluded,sparseGaussiansignalsandcertainpowerlawdecayingsignals.Ineachiteration,athreshold(e)isgeneratedtocomparewithxiandthenKisdetermined.Thethreshold(e)isinouterloopbycountingthesignalfastdecaying.Themostsimplethresholdgeneratingruleise(l)=jx(l)j¥=b(l+1),b>0[192].Thisisaneffectiverulewhenselectinganappropriatebbecausethelargebintroducesasmallnumberofiterationbutlowsolutionqualitywhilethesmallbcausesalargenumberofiterations.Inthiswork,theruleisbasedonlocationoffistjumpflintheincreasinglyorderedsequencejx(l)j.Findingthesmallestisuchthatjx(l)[i+1]x(l)[i]j>e(l),thensete(s)=jxs[i]j,wherex[i]representstheithlargestcomponentinx(l)bymagnitude.Insparsesignal,thetruenonzerosarelargevaluebutsmallinnumber,whilethefalsesignalarelargeinnumberbutsmallvalueasnoise.Bythismethod,thetrueonesarespreadoutwhilethefalseelementsareclustered.Thisisprovedandexperimentallyvin[192].Besides,theaccumulatedresidualerrorswithininterframesisanotherchallengeinproposedcompressivevideosampling.Sincethenoiseerroronlyhappensonthechangespart,especiallyontheimageedgeduetoframedifferencesensingmechanism,theerrorofframediffer-encereconstructioncanbeconvertedintoedgedetectionproblem.Theedgeshasstrongintensitycontrastanditisajumpintensityfromonepixeltothenext,sothattheedgedetectionwillremovebackgroundnoiseeventheresidualerrors.Themajorityofdistinctedgedetectioncouldbegroupedintotwocategories,gradientandlaplacian.Thegradientmethoddetectstheedgesbylookingforthemaximumandminimuminthederivativeoftheimage,whilethelaplacianmethodsearchesforzerocrossingsinthesecondderivativeoftheimagetoedges[193].Thegradientedgedetectionshowsamaximumlocated141Table7.1Iterativelyreweighted`1minimizationwiththresholding.AlgorithmInitializex0;wfori=1:maxita.Computethresholde.b.Updatew(i)bysupportdetection,andcheckstoppingrules.c.Updatex(l+1)withedweights.Yall1solver[195]for`1-minimizationmodelsendd.SobeledgedetectionofŸxandselectchangesparte.Reconstructnextframeimagebycurrentframeandthedenoisedframedifference.atthecenteroftheedgeintheoriginalsignal[194].Inthiswork,Sobeloperator,usingapairof33convolutionmaskonx-y,isusedtodetectframedifference,whichtheapproximateabsolutegradientmagnitudeateachpointingrayscaleimageandenhancesframedifferencetoimproveimagequality.7.3.5NumericalAnalysisInthissection,agroupofnumericalexperimentsareanalyzedandcomparedtoillustratetheper-formanceofproposedalgorithm.Inorderforcomparison,iterativereweighted`1(IRL1),twodifferentweightsmethodincludingISD(ISD),2-Level(2-Level)andmultipleweights`1mini-mization(M-Level)arerunforsameobject.InIRL1,theweightsareupdatedbyEq.(7.17).ThedifferencebetweenISDand2-Levelistheweightvalue(0,1)pairor(a,1)a2(0;1)pair.Inthisanalysis,aisselectedas0:6.w(l)i=1jxij+maxf0:5l1;0:88g(7.17)Thenonconvexcompressivesensinghasadvantageonsparsesignalrecoverycomparedtocon-vexproblem.Incompressivevideorecovery,theframedifferenceisquitesparsitysuchthat142thenonconvexalgorithmhasbetterrecovery.Firstly,itexaminestherelationbetweenoutcomeofreconstructionandsignalsparsity.AsshowninFigure7.17,thetargetistoreconstructasparsesignal(4096inlength)whichhasknonzerovalue(k2[100;1000]).Theindexofkispseudorandomvaluesselectedwithin4096andsignalintensityisalsopseudorandom,asx=(0;0;:::;1:5293;0;0:::;0:9123;0;:::).Inorderforsamplinghardwarecompatible,themea-surementmatrixisBernoullimatriceswith1entriesandthedimensionofmatrixis6144096(15%).Therearetwocriteriatoevaluaterecoveryalgorithm,runtimeandroot-mean-squareerror(RMSE).AsshowninFigure7.17,undersamemeasurementmatrix,whensetting500nonzerovaluesinobservesignal,therecoverybyIRL1costmosttimearound28sandRMSEiscloseto0.32,largesterroramongfouralgorithms.Ingeneral,IRL1algorithmhastheworstresultswithlongestruntimeandlargestRMSE,whiletheproposed2-LevelandM-LevelhavesmallerRMSEandlesstime.Inthecompressivevideosensing,thelessRMSEwillthegoaltoreducethenumberofsamplingsandincreasevideospeed.Asdiscussedinprevioussection,nonconvexapproachwillreducethenumberofmeasure-mentstoachievefullyrecovery.Thelessmeasurementsarerequired,morepowerfulalgorithmisdeployed.Thesecondanalysisistoexploretheoutcomeofreconstructionandsamplingrate.AsshowninFigure7.18,therearetwoadjacentframes(particlemoving)anditsdifferencebysimplysubtraction.Inthissimulation,thenumberofmeasurementsisadjustedfrom6%to35%(4096),asshowninFigure7.19.Eachrowofmeasurementmatrixwillkeepsameonrowindexandthenewlyaddedlineisgeneratedbypseudorandombernoullidistributionwith1entries.InFigure7.19,thex-axisrepresentsthepercentagebetweennumberofmeasurementsandsignallength(4096).Fromtheanalysisresults,theIRL1stillhasworstoutputwithlongestruntimeandlargestRMSE.TheM-LevelalgorithmhasleastRMSEwhenmeasurementpercentageislessthan143Figure7.17Signalrecoveryondistinctsparsity,4096inlength.0.1(10%).Meanwhile,thePeakSignaltoNoiseRatio(PSNR)whichdescribesthequalityofrecoveryimageanditsgroundtruthisalsodiscussed.TheM-LevelproposedalgorithmhaslargestPSNRonabove8%percentagemeasurement.Figure7.18Adjacentframeintensitydifference.Thethirdanalysisdiscussestheinterframeerror.Accumulationresidualerrordeterioratesimagereconstructioninfarbehind,e.g.thereare10framesinonegroup,1referenceframe(I-frame)and9interframes(P-frames,namedP1;P2;:::;P9),P9framewillhavelargesterrorsinceallerrorshappeningbeforewillbeaccumulatedonthisframe.AsshowninFigure7.20,thereareveframes(particlemoving)recoverybasedononeI-frame.P01-P05showsvegroundtruth144Figure7.19Signalrecoveryondifferentsamplingrate.frames.P11-P15arereconstructimagesdirectlyusingM-Levelsortedalgorithm,whileP21-P25comeafteredgedetectiondenoising.Sincetheimagedimensionisonly6464,itishardlyseeingdifferenceinvisual.TheRMSEandPSNRarelistedinTable7.2forcomparison.Theresidualerrorsincreasewhenrecoveringmoreinterframes.However,theedgedetectiondenoisingcanimproveframereconstructionby1.5dBinPSNRbecausemosterrorhappensontheframedifferenceinproposedmethod.Theedgedetectionremovesnoisebyselectingthemostchangingpartandimprovestheinterframeimagequality.145Figure7.20Accumulationresidualerror.Table7.2CharacterizingvideoframesaccumulationresidualerrorbyPSNRandRMSE.P01P02P03P04P05PSNRMLevel40.898737.458736.274535.632335.0012PSNRDenoising42.338237.583936.692835.937135.0856RMSEMLevel2.29943.41673.91574.21634.5340RMSEDenoising1.94823.36783.73174.07094.49017.3.6ExperimentalImplementationandResultsThecompressivevideosamplingisimplementedonaspatiallightmodulator,digitalmicromirrordevicefromTexasInstruments.ThetargetdynamicsceneisprojectedontoDMDplanebyobjec-tivelens.Afterprojectionwithmeasurementmatrix(onDMD),anotherlensfocusestheprojectedimagetoasinglesensor.Intheexperiments,theirradiator(THORLABSLIU850A)is850nmnearIRsourceandcommercialsiliconphotodiode(THORLABSFDS1010)isselectedasreceiversensor.TheDMDsamplingrate(projection)is6000Hz.Therearetwoexperimentsincludinglinearmovingobjectandrotatingobjecttovalidatetheproposedcompressivevideosystem.Figure7.21showsoneairplanemovingwiththeframerate10fps.Inthisexperiments,thereisonlyoneI-frameandoneP-frame,e.g.t01;t03;:::t17areP-framesandt00;t02;:::t16areI-frames.Thesamplingratiois18%and8.5%forreferenceframeandP-framerespectively.Theproposedvideosystemclearlyrecordswholesceneonrealtime.146Figure7.21Movingobjectvideorecording.Thesecondexperimentistocapturethefanrotation.AsshowninFigure7.22,eachbladeisdesignedwithdifferentlengthforeasilyThereisonereferenceframe(I-frame)andthreeP-framesintesting.Eachlineshowsonegroupofframes,includingoneI-frameandthreeP-frames.Thevideoframeisdemonstratedfromtoseventhlineontimesequence.Theframerateis18fpsandthesamplingratioofreferenceframeis20%,9%forP-frame.Figure7.22Rotatingobjectvideorecording.1477.4ChaptersummaryInthischapter,thevideocompressionwasdiscussedbyspacialandtemporalredundancy.ThecombinedmethodusesH.263standardtodividevideoframeasintra-frame(referenceframe)andinter-frame.Thenon-convexcompressivevideosensinghasadvantageonrequiringveryfewnumberofmeasurementstorecordarealtimedynamicscene.Theproposednon-convexsorted`1approachhasfastconvergencetolocalminimizerandachievesahighaccuracy.Besides,theedgedetectionbaseddenoisingreducestheaccumulatedresidualerroronframedifferencetoincreaseframerate.Comparedwiththeconventionalcompressivevideosensing,thereisnoimageanalysisduringsamplingprocess.Althoughtheexperimentalvideoframeisonly18fps,thesamplingframeratecanreachto105fpsbasedoncurrentDMDmirrorlimitation(maximum32,000Hz).Moreover,thisnon-convexcompressivevideosensinggivesareal-timevideoandmakessinglepixelcompressivevideocameraavailable.148Chapter8ConclusionsandFutureWork8.1ConclusionsInfraredspectrumexpandsvisualtoexploremorestoriesbeyondvisibleimaging.Inthestateofartinfrareddetection,thehighersensitivitysensormostlyrequiresthelargerbuckyandexpensivecryogeniccooling,althoughthisprovidesbetterspatialresolutionanddetermineslongerdetectiondistance.ThelowsensitivityuncooledIRimagingsystemneedsasmallf-numberlenstoincreasethelightsignaltransmittedthroughit.Botharesubjecttothecriticalcomponentofphotodetectorwhichdeterminestheoverallsystemperformance.Thelowdimensionalmaterials,e.g.carbonnanotube,graphene,haveattractedattentionsinceitsdiscoverybecauseofultrahighsurfacetovolumeratioandnearquasi1Dballisticelectronictransport.Thenovelmaterialshavepromisingoptoelectronicandopticalproperties.Meanwhile,singlepixelcompressivesensingbasedcamerasamplesandreconstructstargetsignalsovercomingthelimitsofNyquist-Shannontheorem.Thismechanismusesfewersensingcellsandmeasure-mentstoreconstructimageusingiterativealgorithm.Inthisresearch,theCNTbasedinfraredphotodetectorandinfraredimagingsystemarebuilt.Basedontheresultsanddiscussionspre-sentedinpreviouschapters,thefollowingconclusionscanbemade:NanoFabrication:thereal-timeelectricalfeedbacksystemwasintroducedintoDEPas-sembly.Bymeasuringtheimpedancechanges,thesystemcanquantitativelycountthenum-149berofcarbonnanotubesbridgedbetweentwoelectrodes.Theexperimentalresultsshowthatthesystemresponsespeedisfastenoughtosinglewallandmulti-wallCNTsalignment,althoughtheimpedancediffersmuch.ItreducesthetimeofCNTselectionandsamplecleancomparedtoconventionalmethodandwillalsobeapplicabletothinlayergraphenedepositioncontrolandothernanomaterialsdepositionandlocalization.NanoSensorCharacterization:thein-situmeasurementmethodusingdigitalmicroscopeandpreciseveaxissubstagelocalizetherelativepositionbetweennanoscalesensorandIRbeambymappingthephotocurrentonlaserspot.TheexperimentalresultsindicatethatphotovoltaicdominatesphotoresponseonCNT-metalSchottkyphotodetectoralthoughthephotovoltaicvoltagecannotbesampledduetoitstinyvaluemergedinnoise.ThedetectorIRresponsearedependentonCNT-metalcontactsizeandmetalworkfunction.There-sponsivitycanbecalculatedbyphotocurrentmapping.Theproposedmeasurementmethodprovidesarobustandpreciseapproachtocharacterizesub-microandnanoscalephotode-tectorwhichisimportantforsub-wavelengthscalephotodetectorcharacterization.NanoSensorReliability:thereliabilityrelatedcharacteristicsofnanoelectronicsandnanosensorscouldbesummarizedasmulti-scaleinbothgeometricandtimedomain.Theformerishighlydependentonsensorarrayuniformitywhilethelatternano-reliabilitymeasurestheprobabilitythatanano-scaledproductperformsitsintendedfunctionalitywithoutfail-ureundergivenconditionsforaperiodoftime.Thesandwichstructuredsensorwasfabricatedwithintwopolymerlayers,inwhichthesubstratepolyimideprovidedsen-sorwithhighbendingstiffness,andtopparylenepackingblockedhumidityenvironmentalnoise.ThedesignedstructureisolatesstimulusexceptselectedIRwavelength,especiallyfromsubstratecharges.Theproposeddesignimprovessensorresponseandavoidsfailure150underdifferentenvironmentalconditions.Thefabrication,substratematerial,andsensorstructurearecriticaltoCNT-IRphotodetectorreliabilitywhichisessentialforthetransitionfromnanosciencetonanoengineering.WeakSignalReadout:theelectricalpropertiesofCNTIRphotodetectoraresuchcompli-catedduetounknowninternalstructure.Theweaksignalintensity(nA),biassensitive(zerobias)limitthereadoutdesign.Inthisresearch,thezerobias,highgain,twostagereadoutisdesignedtotestthelowcurrenttosubnanoampereinCNTbasedIRdetector.Meanwhile,italsoworksinCNTbasedsinglepixelIRimagingsystem.Althoughitworksinhundredshertz,thatisnotenoughforfasthighresolutionimagerecovery.Theultrafastreadoutmethodisstillachallengingprojectinnanoelectronics.IRCompressiveImager&3D:acompressivesensingbasedIRcamerasystemwasdevel-oped.Inthecamerasystem,asingleCNTeffecttransistorintegratedwithaphotoniccavitywasemployedtomeasurethecompressedsignals.Thiscamerasystemwascapableofobservingthedynamicmovementofalaserspot.Thebinocular3DIRcameraisalsoim-plementedinthissystem.ThiscameraarchitectureprovidesanovelandalternativeplatformforfutureIRcameras,particularlythecamerasusingnano-photodetectors.CompressiveLightField:bytakingmultipleangularimagesthroughcodedaperture,wecanusesingleIRsensortoreconstructalargeobject.SyntheticaperturephotographywillmaketheIRfocusproblemgoawayandalsoget3Dimagingusingsimilaroptics.Theexperimentalresultsshowthatthemoreangularimageswillachievethelargerobjectre-covery,betterrefocusinsyntheticaperturephotography.Thedoublecompressivesensingreducesthesamplingratiobyangularimageredundancy.Itimprovessamplingspeedwithhighaccuracy.151NonConvexCompressiveVideoSensing:thenon-convexcompressivevideosensinghasadvantageonthetradeoffproblembetweentemporalandspatialresolutioninvideocaptur-ing.Besides,itrequiresveryfewnumberofmeasurementstorecordarealtimedynamicscene.Thenon-convexsorted`1approachhasfastconvergencetolocalminimizerandachieveahighaccuracy.Theedgedetectionbaseddenoisingreducestheaccumulatedresid-ualerroronframedifferencesoastoincreaseframerate.Comparedwiththeconventionalcompressivevideosensing,theproposed`psolverwillgeneratevideoframeinrealtime.Thisisthereportedreal-timesinglepixelcompressivevideocamera.InfraredSuperResolution:thesuperresolutionisaspecialapproachtosolveinfraredimagelowresolutionproblem.The2DsplinebasedRKHSmodelisusedforimagebase(smoothcomponent)whileamodulatedheavisidefunctionisappliedforimageedge.Thesuperresolutioniscastedintoamodelbasedestimationproblem.Bycomputingtheco-efoftheredundantbasisoflowresolutionimage,itwasappliedassameinhighresolutionimageforcomputing.Theexperimentalresultsshowthemodelisrobustandeas-ilywithdifferentIRimage.Thesinglelowresolutionimagebasedsuperresolutionwillbringinfraredtechnologyintoconsumerelectronics.8.2FutureResearchTherearethreepartsoffutureworksfrominfraredsensoranditsbroadlyimagingapplication.UltraHighSensitivityIRSensor:thesensitivityisacriticalparametertocharacterizeinfraredsensor.Theconventionalmaterialsbasedtechnologyhasitslimitationonthermalnoiseandthenoise-equivalenttemperaturedifference(NETD)ismorethan20mK.Anotherrelatedparametricvalueissensingareawhichdominatesquantumefy.Thesingle152CNTbasedIRphotodetectorhassuchtinysensingareathatitmightnotapplicableforlargeareaIRimager.Besides,thesensingareaalsodependsonIRwavelengthduetolightdiffraction.The2Dlowdimensionalmaterials,e.g.grapheneandgraphenederivative(rGO),Molybdenum(MoS2),Phosphoreneoritshibridmaterials,havelargeareaandgoodsemiconductorperformance.Theywillopenanewapproachforhighsensitivity,highquantumefyIRphotodetector.CompressiveIRLightFieldApplication:infraredwavelengthdiscoversdistinctcharac-teristicsofobjectcomparedtovisibleimagingbecausethematerialhasuniquedistributionalongwholeelectromagneticspectrumespeciallyonmaterial,biology,plantsandanimalsscience.Thebroadinfraredlightincludes3D,generallightandfastinfraredvideo.Thesethreetechnologywillexploremoredistinctdetailsforallapplicationsabove.Theinfraredmicroscopytogetherwith3D,lightandfastcompressivevideoisapromisingtechnologyforthecutting-edgeresearch.IRSuperResolution:therearetwoconceptsonsuperresolutioninresearch.Theisalgorithmbasedincomputerscience.Itincreasesimagespatialresolutionfromalowresolutionimage.Thebasicideaistomodelthetargetandestimatethehighresolutionusingsignalprocessing.However,anothersuperresolutionistoresolvelightdiffractionlimitationinopticsandscienceresearch.Theinfraredwavelengthissubmicrototensmicrometersothatitisextremelydiftodiscoverfeathersunderneaththewavelength.Anextrahardwareorareisrequiredtoincreasesamplingresolution.Thissuperresolutionismorechallengingandmoremeaningfulinmicroscopyimaging.153BIBLIOGRAPHY154BIBLIOGRAPHY[1]MichaelRowan-Robinson.NightVision:ExploringtheInfraredUniverse.CambridgeUniversityPress,2013.[2]F.Szmulowicz,GailJ.Brown,HuiC.Liu,AidongShen,ZbigniewR.Wasilewski,andMargaretBuchanan.GaAs/AlGaAsp-typemultiplequantumwellsforinfrareddetectionatnoimalincidence:modelandexperiment.Optoelectronicsreview,9:164Œ173,2001.[3]BarbaraH.Stuart.InfraredSpectroscopy:FundamentalsandApplications.Wiley,2004.[4]ChristianW.Freudiger,WenlongYang,GaryR.Holtom,NasserPeyghambarian,X.SunneyXie,andKhanhQ.Kieu.Stimulatedramanscatteringmicroscopywitharobustlasersource.NaturePhotonics,8:153Œ159,2014.[5]H.Z.Chen,N.Xi,K.W.Lai,L.L.Chen,R.G.Yang,andB.Song.Gatedependentphoto-responsesofcarbonnanotubeeffectphototransistors.Nanotechnology,23:385203,2012.[6]C.Hildebrandt,C.Raschner,andK.Ammer.AnoverviewofrecentapplicationofmedicalinfraredthermographyinsportsmedicineinAustria.IEEEEngineeringinMedicineandBiology,1:21Œ57,2010.[7]J.F.Head,F.Wang,C.A.Lipari,andR.L.Elliott.Theimportantroleofinfraredimaginginbreastcancer.InegrativeandComparativeBiology,48(365):50Œ59,2000.[8]M.Massoud.Blackbodyradiation.Engineeringthermodynamics,me-chanics,andheattransfer.Springer,2005.[9]J.MehraandH.Rechenberg.TheHistoricalDevelopmentofQuantumTheory.Springer-Verlag.,1982.[10]A.P.FrenchandE.F.Taylor.AnIntroductiontoQuantumPhysics.NortonCompany,1978.[11]P.Madejczyk,W.Gawron,A.Piotrowski,K.Klos,J.Rutkowski,andA.Rogalski.Improve-mentinperformanceofhighoperatingtemperatureHgCdTephotodiodes.InfraredPhysicsandTechnology,54:310Œ315,2011.155[12]A.Rogalski.Infrareddetectors:anoverview.InfraredPhysics&Technology,43:187Œ210,2001.[13]G.L.Hansen,J.L.Schmit,andT.N.Casselman.EnergygapversusalloycompositionandtemperatureinHg1xCdxTe.J.Appl.Phys,53:7099,1982.[14]ChineduE.Ekuma,C.E.Singh,andD.J.Moreno.OpticalpropertiesofPbTeandPbSe.PhysicalReviewB,85:085205,2012.[15]RogerK.Richards,DonaldP.Hutchinson,andCharlesA.Bennett.Room-temperatureQWIPdetectionat10mm.InProc.SPIE,InfraredTechnologyandApplications,volume4820,pages1Œ4.SPIE,2003.[16]Yole.Uncooledinfraredimagingtechnologyandmarkettrends2014,January2014.[17]Markets.Infraredimagingmarketworth$8450millionby2020,January2014.[18]FLIR.InfraredcamerasfromFLIR,January2014.[19]Fluke.Flukeinfraredcamerasareonthejobbecausetheydothejob,January2014.[20]P.L.Richards.Bolometersforinfraredandmillimeterwaves.JournalofAppliedPhysics,76(1):1Œ24,1994.[21]S.Nudelman.Thedetectivityofinfraredphotodetectors.AppliedOptics,1(05):627Œ636,1962.[22]R.C.Jones.Proposalofthedetectivityfordetectorslimitedbyradiationnoise.JournaloftheOpticalSocietyofAmerica,50(11):1058Œ1059,1960.[23]MichaelA.Kinch.FundamentalsofInfraredDetectorMaterials.SPIEPress,2007.[24]VanThanhDaua,TakeoYamadab,DzungVietDaoa,BuiThanhTunga,andKenjiHatab.IntegratedCNTsthinforMEMSmechanicalsensors.MicroelectronicsJournal,41(12):860Œ864,2010.[25]YanWang,ZhiqiangShi,YiHuang,YanfengMa,ChengyangWang,MingmingChen,andYongshengChen.Supercapacitordevicesbasedongraphenematerials.J.Phys.Chem,113:13103Œ13107,2009.156[26]MaogangGong,TejasA.Shastry,YuXie,MarcoBernardi,DanielJasion,KyleA.Luck,TobinJ.Marks,JeffreyC.Grossman,ShenqiangRen,andMarkC.Hersam.Polychiralsemiconductingcarbonnanotubefullerenesolarcells.Nano.Lett,14:5308Œ5314,2014.[27]VeraSazonova,YuvalYaish,HandeUstunel,DavidRoundy,TomasA.Arias,andPaulL.McEuen.Atunablecarbonnanotubeelectromechanicaloscillator.Nature,431:284Œ287,2004.[28]M.MittalaandA.Kumara.Carbonnanotube(CNT)gassensorsforemissionsfromfossilfuelburning.SensorsandActuatorsB:Chemical,203:349Œ362,2014.[29]M.S.Marcus,J.M.Sinnons,O.M.Castellini,R.J.Hammers,andM.A.Eriksson.Photo-gatingcarbonnanotubetransistors.JournalofAppliedPhysics,100(07):084306,2006.[30]I.A.LevitskyandW.B.Euler.Photoconductivityofsingle-wallcarbonnanotubesundercontinuous-wavenearinfraredillumination.AppliedPhysicsLetters,83:1857Œ1859,2003.[31]AkihikoFujiwara,YasuyukiMatsuoka,HiroyoshiSuematsu,NaokiOgawa,KenjiroMiyano,HiromichiKataura,YutakaManiwa,ShinzoSuzuki,andYohjiAchiba.Photocon-ductivityinsemi-conductingsinglewalledcarbonnanotubes.JapaneseJournalofAppliedPhysics,40:L1229ŒL1231,2001.[32]Jiangbozhang,NingXi,HongzhiChen,KingWaiChiuLai,GuanyongLi,andUchechukwuC.Wejinya.Design,manufacturing,andtestingofsinglecarbonnanotubebasedinfraredsensors.IEEEtransactionsonnanotechnology,8(2):245Œ251,2009.[33]JiangboZhang,NingXi,KingW.C.Lai,HongzhiChen,andYilunLuo.Singlecarbonnan-otubebasedphotodiodesforinfrareddetection.InProceedingofthe7thIEEEInternationalConferenceonNanotechnology,pages1156Œ1160.IEEE,2009.[34]JiangboZhang,NingXi,HongzhiChen,KingW.C.Lai,GuanyongLi,andUchechukwuWejinya.Photovoltaiceffectinsinglecarbonnanotube-basedschottkydiodes.Int.JournalNanoparticles,1(2):108Œ118,2008.[35]HongzhiChen,NingXi,KingW.C.Lai,CarmenK.M.Fung,andRuiguoYang.Develop-mentofinfrareddetectorsusingsinglecarbonnanotubesbasedeffecttransistors.IEEETransactiononNanotechnology,5(9):582Œ589,2010.[36]JaveyA.,GuoJ.,WangQ.,LundstromM.,andDaiH.Ballisticcarbonnanotubetransistors.Nature,424:654Œ657,2003.157[37]CollinsP.G.,ArnoldM.S.,andAvourisP.Engineeringcarbonnanotubesandnanotubecircuitsusingelectricalbreakdown.Science,292:706Œ709,2001.[38]BockelmannU.andBastardG.Phononscatteringandengergyrelaxationintwooneandzerodimensionalelectrongas.Phys.Rev.B,42:8947Œ8951,1990.[39]KuoD.,FangA.,andChangY.Theoreticalmodelingofdarkcurrentandphotoresponseforquantumwellandquantumdotinfrareddetectors.InfaredPhys.Technol.,42:422Œ433,2001.[40]DaJiangYang,QingZhang,andGeorgeChen.Thermalconductivityofmultiwalledcarbonnanotubes.PHYSICALREVIEWB,66:165440,2002.[41]ZhidongHanaandAlbertoFinab.Thermalconductivityofcarbonnanotubesandtheirpolymernanocomposites:Areview.ProgressinPolymerScience,36:914Œ944,2011.[42]RahatAfrin,JibranKhaliq,MohammadIslam,IftikharHussainGul,ArshadSaleemBhatti,andUmairManzoor.Synthesisofmultiwalledcarbonnanotube-basedinfraredradiationdetector.SensorsandActuatorsA,187:73Œ78,2012.[43]M.Freitag,Y.Martin,J.A.Misewich,R.Martel,andPh.Avouris.Photoconductivityofsinglecarbonnanotubes.NanoLetters,3:1067Œ1071,2003.[44]BasudevPradhan,KristinaSetyowati,HaiyingLiu,DavidH.Waldeck,andJianChen.Car-bonnanotubepolymernanocompositeinfraredsensor.NanoLetters,8:1142Œ1146,2008.[45]R.Lukac.ComputationalPhotography:MethodsandApplications.CRCPress,2010,2010.[46]S.D.Babacan,R.Ansorge,M.Luessi,P.R.Mataran,R.Molina,andA.K.Katsaggelos.Compressivelightsensing.IEEETransactionsonimageprocessing,21:4746Œ4758,2012.[47]S.K.NayarandT.Mitsunaga.Highdynamicrangeimaging:Spatiallyvaryingpixelexpo-sures.InIEEEConferenceonComputerVisionandPatternRecognition,pages472Œ479.IEEE,2000.[48]P.Sen,N.K.Kalantari,M.Yaesoubi,S.Darabi,D.B.Goldman,andE.Shechtman.Ro-bustpatch-basedHDRreconstructionofdynamicscenes.ACMTransactionsonGraphics,31:203:1Œ203:12,2012.158[49]J.Im,J.Jeon,M.H.Hayes,andJ.Paik.Singleimage-basedghost-freehighdynamicrangeimagingusinglocalhistogramstretchingandspatially-adaptivedenoising.IEEETransac-tionsonConsumerElectronics,57:1478Œ1485,2011.[50]M.Levoy.Lighteldsandcomputationalimaging.IEEEComputer,8:45Œ56,2006.[51]Z.Xu,J.Ke,andE.Y.Lam.High-resolutionphotographyusingtwomasks.Opt.Express,20:10971Œ10983,2012.[52]R.F.Marcia,Z.T.Harmany,andR.M.Willett.Compressivecodedapertureimaging.IninComputationalImagingVII,Proc.SPIE,volume7246,pages1Œ13.SPIE,2009.[53]R.F.MarciaandR.M.Willett.Compressivecodedaperturesupperresolutionimagerecon-struction.InICASSP2008.IEEEInternationalConferenceon,pages833Œ836.IEEE,2008.[54]M.F.Duarte,M.A.Davenport,D.Takhar,J.N.Laska,K.F.KellyT.Sun,andR.G.baraniuk.Single-pixelimagingviacompressivesampling.IEEESIGNALPROCESSINGMAGAZINE,8:82Œ90,2008.[55]E.Candes,J.Romberg,andT.Tao.Robustuncertaintyprinciples:Exactsignalreconstruc-tionfromhighlyincompletefrequencyinformation.IEEETrans.Inform.Theory,52:489Œ509,2006.[56]E.CandesandJ.Romberg.Quantitativerobustuncertaintyprinciplesandoptimallysparsedecompositions.Found.Comput.Math.,6:227Œ254,2006.[57]E.CandesandT.Tao.Decodingbylinearprogramming.IEEETrans.Inform.Theory,51:4203Œ4215,2005.[58]DanielDregely,RichardTaubert,JensDorfmuller,RalfVogelgesang,KlausKern,andHar-aldGiessen.3DopticalYagi-Udananoantennaarray.NatureCommunications,2(267):1Œ7,2011.[59]J.R.BergenE.H.Adelson.Theplenopticfunctionandtheelementsofearlyvision.InComputationModelsofVisualProcessing,51:3Œ20,1991.[60]RenNg.Digitallightphotography.StanfordUniversity,Stanford,CA,2006.[61]SumioIijima.Helicalmicrotubulesofgraphiticcarbon.Nature,354(07):56Œ58,1991.159[62]KatauraH.,KumazawaY.,ManiwaY.,UmezuI.,SuzukiS.,OhtsukaY.,andAchibaY.Opticalpropertiesofsinglewallcarbonnanotubes.SyntheticMetals,103(01):2555Œ2558,1999.[63]CollinsP.G.,ArnoldM.S.,andAvourisP.Engineeringcarbonnanotubesandnanotubecircuitsusingelectricalbreakdown.Science,292:706Œ709,2001.[64]Y.Matsuda,W.Deng,andW.Goddard.Contactresistancepropertiesbetweennanotubesandvariousmetalsfromquantummechanics.J.Phys.Chern,I11:1113Œ11116,2007.[65]I.DeretzisandA.Magna.Roleofcontactbondingonelectronictransportinmetal-carbonnanotube-metalsystems.Nanotechnology,17:5063,2006.[66]Y.Chai,A.Hazeghi,K.Takei,Hong-YuChen,ChanPhilipC.H.,JaveyA.,andWongH.S.P.Low-resistanceelectricalcontacttocarbonnanotubeswithgraphiticinterfaciallayer.IEEETrans.onElectronDevices,59:12Œ20,2012.[67]CarlaM.Aguirre,PierreL.Levesque,MatthieuPaillet,FrancoisLapointe,BenoitC.St-Antoine,PatrickDesjardins,andRichardMartel.Theroleoftheoxygen/waterredoxcoupleinsuppressingelectronconductioninfecttransistor.AdvancedMaterials,21:3087Œ3091,2009.[68]HarishM.Manohara,EricW.Wong,ErichSchlecht,BrianD.Hunt,andPeterH.Siegel.CarbonnanotubeschottkydiodesusingTischottkyandPtohmiccontactsforhighfrequencyapplications.NanoLetters,5(7):1469Œ1474,2005.[69]HongzhiChen,NingXi,BoSong,LiangliangChen,JianguoZhao,KingWaiChiuLai,andRuiguoYang.Infraredcamerausingasinglenano-photodetector.IEEESensorsJournal,13:949Œ958,2013.[70]VikrantJayantGokhale,OlgaA.Shenderova,GaryE.McGuire,andRais-ZadehM.In-fraredabsorptionpropertiesofcarbonnanotube/nanodiamondbasedthincoatings.JOURNALOFMICROELECTROMECHANICALSYSTEMS,23(1):191Œ197,2014.[71]M.MonthiouxandV.Kuznetsov.Whoshouldbegiventhecreditforthediscoveryofcarbonnanotubes.Carbon,44(9):1621,2006.[72]FengYang,XiaoWang,DaqiZhang,JuanYang,DaLuo,ZiweiXu,JiakeWei,Jian-QiangWang,ZhiXu,FeiPeng,XuemeiLi,RuomingLi,YilunLi,MeihuiLi,XuedongBai,FengDing,andYanLi.growthofsingle-walledcarbonnanotubesonsolidalloycatalysts.Nature,510:522Œ524,2014.160[73]MukulKumarandYoshinoriAndo.Chemicalvapordepositionofcarbonnanotubes:Areviewongrowthmechanismandmassproduction.JournalofNanoscienceandNanotech-nology,10:3739Œ3758,2010.[74]MartialDuchamp,KyuminLee,BenjaminDwir,JinWonSeo,EliKapon,LaszloForro,andArnaudMagrez.Controlledpositioningofcarbonnanotubesbydielectrophoresis:Insightsintothesolventandsubstraterole.ACSnano,4(1):279Œ284,2010.[75]ZhifengDeng,ErhanYenilmez,AmyReilein,JoshuaLeu,HongjieDai,andKathrynA.Moler.Nanotubemanipulationwithfocusedionbeam.AppliedPhysicsLetters,88(1):023119,2006.[76]LibaoAnandCraigR.Friedrich.Real-timegapimpedancemonitoringofdielectrophoreticassemblyofmultiwalledcarbonnanotubes.AppliedPhysicsLetters,92:173103Œ173106,2008.[77]SebastianSorgenfrei,InancMeric,SarbajitBanerjee,AustinAkey,SamiRosenblatt,Irv-ingP.Herman,andKennethL.Shepard.Controlleddielectrophoreticassemblyofcarbonnanotubesusingreal-timeelectricaldetection.AppliedPhysicsLetter,94:053105Œ053108,2009.[78]MingZheng,AnandJagota,MichaelS.Strano,AdelinaP.Santos,PaulBarone,S.GraceChou,BruceA.Diner,MildredS.Dresselhaus,RobertS.Mclean,G.BibianaOnoa,GeorgiiG.Samsonidze,EllenD.Semke,MonicaUsrey,andDennisJ.Walls.Structure-basedcarbonnanotubesortingbysequence-dependentDNAassembly.Science,302(5650):1545Œ1548,2003.[79]DidiXu,A.Subramanian,LixinDong,andB.J.Nelson.Shapingnanoelectrodesforhigh-precisiondielectrophoreticassemblyofcarbonnanotubes.IEEETRANSACTIONSONNANOTECHNOLOGY,8(4):449Œ460,2009.[80]S.E.Harding,M.Dampier,andA.J.Rowe.Modelingbiologicalmacromoleculesinsolu-tion:Thegeneraltri-axialellipsoid.IRCSMed.Sci.,7:1813Œ1829,1979.[81]E.W.SmallandI.Isenberg.Hydrodynamicpropertiesofarigidmolecule:Rotationalandlineardiffusionandanisotropy.Biopolymers,16:1907Œ1928,1977.[82]KingW.C.Lai,NingXi,andUchechukwuC.Wejinya.Automatedprocessforselectionofcarbonnanotubebyelectronicpropertyusingdielectrophoreticmanipulation.JournalofMicro-NanoMechatronics,4:37Œ48,2008.161[83]ChechukwuC.Wejinya,YantaoShen,NingXi,KingW.C.Lai,andJiangboZhang.Anefapproachofhandlinganddepositionofmicroandnanoentitiesusingsensorizedend-effectorsystem.SensorsandActuatorsA:Physical,147:6Œ16,2008.[84]Tsung-TsongWu,Yung-YuChen,andTai-HsuChou.AhighsensitivitynanomaterialbasedSAWhumiditysensor.JournalofPhysicsD:AppliedPhysics,41(8):085101,2008.[85]J.SvenssonandE.Campbell.Schottkybarriersincarbonnanotube-metalcontacts.JournalofAppliedPhysics,110:111101,2011.[86]H.L.SkriverandN.M.Rosengaard.Surfaceenergyandworkfunctionofelementalmetals.PhysicalReviewB,46:7157Œ7168,1992.[87]MitsumasaIWAMOTO,AtsushiFUKUDA,andFuminoriHIRAYAM.Chargestoragephe-nomenaobservedinultrathinpolyimidelangmuirblodgettdepositedonmetalelec-trodes.InProceedingofthe3rdinternaltionalconferenceonpropertiesandapplicationsofdielectricmaterials,pages193Œ196,1991.[88]MiaoYu,LiangliangChen,K.W.C.Lai,C.K.M.Fung,N.Xi,L.Zhang,andL.Dong.3Dassemblyandsimulationofhelicalopticalantenna-enhancedcarbonnanotubeIRsensors.InNanotechnology(IEEE-NANO),2014IEEE14thInternationalConferenceon,pages849Œ852.IEEE,2014.[89]D.J.Bell,L.Dong,andB.J.Nelson.Fabricationandcharacterizationofthree-dimensionalInGaAs/GaAsnanosprings.NanoLetters,6(4):725Œ729,2006.[90]L.Chen,N.Xi,Z.Zhou,andB.Song.Quantitativelycontrolofcarbonnanotubesusingrealtimeelectricaldetectiondielectrophoresisassembly.InNanotechnology(IEEE-NANO),201515thIEEEConferenceon,pages1029Œ1032.IEEE,2015.[91]X.ZhangandZ.Liu.Superlensestoovercomethediffractionlimit.Nature,7:435Œ441,2008.[92]J.U.Lee.Photovoltaiceffectinidealcarbonnanotubediodes.AppliedPhysicsLetters,87:073101Œ1C073101Œ3,2005.[93]A.ROGALSKIandK.CHRZANOWSKI.Infrareddevicesandtechniques.Opto-ElectronicsReview,10:111Œ136,2002.[94]K.Besteman,J.-O.Lee,F.G.M.Wiertz,H.A.Heering,andC.Dekker.Enzyme-coatedcarbonnanotubesassingle-moleculebiosensors.NanoLetter,3:727Œ730,2003.162[95]S.J.Wind,J.Appenzeller,R.Martel,V.Derycke,andP.Avouris.Verticalscalingofcarbonnanotubed-effecttransistorsusingtopgateelectrodes.AppliedPhysicsLetters,80:3817Œ3819,2002.[96]MichaelB.Wakin,JasonN.Laska,MarcoF.Duarte,DrorBaron,ShriramSarvotham,DharmpalTakhar,KevinF.Kelly,andRichardG.Baraniuk.Compressiveimagingforvideorepresentationandcoding.In2006ProceedingsofPictureCodingSymposium(PCS),pages1Œ6.IEEE,2006.[97]YanfeiWang,JingjieCao,andChangchunYang.Recoveryofseismicwavbasedoncompressivesensingbyan`1-normconstrainedtrustregionmethodandthepiecewiserandomsub-sampling.Geophys.J.Int.,000(1):1Œ9,2010.[98]BoSong,JianguoZhao,NingXi,HongzhiChen,KingWaiChiuLai,RuiguoYang,andLiangliangChen.Compressivefeedback-basedmotioncontrolfornanomanipulation:the-oryandapplications.IEEETransactionsonRobotics,30(1):103Œ114,2014.[99]PengZhang,ZhenHu,R.C.Qiu,andB.M.Sadler.Acompressedsensingbasedultra-widebandcommunicationsystem.InIEEEInternationalConferenceonCommunications,2009.,pages1Œ5.IEEE,2009.[100]MichaelLustig,DavidL.Donoho,JuanM.Santos,andJohnM.Pauly.CompressedsensingMRI.IEEESIGNALPROCESSINGMAGAZINE,3:72Œ82,2008.[101]M.A.Davenport,P.T.Boufounos,M.B.Wakin,andR.G.Baraniuk.Signalprocessingwithcompressivemeasurements.IEEEJournalofSelectedTopicsinSignalProcessing,4(2):445Œ460,2010.[102]E.CandesandM.Wakin.Anintroductiontocompressivesampling.IEEESignalProcessingMagazine,25(2):21Œ30,2008.[103]ErwanLePennecandStphaneMallat.Sparsegeometricimagerepresentationwithban-delets.IEEETransactionsonImageProcessing,14(4):423Œ438,2005.[104]DavidL.Donoho.Compressedsensing.IEEETransactionsonInformationTheory,52(4):1289Œ1306,2006.[105]A.Cohen,W.Dahmen,andR.DeVore.CompressedsensingandbestK-termapproxima-tion.J.Am.Math.Soc.,22(1):211Œ231,2009.[106]JoelA.Tropp.Greedisgood:algorithmicresultsforsparseapproximation.IEEETransac-tionsonInformationTheory,50(10):2231C2242,2004.163[107]G.M.Davis,S.Mallat,andM.Avellaneda.Adaptivegreedyapproximations.JournalofConstructiveApproximation,13(1):57Œ98,1997.[108]P.L.CombettesandV.R.Wajs.Signalrecoverybyproximalforward-backwardsplitting.MultiscaleModelingandSimulation,4(4):1168Œ1200,2005.[109]D.NeedellandJ.A.Tropp.COSAMP:iterativesignalrecoveryfromincompleteandinac-curatesamples.Appl.Comput.Harmon.Anal.,26(3):301Œ321,2009.[110]W.DaiandO.Milenkovic.Subspacepursuitforcompressivesensingsignalreconstruction.IEEETrans.Inf.Theory,55(5):2230Œ2249,2009.[111]A.BeckandM.Teboulle.Afastiterativeshrinkage-thresholdingalgorithmforlinearinverseproblems.SIAMJournalonImagingSciences,2(1):183Œ202,2009.[112]J.YangandY.Zhang.Alternatingdirectionalgorithmsfor`1-problemsincompressivesensing.SIAMJournalonComputing,33(1):250Œ278,2011.[113]J.YangandX.Yuan.Linearizedaugmentedlagrangianandalternatingdirectionmethodsfornuclearnormminimization.MathematicsofComputation,82(281):301Œ329,2013.[114]L.Chen,N.Xi,H.Chen,andKingWaiChiuLai.ReadoutsystemdesignforMWCNTinfraredsensors.InProceedingoftheIEEENanotechnologyConference,2011,pages1Œ4.IEEE,2011.[115]J.G.Graeme.Photodiodes,OPAMPSolutions.McGraw-HillPress,NewYork,1995.[116]DonaldA.Neamen.SemiconductorPhysicsAndDevices:BasicPrinciples.McGraw-HillPress,NewYork,2003.[117]MichaelFaraday.Thoughtsonrayvibrations.ExperimentalResearchesinElectricity,3:447Œ452,1846.[118]A.Gershun.ThelightTranslatedbyP.MoonandG.TimoshenkoinJournalofMathematicsandPhysics,18:51Œ151,1939.[119]R.Ng.Fourierslicephotography.ACMTrans.Graph.,24(3):735Œ744,2005.164[120]GordonWetzstein,DouglasLanman,MatthewHirsch,andRameshRaskar.Tensordisplays:Compressivelightsynthesisusingmultilayerdisplayswithdirectionalbacklighting.ACMTrans.Graph.,31:1Œ11,2012.[121]A.LumsdaineandT.Georgiew.Thefocusedplenopticcamera.InInProc.ICCP.,pages1Œ8.ACM,2009.[122]Chia-KaiLiang,Tai-HsuLin,Bing-YiWong,ChiLiu,andHomerH.Chen.Programmableaperturephotography:Multiplexedlightacquisition.ACMTrans.Graph.,27(3):1Œ10,2008.[123]M.Kamal,M.Golbabaee,andP.Vandergheynst.Lightcompressivesensingincameraarrays.InInProc.ICASSP.,pages5413Œ5416.ACM,2008.[124]EdwardH.AdelsonandJohnY.A.Wang.Singlelensstereowithaplenopticcamera.IEEETRANSACTIONSONPATTERNANALYSISANDMACHINEINTELLIGENCE,14(2):99Œ106,1992.[125]MarcLevoy,BillyChen,VaibhavVaish,MarkHorowitz,IanMcDowall,andMarkBo-las.Syntheticapertureconfocalimaging.AcmTransactionsonGraphics(Proceedingsofsiggraph2004),23(3):822Œ831,2004.[126]M.LevoyandP.Hanrahan.rendering.InProc.SIGGRAPH,pages31Œ42.ACMPress,1996.[127]S.Gortler,R.Grzeszczuk,R.Szeliski,andM.F.Cohen.Thelumigraph.InProc.SIG-GRAPH,pages43Œ54.ACMPress,1996.[128]ChengboLi.AnEfAlgorithmForTotalVariationRegularizationwithApplicationstotheSinglePixelCameraandCompressiveSensing.RiceUniversity,2009,2009.[129]D.GoldfarbandW.Yin.Second-orderconeprogrammingmethodsfortotalvariationbasedimagerestoration.SIAMJournalonComputing,27:622Œ645,2005.[130]J.Bioucas-DiasandM.Figueiredo.AnewTwIST:Two-stepiterativethresholdingalgo-rithmforimagerestoration.IEEETrans.Imag.Process.,16:2992Œ3004,2007.[131]JunfengYang,YinZhang,andWotaoYin.AfastalternatingdirectionmethodforTVL1-L2signalreconstructionfrompartialfourierdata.IEEEJOURNALOFSELECTEDTOPICSINSIGNALPROCESSING,4(2):288Œ297,2010.165[132]J.HarguessaandS.Strangeb.Infraredstereocalibrationforunmannedgroundvehiclenavigation.InProc.SPIE9084,UnmannedSystemsTechnologyXVI,pages1Œ8.SPIE,2014.[133]P.ChakravartyandR.Jarvis.Panoramicvisionandlaserrangefusionformultiplepersontracking.InProc.IEEE/RSJConfofIntelligentRobotsandSystems,Beijing,China,pages2949Œ2954.IEEE,2006.[134]JeremyM.Wolfe.Stereopsisandbinocularrivalry.PsychologicalReview,93(3):269Œ282,1986.[135]D.Geronimo,A.M.Lopez,A.D.Sappa,andT.Graf.Surveyofpedestriandetectionforadvanceddriverassistancesystems.IEEETrans.onPatternAnalysisandMachineIntelli-gence,32(7):1239Œ1258,2010.[136]ReinhardKoch,IngoSchiller,BogumilBartczak,FalkoKellner,andKevinKoser.MixIn3D:3DmixedrealitywithToF-camera.Dynamic3DImaging,5742(1):126Œ141,2010.[137]B.Huhle,P.Jenke,andW.Strasser.sceneacquisitionwithahandymulti-sensorsystem.InternationalJournalofIntelligentSystemsTechnologiesandApplications,5(3):255Œ263,2008.[138]M.Hansard,S.Lee,O.Choi,andR.Horaud.TimeofFlightCameras:Principles,Methods,andApplications.SpringerBriefsinComputerScience,2012.[139]M.Fahle.Wozuzweiaugen?[whytwoeyes?].Naturwissenschaften,74:383Œ385,1987.[140]R.BlakeandR.Sekuler.Perception(5thed.).NewYork,NY:McGraw-Hill,2006.[141]M.DoandM.Vetterli.Theridgelettransformforimagerepresentation.IEEETrans-actionsonInformationTheory,12:16Œ28,2003.[142]T.Komatsu,K.Aizawa,T.Igarashi,andT.Saito.Signal-processingbasedmethodforacquiringveryhighresolutionimagewithmultiplecamerasanditstheoreticalanalysis.Proc.Inst.Elec.Eng.,140(1):19Œ25,1993.[143]S.Park,M.Park,andM.Kang.Super-resolutionimagereconstruction:atechnicaloverview.SignalProcessingMagazine,IEEE,20(3):21Œ36,2003.166[144]M.IraniandS.Peleg.Superresolutionfromimagesequences.InProceedingsoftheInternationalConferenceonPatternRecognition,pages115Œ120.IEEE,1990.[145]R.R.SchultzandR.L.Stevenson.Extractionofhigh-resolutionframesfromvideose-quences.IEEETransactionsonImageProcessing,5(6):996Œ1011,1996.[146]M.K.Ng,H.Shen,E.Y.Lam,andL.Zhang.Atotalvariationregularizationbasedsuper-resolutionreconstructionalgorithmfordigitalvideo.EURASIPJournalonAdvancesinSignalProcessing,1:1Œ16,2007.[147]M.V.Joshi,S.Chaudhuri,andR.Panuganti.Alearning-basedmethodforimagesuper-resolutionfromzoomedobservations.IEEETransactionsonSystems,Man,andCybernet-ics,35(3):527Œ537,2005.[148]A.N.RajagopalanandS.Chaudhuri.Space-variantapproachestorecoveryofdepthfromdefocusedimages.ComputerVisionandImageUnderstanding,68:309Œ329,1997.[149]H.Shen,L.Zhang,B.Huang,andP.Li.AMAPapproachforjointmotionestimation,segmentationandsuper-resolution.IEEETransactionsonImageProcessing,16(2):479Œ490,2007.[150]M.Protter,M.Elad,H.Takeda,andP.Milanfar.Generalizingthenonlocal-meanstosuper-resolutionreconstruction.IEEETransactionsonImageProcessing,18(1):36Œ51,2009.[151]N.Nguyen,P.Milanfar,andG.Golub.Acomputationallyefsuperresolutionimagereconstructionalgorithm.IEEETransactionsonImageProcessing,10:573Œ583,2001.[152]D.Glasner,S.Bagon,andM.Irani.Super-resolutionfromasingleimage.InInternationalConferenceonComputerVision(ICCV),pages1Œ8.IEEE,2009.[153]X.LiandM.Orchard.Newedge-directedinterpolation.IEEETrans.ImageProcessing,10:1521Œ1527,2001.[154]N.Mueller,Y.Lu,andM.Do.Imageinterpolationusingmultiscalegeometricrepresen-tations.InSPIEproceedingsComputationalImagingV,volume6498,pages1Œ11.SPIE,2007.[155]L.ZhangandX.Wu.Anedge-guidedimageinterpolationalgorithmviadirectionalanddatafusion.IEEETransactionsonImageProcessing,15:2226Œ2238,2006.167[156]R.Fattal.Imageupsamplingviaimposededgestatistics.ACMTransactionsonGraphics,26:1,2007.[157]J.Sun,J.Sun,Z.Xu,andH.Y.Shum.Imagesuper-resolutionusinggradientprior.In2008CVPR,pages1Œ8.ACM,2008.[158]Q.Xie,H.Chen,andH.Cao.Improvedexample-basedsingle-imagesuperresolution.In-ternationalCongressonImageandSignalProcessing,3:1204Œ1207,2010.[159]J.Yang,J.Wright,T.Huang,andY.Ma.Imagesuper-resolutionviasparserepresentation.IEEEtransactionsonimageprocessing,19:2861Œ1873,2010.[160]W.Yin,S.Osher,D.Goldfarb,andJ.Darbon.Bregmaniterativealgorithmfor`1mini-mizationwithapplicationstocompressedsensing.SIAMJ.ImagingSciences,1:143Œ168,2008.[161]E.T.Hale,W.Yin,andY.Zhang.Fixed-pointcontinuationfor`1-minimization:Methodol-ogyandconvergence.SIAMJournalonOptimization,19:1107Œ1130,2008.[162]L.Bregman.Therelaxationmethodofthecommonpointofconvexsetsanditsapplicationtothesolutionofproblemsinconvexprogramming.USSRComputationMath-ematicsandMathematicalPhysics,7:200Œ217,1967.[163]Y.LiandS.Osher.Coordinatedescentoptimizationfor`1minimizationwithapplicationtocompressedsensing;agreedyalgorithm.InverseProblemsandImaging,3:487Œ503,2009.[164]R.Chartrand.Fastalgorithmsfornonconvexcompressivesensing:MRIreconstructionfromveryfewdata.BiomedicalImaging:FromNanotoMacro,2009.ISBI,1:262Œ265,2009.[165]L.Bai,P.Maechler,M.Muehlberghuber,andH.Kaeslin.High-speedcompressedsensingreconstructiononFPGAusingOMPandAMP.IninElectronics,CircuitsandSystems(ICECS),19thIEEEInternationalConferenceon,pages53Œ57.IEEE,2012.[166]JinyuChu,JuLiu,JianpingQiao,XiaolingWang,andYujunLi.Gradient-basedadaptiveinterpolationinsuper-resolutionimagerestoration.InSignalProcessing,2008.ICSP2008.9thInternationalConferenceon,pages1027Œ1030.IEEE,2008.[167]LiangjianDeng,WeihongGuo,andTingzhuHuang.Singleimagesuper-resolutionviaaniterativereproducingkernelhilbertspacemethod.IEEETransactionsonCircuitsandSystemsforVideoTechnology,1051:1Œ14,2015.168[168]G.Wahba.Splinemodelsforobservationaldata.SIAM.RegionalConferenceSeriesinAppliedMathematics,59,1990.[169]Y.Wang,J.Yang,W.Yin,andY.Zhang.Anewalternatingminimizationalgorithmfortotalvariationimagereconstruction.SIAMJournalonImagingSciences,1:248Œ272,2008.[170]T.Vaughan.Multimedia:MakingitWork(4thed.).Berkeley,CA:McGraw-Hill,1998.[171]M.Irvin,T.Kitazawa,andT.Suzuki.AnewgenerationofMPEG-2videoencoderASICan-ditsapplicationtonewtechnologymarkets.BroadcastingConvention,International,1:391Œ396,1996.[172]G.J.SULLIVANandT.WIEGAND.VideocompressionfromconceptstotheH.264/AVCstandard.PROCEEDINGSOFTHEIEEE,93(1):18Œ31,2005.[173]http://www.gigapan.com/gigapans/102753.Gigapan,2015.[174]M.Hepher,D.Duckett,andA.Loening.High-speedvideomicroscopyandcomputeren-hancedimageryinthepursuitofbubbledynamics.UltrasonSonochem.,7:229Œ33,2000.[175]Q.Cossairt,M.Gupta,andS.Nayar.Whendoescomputationalimagingimproveperfor-mance?IEEETRANSACTIONSONIMAGEPROCESSING,22(2):447,2013.[176]V.Stankovic,L.Stankovic,andS.Cheng.Compressivevideosampling.In16thEur.SignalProcessingConf.,pages1Œ6,2008.[177]J.ZhengandE.L.Jacobs.Videocompressivesensingusingspatialdomainsparsity.Opt.Eng.,48:087006,2009.[178]A.Veeraraghavan,D.Reddy,andR.Raskar.Codedstrobingphotography:Compressivesensingofhighspeedperiodicvideos.IEEETransactionsonPatternAnalysisandMachineIntelligence,33(4):671Œ686,2011.[179]A.Sankaranarayanan,P.Turaga,R.Baraniuk,andR.Chellappa.Compressiveacquisitionofdynamicscenes.IninProc.11thEuropeanConf.onComput.Vis.,pages129Œ142,2010.[180]M.LaiandN.Kingsbury.Codingofimagesequencesusingcoarsequantizationandfeaturebasedhierarchicalblockmatching.InInternationalConferenceonImageProcessinganditsApplications,pages586Œ589,1992.169[181]N.NasrabadiandR.King.Imagecodingusingvectorquantization:Areview.IEEETRANS-ACTIONSONCOMMUNICATIONS,36(8):957Œ971,1988.[182]D.HeinandN.Ahmed.Videocompressionusingconditionalreplenishmentandmotionprediction.IEEETRANSACTIONSONELECTROMAGNETICCOMPATIBILITY,EMC-26(3):134Œ142,1984.[183]E.Chan,A.Rodriguez,R.Ghandi,andS.Panchanathan.Experimentsonblock-matchingtechniquesforvideocoding.MultimediaSystems,2(5):228Œ241,1994.[184]D.Lin,C.Chen,andT.Hsing.Videoonphonelines:technologyandapplications.Pro-ceedingsoftheIEEE,83(2):175Œ193,1995.[185]F.LauuauretandD.Barba.Motioncompensationbyblockmatchingandvectorpost-processinginsub-bandcodingofTVsignalsat15Mbit/s.ProceedingsofSPIE,1605:26Œ36,1991.[186]RayanSaabandOzgurYilmaz.Sparserecoverybynon-convexoptimization-instanceopti-mality.Appl.Comput.Harmon.Anal.,29(1):30Œ48,2010.[187]Z.Xu,X.Chang,F.Xu,andH.Zhang.`1=2regularization:athresholdingrepresentationtheoryandafastsolver.IEEETrans.NeuralNetw.Learn.Syst.,23(7):1013Œ1027,2012.[188]Xiao-LinHuang,LeiShi,andMingYan.Nonconvexsorted`1minimizationforsparseapproximation.J.Oper.Res.Soc.China,3:207Œ229,2015.[189]E.J.Candes,M.B.Wakin,andS.P.Boyd.Enhancingsparsitybyreweighted`1minimiza-tion.J.FourierAnal.Appl.,14(5):877Œ905,2008.[190]G.Gasso,A.Rakotomamonjy,andS.Canu.RecoveringsparsesignalswithacertainfamilyofnonconvexpenaltiesandDCprogramming.IEEETransactionsonSignalProcessing,57(12):4686Œ4698,2009.[191]R.Chartrand.Exactreconstructionofsparsesignalsvianonconvexminimization.IEEESignalnProcessingLetters,14(10):707Œ710,2007.[192]YilunWangandWotaoYin.Sparsesignalreconstructionviaiterativesupportdetection.SIAMJ.ImagingSci.,3(3):462Œ491,2010.[193]O.R.VincentandO.Folorunso.Adescriptivealgorithmforsobelimageedgedetection.InProceedingsofInformingScienceandITEducationConference,pages1Œ11,2009.170[194]G.T.Shrivakshan.Acomparisonofvariousedgedetectiontechniquesusedinimagepro-cessing.InternationalJournalofComputerScienceIssues,9(1):269Œ276,2012.[195]J.YangandY.Zhang.Alternatingdirectionalgorithmsfor`1-problemsincompressivesensing.SIAMJournalonComputing,33(1):250Œ278,2011.171