LIDARANDCAMERACALIBRATIONUSINGAMOUNTEDSPHERE By JiajiaLi ATHESIS Submittedto MichiganStateUniversity inpartialful˝llmentoftherequirements forthedegreeof ElectricalEngineeringMasterofScience 2020 ABSTRACT LIDARANDCAMERACALIBRATIONUSINGAMOUNTEDSPHERE By JiajiaLi Extrinsiccalibrationbetweenlidarandcamerasensorsisneededformulti-modalsensordata fusion.However,obtainingpreciseextrinsiccalibrationcanbetedious,computationallyexpensive, orinvolveelaborateapparatus.Thisthesisproposesasimple,fast,androbustmethodperforming extrinsiccalibrationbetweenacameraandlidar.Theonlyrequiredcalibrationtargetisahand-held coloredspheremountedonawhiteboard.Theconvolutionalneuralnetworksaredevelopedto automaticallylocalizethesphererelativetothecameraandthelidar.Thenusingthelocalization covariancemodels,therelativeposebetweenthecameraandlidarisderived.Toevaluatethe accuracyofourmethod,werecordimageandlidardataofasphereatasetofknowngrid positionsbyusingtworailsmountedonawall.Theaccuratecalibrationresultsaredemonstrated byprojectingthegridcentersintothecameraimageplaneand˝ndingtheerrorbetweenthese pointsandthehand-labeledspherecenters. ACKNOWLEDGMENTS The˝rstpersonIshouldexpressmydeepgratitudeismyadvisor,Dr.DanielMorris,forhis utmostguidance,advice,support,patienceandencouragementIreceivedthroughoutmyresearch. Heintroducedmeintotheautonomousdrivingsystemandhelpedmytremendouslyfromthe˝rst dayIjoinedthegroupofthe3Dvisionlab.Hisconstantenthusiasmandpatienceinanswering questionsduringtheresearchhasbeenarealinspirationtome.Iamgrateful,thankful,andhonored forallthevaluablelearningandadvicehehasgivenme.HeservesasarolemodelforwhatIwant tobewhetherinmycareerorlife.Iwishhimbestforhishealth,workandfamily. Besides,Iwanttothankmycommitteemembers,Dr.HayderRadhaandDr.ZhaojianLi,for theirvaluableadviceandhelp.IappreciateallECEdepartmentadministratorsandsta˙members fortheirhelp.IamalsogratefultotheO˚ceforInternationalStudentsandScholars,whichislike abigfamily.AhugethankstoallmyfriendsandcolleaguesofUSandChinawhohaveassisted andsupportedmeinmanywaysduringmystudy. Lastandcertainlynotleast,Iwanttothankmyfamilies:myparents,mybrother,myauntsand myunclesfortheirendlessandunconditionallove,faithandsupportthattheyhavealwaysgiven me. iii TABLEOFCONTENTS LISTOFTABLES ....................................... vi LISTOFFIGURES ....................................... vii CHAPTER1INTRODUCTION ............................... 1 1.1Background......................................1 1.2IntroductiontoExtrinsicCalibration.........................2 1.3Contribution......................................3 1.4ScopeandOrganization................................4 CHAPTER2RELATEDWORK ............................... 6 2.1Calibrationwithouttargets..............................6 2.2Calibrationwithtargets................................6 CHAPTER3IMAGE-BASEDSPHEREDETECTIONAND3DESTIMATION ..... 9 3.1SphereandBoard...................................9 3.2SphereProjection...................................10 3.3ProjectedSphereCenterandAreaEstimation....................11 3.3.1Area-Net...................................11 3.3.2Center-Net...................................13 3.3.3NetworkandTraining.............................13 3.3.4LossFunction.................................14 3.3.4.1Area-NetWeights.........................15 3.3.4.2Center-NetWeights........................15 3.4CovarianceModelforCamera............................16 CHAPTER4LIDAR-BASEDSPHEREDETECTIONAND3DESTIMATION ...... 17 4.1Lidar-BasedSphereDetection............................17 4.2SphereCenterEstimationfromPointCloud.....................18 4.3CovarianceModelforLidar..............................19 CHAPTER5SOLVEFOREXTRINSICPARAMETERS .................. 20 5.1Point-to-PointDistance................................20 5.2Point-to-RayDistance.................................21 5.3Point-to-PointDistancewithCovarianceModels...................21 CHAPTER6EXPERIMENTSANDRESULTS ....................... 23 6.1Image-BasedSphereDetectionand3DEstimation..................23 6.2Lidar-BasedSphereDetectionand3DEstimation..................24 6.3CalibrationResults..................................24 6.4MethodValidation...................................24 iv CHAPTER7CONCLUSIONANDFUTUREWORK ................... 28 APPENDICES ......................................... 29 APPENDIXATHECAMERACOVARIANCEMODEL ................ 30 APPENDIXBTHELIDARCOVARIANCEMODEL ................. 33 BIBLIOGRAPHY ........................................ 34 v LISTOFTABLES Table6.1:Estimationaccuracyforspherearea(A)andcenter ¹ G 2 ŒH 2 º measuredinpixels atdi˙eringdepthsmeasuredonmeanabsoluteerror(MAE)andstandard deviation( f ).....................................23 Table6.2:ThemeanprojectederrorinpixelsforagridpatterninFig6.1usedtocom- parealignmentmethods.Extrinsiccalibrationerrorsresultinprojectedgrid alignmenterrors.ThespherecenterestimationisbasedoncostEq.(4.1). Cov1isbasedonthelidarcovariancemodelinEq.(4.3)andCov2isbasedon Eq.(4.4).P2R isthepoint-to-raydistanceusedin[16,17]............25 vi LISTOFFIGURES Figure1.1:Theautonomousvehicleusedinourextrinsiccalibrationexperimentsequipped withacameraandalidar.............................2 Figure1.2:Illustrationforextrinsiccalibrationbetweensensors...............3 Figure1.3:Fastandsimpleextrinsiccalibrationfromaspheremovedinfrontofalidar- camerapair.(a)Oneimagefromthesequence,withasubsetofdetected spherelocationsplottedonit.(b)3Dspherelocationsandcovariances estimatedfromtheimagesequence.(c)Correspondingdetected3Dlidar spherelocationsandcovariances.(d)and(e)Extrinsiccalibrationthrough covariance-basedalignmentofpoints.......................5 Figure2.1:Spheredetectionand3Destimationpipelines.(a)Colorimagesareundis- torted,andCNNsaretrainedtoclassifyspherepixelsandspherecenters. Fromtheareaandimagecenter,thesphere3Dcenterofeachsphereises- timated[24].(b)Lidarpointcloudsaretransformedintorangeimagesin whichaCNNidenti˝espixelsbelongingtothesphere.Usingthesepixelsin thepointclouds,thespherecentersareestimated.................8 Figure3.1:Thecalibrationtargetusedinextrinsiccalibration................9 Figure3.2:Asphereprojectsasanellipseinthecameraplane.Knowingtheimagecenter andimageareaissu˚cienttoestimatethe3Dlocationofthesphere.......10 Figure3.3:Anexampleofimage-basedspheredetectionwithCNNs.(a)TheRGB image.(b)Abinarylabelingofsphereandnon-spherepixels.(c)The Area-Net outputpixelclassi˝cationwithanaverageprecisionof0.998.From thisweestimatetheprojectedspherearea.(d)Thespherecenterlabelas anexponentialfunctionaroundtheprojectedcenter,seeEq.(3.5).(e)The Center-Net prediction,fromwhichweusethepeaklocationtodeterminethe sphere-centerprojection..............................12 Figure3.4:Thenetworkarchitecture.Theyellowboxesdenotethe8-channeloutputs ofthe 3 3 convolutional˝lters.Downsamplingbytwoisperformedby max-poolingandupsamplingbynearest-neighbor.Skipconnectionsare element-wisesummedwiththedecodingblock..................14 Figure3.5:IllustrationforCenter-NetandArea-Netweights.Thepixelswithintheregion oftheyellowellipseareconsideredashigh-importancepixels.(a)Area-Net. (b)Center-Net....................................15 vii Figure4.1:Lidarspheredetection:(a)Lidarrangeimage.(b)Groundtruthlabelimage. (c)Lidarpixelclassi˝cationby Area-Net trainedonlidarrangeimages......17 Figure4.2:IllustrationofthecostEq.(4.1),usedin˝ttingasphereofknownradiusto lidarpixelsdetectedinFig.4.1..........................18 Figure6.1:Extrinsiccalibrationaccuracyisvalidatedusingaspheremovedthrougha setofknownlocationsintheworld........................25 Figure6.2:Methodvalidationillustration...........................26 Figure6.3:Asampleofthebinaryclassi˝cation(areaestimation),centerestimation,and comparisonswithgroundtruthsforthecamera.(a)RGBimage.(b)Binary label.(c)Binaryclassi˝cation.(d)Centerlabel.(e)CenterEstimation.....26 Figure6.4:Asampleofthebinaryclassi˝cation,andcomparisonswithgroundtruthsfor thelidar.(a)Rangeimage/m.(b)Binarylabel.(c)Binaryclassi˝cation....27 viii CHAPTER1 INTRODUCTION 1.1Background Camerasandlidars[1,2]arewidelyusedinautonomousdrivingsystems,suchascollision avoidanceandobjectdetection.However,thesetwokindsofsensorshaveadvantagesanddisad- vantages.Theadvantageofthecameraisitslowcost,andcamera-basedperceptiontechnologyis relativelymature.Still,itishardforthecameratoobtaindepthinformation.Anotherdisadvantage isthatitisrelativelylimitedbyambientlight.Thestrongshadoworbrightlightsfromthesunor oncomingcarsmaycauseconfusion,whichisextremelydangerousinthesystemofself-driving cars.Theadvantageoflidaristhatitgivestheself-drivingcarsalongdetectionrangeandcan accuratelyobtainthree-dimensionalinformationforanobject.Inaddition,ithashighstabilityand goodrobustnesscomparedtocamerasbecausethelidarsarenotfooledbyshadows,brightsunlight, ortheoncomingheadlightsofothercars.However,thecurrentlidarcostisrelativelyhigh,and lidardoesnotprovideinformationthatcamerascantypicallysee-likethewordsonasignorthe stoplightcolor. OneviablesolutiontotheLiDARversuscameradebateistocombinethetechnology,whichhas becomeanincreasinglyimportantpartoftheself-drivingcarssystem.Heterogeneoussensors,such ascamerasandlidars,generatecomplementarydatathatcanbecombinedtoimprovetheoverall accuracy,robustness,andperformanceoftaskssuchasobjectdetectionandsceneestimation[3,4, 2]. Anessentialrequirementofsensorfusionisthatpreciseextrinsiccalibrationisknown.The intrinsiccalibrationparametersde˝netheopticalcenterandfocallengthofthecamera.The extrinsicparametersrepresenttheorientationandthepositionofthesensorswithrespecttoeach other.Inthecaseoflidarandcamerasensors,extrinsiccalibration[5]enableslidarpixels,clusters, anddetectionstobeprojectedintothecameraimagewherefusionisoftenperformed.Now 1 Figure1.1:Theautonomousvehicleusedinourextrinsiccalibrationexperimentsequippedwitha cameraandalidar wheneversensorposesareadjustedoraccidentallyshifted,theextrinsicparametersneedtobe re-estimated,whichcanbefrequentduringproductdevelopment.Estimatingextrinsiccalibration inafastandsimplewayisthegoalofthispaper. 1.2IntroductiontoExtrinsicCalibration Transformingapoint ( 8 giveninthesensor1coordinatesystemintoapoint ( 9 inanothersensor 2coordinatesystemistypicallymodeledviaanEuclideana˚netransformationmatrix » 'ŒC ¼ : ( 9 = '( 8 ¸ C (1.1) Thetaskofestimatingthetransformationmatrix » 'ŒC ¼ iscalledextrinsiccalibrationandhasbeen studiedforavarietyofsensormodalitiesandcombinations. Ade˝ningcharacteristicofextrinsiccalibrationmethodsisthefeaturetypeusedforalignment. Givenasu˚cientnumberoffeaturesdetectedintwosensorsandtheirassociation,therelative posebetweensensorscanbecalculated.However,˝ndingcorrespondingpointsthatalignwell 2 Figure1.2:Illustrationforextrinsiccalibrationbetweensensors. between2Dcolorimagesandmostlytextureless3Dscansismorechallengingthanrequiredby 2D-to-2Dcameraalignmentor3D-to-3Dlidaralignment.Targetlessmethodsseekfeaturesin thesensorplatform'senvironment,suchasdepthedgefeatures[6]orleveragemutualinformation betweenlidarre˛ectanceorshapeandcameraintensity[7,8].Thesemethodsmustovercomethe environment'svagariesandincorrectmatchesandsotypicallydependonlargenumbersofscans andcomplexoptimization.Thealternativeistousetargetsthatfacilitatematchingbetweencamera imagesandlidarscans.Avarietyoftargetshavebeenproposedforthisincludingcheckerboards[9, 10,11],polygonalplanarshapes[12],cubes[13],cardboardwithholes[14,15]andspheres[16, 17].Thesemethodsleveragethegeometryandtextureofthetargetstomakefeaturedetection simpleandalignmentunambiguous.Nevertheless,thesemethodsrequiresigni˝cantmanuale˙ort toachieveprecisecalibrationorcontrolledenvironmentswithspeciallighting. Ideally,calibrationshouldbesimpletoimplement,inexpensivetosetup,portable,fastto perform,robusttopossiblefailures,andaccurate.Unfortunately,currentmethodsperformpoorly ononeormoreofthesecriteria.Consequently,performingcalibrationcanentailsigni˝cante˙ort andcostanddelayprojectsthatneedpreciselyalignedcamerasandlidars. 1.3Contribution Toovercomethedrawbacksoftheexistingworks,weproposeafast,inexpensive,androbust extrinsiccalibrationmethodusingasingle-colorspheremountedonawhiteboardasthecalibration target.Thereareseveraladvantagestousingasphere.Asphere'sappearanceisidenticalregardless 3 ofviewingdirectionandorientation,enablingasimpleautomaticdetectionmethodthatcanbe trainedonasmalldataset.Also,anacrylicspherecanbeinexpensive,light-weight,andeasily re-positionedbyhand.Mostimportantly,whilethespherecentercannotbeobserveddirectly,its 3Dlocationcanbereadilyestimatedusingasinglecameraimageandasinglelidarscan.Inthis way,asphereobservationprovidesasingle3Dcorrespondencepointbetweenacameraandlidar. Movingaspherethroughtheoverlapping˝eldofviewofthesensorsprovidesatrajectoryofmany 3Dpointcorrespondencestoperformextrinsiccalibration.Thisthesismakesthefollowingthree contributions: 1. Automatedsphere-targetdetectionand3Dlocationinbothimagesandlidarscans.The CNNsareusedtoautomaticallydetectasphereandestimatethe3Dpositionofitscenter. 2. Estimationof3Dspherelocationcovariancemodelsforcamearandlidarandincorporation ofthesetoimprovealignmentaccuracy. 3. Automatedandrobust-to-lightingcamera-lidarextrinsiccalibrationusingaboard-mounted sphere. Ourapproachprovidesasolutiontoanimportantproblemfacingroboticplatformswithmultiple heterogeneoussensors.Thesamemethodcaneasilybeextendedtoaligningmoresensors. 1.4ScopeandOrganization Ourextrinsiccalibrationisbasedonestimatingthelocationofa3Dspherecenterrelativeto boththecameraandthelidarasshowninFig.2.1,andaligningthembasedontheMahalanobis distance.Eachstepisexplainedindetailinthefollowingchapters.Relatedworkisintroduced inChapter2.Chapters3and4presentthemethodsof3Dspheredetectionandestimationin cameraandlidar.Chapter5explainsthemethodsofsolvingfortheextrinsicparameters.Chapter6 showstheimageandlidarbasedspheredetectionand3Destimationresults,calibrationandmethod validationresults,andchapter7istheconclusion. 4 (a)Detectedspherelocationsasholdermoves (b) 8 f cameraellipsoids (c) 60 f lidarellipsoids (d)Extrinsiccalibrationbyaligningsphere centers (e)Extrinsiccalibrationbyaligningsphere centers Figure1.3:Fastandsimpleextrinsiccalibrationfromaspheremovedinfrontofalidar-camera pair.(a)Oneimagefromthesequence,withasubsetofdetectedspherelocationsplottedonit. (b)3Dspherelocationsandcovariancesestimatedfromtheimagesequence.(c)Corresponding detected3Dlidarspherelocationsandcovariances.(d)and(e)Extrinsiccalibrationthrough covariance-basedalignmentofpoints. 5 CHAPTER2 RELATEDWORK Overthepastseveralyears,theextrinsiccalibrationbetweensensorshasbeenwidelystudied. Ingeneral,thetwocategoriesoftheextrinsiccalibrationbetweensensorsarebasedon(1)targetless and(2)withtargets.Consideringtheadvantagesofthesphereastargets,theextrinsiccalibration withtargetsareclusteredintotwocategories:(1)non-sphericaltargetsand(2)sphereasthetarget. 2.1Calibrationwithouttargets Whilethepresentworkusesatargetforcalibration,therearealternativeapproachesthateschew targets.Instead,thesemethodsrelyonthesensorplatformmovingthroughanunknownnatural environment.Poseparametersareestimatedthroughmatchingavarietyoffeaturesfromdepth edges[6,18]todensemutualinformation[7,8].Onthepositiveside,thesemethodscanbefully automatedandachievehighaccuracy.However,performancewilldependontheenvironmentas someobjectslikevegetationmaycreateexcessiveclutter.Asaresult,con˝denceinthecalibration willdependoncollectingalarge,diversedataset,whichprecludesfastcalibration.Withtargets, ontheotherhand,onecanquicklygathersu˚cientdataforanaccuratecalibration. 2.2Calibrationwithtargets Thecheckerboardhasbeenapopularcalibrationtargetforthecameraanddepthsensors[9, 10,11].Cornersaredetectedtoestimatetheposerelativetothecamera.Thecheckerboard's pointcloudinthedepthsensorcanbemodelledasaplane,whichcanbeusedasaconstraint tosolvethecalibrationproblem.Forrobustness,thecheckerboardshouldhavewell-distributed orientations.Still,itmaybehardtomakeitformultiplecamerascalibrationwithawidebaseline andencountertheproblemofsimultaneousvisibility.Furthermore,theblackandwhiteareaona checkerboardcanhaveasigni˝canto˙setin-depthasthere˛ectivityoftheobjectsurfacein˛uences thedepthsensormeasuringresults[12].Parketal.[12]proposedtouseapolygonalplanarboard 6 withahomogeneoussurfacetoreducethise˙ectand˝ndpoint-to-pointcorrespondencesbetween 2Dimageand3Dpointcloudsforcalibration.Alimitingfactorofsuchkindofapproachisthe precisionofthecorneroredgedetectionindepthsensorduetoitslowerresolutionthanthecamera. Inaccuratefeaturedetectioninthedepthsensoralsooccursforplanartargetswithholesin[15,14]. Toovercometheproblemofpossibleoccludedfeaturesofcalibrationtarget,thesphereisused asacalibrationtargetasithasaconsistentappearancewhenobservedfromarbitrarydirections. Asphereisanattractivecalibrationtargetasithasanidenticalappearanceregardlessofviewing directionandcanbeinexpensiveandportable.Sphereshavebeenusedforsinglecameraintrinsic calibration[19,20,21]andmulti-cameraextrinsiccalibrationasin[22,23]wheretherelative rotationandtranslationisobtainedby3Dspherecenterregistration.Theaccuracyofbothcamera calibrationmethodsreliesontheaccuracyoftheestimatedconics.Tosolvethisproblem,Guan etal.[24]proposedamethodtoestimatethe3Dpositionofthespherecenterbyonlyusing theprojectedellipsecenteranditsarea.Theaccuracyoftheextrinsiccameracalibrationhighly dependsonthesphereextraction,whichisconductedinadarkroomwithaluminescentsphere underastrictlightingenvironment. Sphereshavebeenusedforposeestimationbetweenlidars[25,26,27]byminimizingapoint- to-raydistanceerror.Ruanetal.[27]proposedacalibrationmethodthatisbasedonahand-held sphericaltarget.Thealgorithm˝rstdetectsthesphereandsegmentsitwitha2.5Dmorphological operation.Theclusterofthepointcloudis˝ttedtoestimatethe3Dspherecenterbynon-linear optimization.However,thisapproach[27]isonlyusedwithKinectsensorswithasmallerballand areducedworkingrange. Ourproposedapproachismostsimilartothatof[16,17]whouseaspheretocalibratecameras withdepthsensors.Whileautomated,[17]usesaHoughTransformonedgefeaturesandissensitive tothelightingofboththesphereandbackground.Toovercomesensitivitytolighting,wemount asphereonawhiteboardanduseConvolutionalNeuralNetworks(CNNs)tolocatethespherein bothcameraandlidarimagesrobustly.Wesuccessfullydopoint-to-pointalignmentratherthan point-to-rayalignment[16,17],byleveragingcovariancemodelsforcameraandlidarrespectively. 7 (a)Image-basedspheredetectionand3Destimation (b)Lidar-basedspheredetectionand3Destimation Figure2.1:Spheredetectionand3Destimationpipelines.(a)Colorimagesareundistorted,and CNNsaretrainedtoclassifyspherepixelsandspherecenters.Fromtheareaandimagecenter, thesphere3Dcenterofeachsphereisestimated[24].(b)Lidarpointcloudsaretransformedinto rangeimagesinwhichaCNNidenti˝espixelsbelongingtothesphere.Usingthesepixelsinthe pointclouds,thespherecentersareestimated. 8 CHAPTER3 IMAGE-BASEDSPHEREDETECTIONAND3DESTIMATION Tomakespheredetectionandlocalizationbothautomaticandrobusttovaryinglightingand environments,CNNsaredevelopedtoestimateitsprojectedcenterandarea.We˝rstderivehow 3Dlocationcanbecalculatedfromprojectedspherecenteranditsarea,andthenproposetwo CNNstoestimatethem. 3.1SphereandBoard Ourgoalistodevelopafastandstraightforwardcalibrationprocedurethatdoesnotrequire speciallabequipment.Weselectaninexpensive,clear,lightweightacrylicsphereofdiameter 45 2< andpaintitgreen.Toenablesharpcontrastatitsimageboundary,wea˚xathinwhiteboard andhandlebehindit.Inthisway,wedonotrequireahigh-contrastlabbackgroundasin[17].The board-mountedsphereishand-heldandmovedthroughthecameraandlidar'soverlapping˝eldof view. Figure3.1:Thecalibrationtargetusedinextrinsiccalibration 9 3.2SphereProjection Fig.3.2illustratestheperspectiveprojectionofasphereontoanellipseinanimage.Wefollow thegeometricestimationmethodinGuanetal.[24]toobtainthe3Dposition, ( 2 = ¹ - 2 Œ. 2 Œ/ 2 º , ofaspherecenter, ( 2 .Weassumethesphereradius A isknown.Weassumeanundistortedimage withknownfocallengths 5 G and 5 H andopticalcenter ¹ G 0 ŒH 0 º . Figure3.2:Asphereprojectsasanellipseinthecameraplane.Knowingtheimagecenterand imageareaissu˚cienttoestimatethe3Dlocationofthesphere. Asderivedin[24],thedepth / 2 ofthespherecanbeobtained: / 2 = A s c5 G 5 H cos ¹ \ º Ł (3.1) Here istheareainpixelsoftheellipticalprojectionofthesphere,and \ istheanglebetweenthe raythroughthecenterofthesphereandtheopticalaxis. Thiscanberefactoredasfollows.Let D 2 and E 2 betheunit-focalplanecoordinatesofthe projectedspherecenter ¹ G 2 ŒH 2 º . D 2 = ¹ G 2 G 0 ºš 5 G ŒE 2 = ¹ H 2 H 0 ºš 5 H Ł (3.2) 10 Thenwecande˝ne F = q D 2 2 ¸ E 2 2 ¸ 1 andso cos ¹ \ º = 1 š F .Thisleadstothefollowing expressionforspheredepth: / 2 = : r F Ł (3.3) where : = A p c5 G 5 H .Usingthestandardperspectiveequation,wecanexpressthefullequationfor thespherecenteras: ( 2 ¹ - 2 Œ. 2 Œ/ 2 º = : r F © « D 2 E 2 1 ª ® ® ® ® ® ¬ Ł (3.4) Fromtheaboveequation,wecanseethatthespherecenterisafunctionofthreeparameters, f D 2 ŒE 2 Œ g or f G 2 ŒH 2 Œ g .Thus,wecandevelopanalgorithmtoestimatethesethreevalues,from whichweobtainthe3Dlocationofthespherecenter. 3.3ProjectedSphereCenterandAreaEstimation Weestimatetheprojectedsphereareaandcenterinthepixelcoordinatesystembyfully convolutionalnetworkswhichoutputarraysequalinsizetotheinputimage.Comparedtoanetwork thatdirectlypredictsourthreeparameters,andsohasonlythreelabelsperimage,fullyconvolutional networkshaveoneormorelabelsperpixelandaretranslationinvariant.Theadvantageisthatwe areabletorobustlytrainournetworksonarelativelysmalldataset.Theprojectedspherecenter isestimatedby Area-Net .Thespherecenterisestimatedusingtwomethods:(a)Centroidofthe Area-Net and(b) Center-Net . 3.3.1Area-Net Area-Net estimatesthepixelsonthesphereandsoobtainstheimage-area .Itpredictswhether thepixelsareonthesphereornot,whichcanbetreatedasabinaryclassi˝cationproblem.Thepixel labels @ 8 2f 0 Œ 1 g ,arebinaryvalueswitha1indicatingthepixelisonthesphere,seeFig.3.3(b)-(c). Thus,the equalsthesumofpixelsonthesphere. 11 (a)RGBimage (b)Binarylabel (c)Binaryclassi˝cation (d)Spherecenterlabel (e)Centerestimation Figure3.3:Anexampleofimage-basedspheredetectionwithCNNs.(a)TheRGBimage.(b)A binarylabelingofsphereandnon-spherepixels.(c)The Area-Net outputpixelclassi˝cationwith anaverageprecisionof0.998.Fromthisweestimatetheprojectedspherearea.(d)Thesphere centerlabelasanexponentialfunctionaroundtheprojectedcenter,seeEq.(3.5).(e)The Center-Net prediction,fromwhichweusethepeaklocationtodeterminethesphere-centerprojection. 12 3.3.2Center-Net Onestraightforwardwaytoestimatethecenter f G 2 ŒH 2 g istocalculatethecentroidofthe predictionimageofthe Area-Net .Thecenter f G 2 ŒH 2 g canalsobeestimatedbyusingthefully convolutionalneuralnetworks Center-Net .Herethelabelsarerealnumbersbetween0and1with apeakattheprojectedspherecenter ¹ G 2 ŒH 2 º ,whichisde˝nedwithascaledGaussianfunction illustratedinFig.3.3(d)-(e): @ 8 = exp ¹ G 2 G 8 º 2 ¸¹ H 2 H 8 º 2 2 f 2 ! Ł (3.5) Here G 8 ŒH 8 arethepixelcoordinatesand f governsthespreadwhichwechoosetobe8pixels. 3.3.3NetworkandTraining Ourgoalistoestimatetheprojectedspherecenteranditsareaaccurately.Theapproach toestimatingtheimage-areaof istoclassifywhichpixelisonthesphere,whichisabinary classi˝cationproblem.Theprojectedspherecentercanbederivedbythecentroidof Area-Net results.Thesecondmethodofestimatingtheprojectedspherecenterby Center-Net istopredict thepixel'svaluesandchoosethepositionoftheonethathasthelargestprobability. Usingtwoseparatenetworksenableseachtofocusononesimplertask.Bothnetworksuse similarfullyconvolutionalarchitecturebasedonthepopularU-Net[28]andillustratedinFig.3.4. Here,asequenceof 3 3 2Dconvolutionlayersisusedastheinformationwhich˛owsupthepyramid tothelowestresolution.Then,thesameconvolutionoperatesastheinformation˛owsdowneach layertothehighestresolution.Inaddition,thiscombineslearningbothalatentencodingwitha largereceptive˝eldandskipconnectionsforhighresolutionpixelclassi˝cation,whichenables featuresathighandlowresolutionstoin˛uenceeachother.A˝xednumberof8channelswas usedthroughoutthenetwork.ThenetworkisimplementedinPytorch[29],andduetotheGPU limitations,trainingisperformedonbatchesof 256 256 -pixelpatches. Theresultofthespheredetectionisadensityimagegivingameasureoftheprobabilityof on-sphere.Forthe Area-Net architecture,asthisdensityimageiscontinuous,wesetathreshold C 13 at0.5,whichconsidersthepixelswhoseprobabilityislargerorequalthan C ason-sphere.Onthe otherhand,forthe Center-Net architecture,thepixel'spositionthathasthelargestprobabilityof pixelvaluesistheprojectedspherecenterinthepixelcoordinatesystem. Figure3.4:Thenetworkarchitecture.Theyellowboxesdenotethe8-channeloutputsofthe 3 3 convolutional˝lters.Downsamplingbytwoisperformedbymax-poolingandupsamplingby nearest-neighbor.Skipconnectionsareelement-wisesummedwiththedecodingblock. 3.3.4LossFunction Thenetworksaretrainedbyoptimizingaweightedcrossentropylossfunction: ˘ˆ ¹ I 8 º = Õ 8 F 8 ¹ @ 8 I 8 ¸ log ¹ 1 ¸ exp ¹ I 8 ºº º Ł (3.6) Where @ 8 islabelsofpixels, F 8 istheweight,and I 8 thenetworkoutput.Thesenetworksdi˙erin theirpixellabelsandpixelweights. Thechoiceofpixelweights F 8 inEq.(3.6)isimportant,asthemajorityofpixelsineachimage areunrelatedtothespherepixelsorcenter,andcanbefairlyeasilyeliminated.Ontheotherhand, misclassi˝cationsonornearthespherehavethepotentialtoharmtheestimatedparameters.Thus, weallocatehalftheweighttothesmallnumberofhigh-importancepixelsandtheremainingweight 14 (a)Area-Net (b)Center-Net Figure3.5:IllustrationforCenter-NetandArea-Netweights.Thepixelswithintheregionofthe yellowellipseareconsideredashigh-importancepixels.(a)Area-Net.(b)Center-Net. tothelargenumberoflow-importancepixels.ThepixelweightsforCenter-NetandArea-Netare asfollows. 3.3.4.1Area-NetWeights High-importancepixelsareboththespherepixelsandsurroundingpixels.Includingthe surroundingpixelsisimportant,otherwisethenetworkhassmallpenalty"bleeding"positive classi˝cationsintothebackground.Tomakesurethepixelsontheedgeareclassi˝edcorrectly,We chooseadiskofradius1.2timesofaveragesphereellipseradiusforthehigh-importancepixels. Fig.3.5(a)showsasamplefortheArea-Netweight.Thepixelswithintheregionofyellowellipse areconsideredashigh-importance. 3.3.4.2Center-NetWeights Themostlikelyandimpactfulerrorsforthisnetworkarepixelsclosetobutnotatthecenterof thesphere.Hencewechooseadiskofpixelsofradius8pixelsaroundtheprojectedspherecenter asthehigh-importancepixels.Fig.3.5(b)showsasamplefortheCenter-Netweight.Thepixels withintheregionofyellowellipseareconsideredashigh-importance. 15 3.4CovarianceModelforCamera Section3.2introducesthatthe3Dspherecentercanbederivedfromtheprojectedspherecenter anditsarea. Area-Net and Center-Net areusedtopredicttheprojectedspherecenteranditsarea. Weshowtheapproachofestimatingthe3Dspherecenterrelativetothecamera. Theextrinsiccalibrationisanalignmentforthe3Dsphererelativetodi˙erentsensors.The resultswegetfromtheabovemethodsin6.1showthatsomeofthespherecenter3Dcoordinates areaccurate,whileothershavelargeuncertaintyanderror.Ideally,weshouldweightthealignments dependingonhowaccuratelyeachspherecenterpositionisknown.Thatis,weshoulddown-weight spherecenterswithlargeexpectedpositionerrormorethanotherspherecenters. Assumingasphereisobservedat # locationsbythecamera,thesphere-centerlocationrelative tothecameraatthe 8 'thmeasurementisindicatedby ( ¹ 2 º 8 .Thecovarianceofthesphere-center, asmeasuredbythecamera,isgivenbytheexpectation + ¹ 2 º 8 = ˆ f ( ¹ 2 º 8 ( ¹ 2 º 8 ) g andcanbe approximatedbysubstitutingthe˝rstorderTaylorexpansionofEq.(3.4). + ¹ 2 º 8 = / 2 28 2 6 6 6 6 6 6 6 6 4 f 2 D 00 0 f 2 E 0 000 3 7 7 7 7 7 7 7 7 5 ¸ / 2 28 f 2 4 2 8 ¸ D 2 28 f 2 D ¸ E 2 28 f 2 E 4 F 4 28 ! 2 6 6 6 6 6 6 6 6 4 D 2 28 D 28 E 28 D 28 D 28 E 28 E 2 28 E 28 D 28 E 28 1 3 7 7 7 7 7 7 7 7 5 Œ (3.7) forthe 8 'thsphere.Thiscovarianceinvolvesthevariancesofthespherecenterimagecoordinates, f 2 D , f 2 E andthevarianceofthespherearea, f 2 .Theseareassumedtobeindependentandobtained empiricallyfromourevaluationinTable6.1.ThedetailsofthederivationisprovidedinSectionA. 16 CHAPTER4 LIDAR-BASEDSPHEREDETECTIONAND3DESTIMATION Estimatingthe3Dspherecenterfromalidarscaninvolvestwosteps:determiningwhichlidar pixelsareonthesphereand˝ttingaspheremodeltothesepoints.Wedescribetheseasfollows: 4.1Lidar-BasedSphereDetection Todetectthesphereinalidarscan,weprocessthelidarpixelsasarangeimageinazimuth- elevationspace.Ourgoalistodeterminewhichpixelsbelongtothesphereinthisimage;whichis preciselythesamegoalasforthe Area-Net inSection3.3.Hence,weusetheexactsamenetwork architectureandlossfunctionas Area-Net ,exceptthattheinputimageisasingle-channelrange image.Toenablelabelaccuracy,welabeleachframeofpointcloudmanually.Asthespherealong withthewhiteboardisheldinfrontofaperson,theclosestpointtothelidarwithintheregion aroundthewhiteboardisalwaysonthespheresurface.Weselectthepointcloudwhichiswithin 1 Ł 5 A distancetotheclosestpointasthepointsonthespheresurface.Theoutputofthisnetwork isabinaryclassi˝cationoflidarpixelsasthosebelongingtothesphere,illustratedinFig.4.1. (a)Rangeimage/m (b)Label (c)Prediction Figure4.1:Lidarspheredetection:(a)Lidarrangeimage.(b)Groundtruthlabelimage.(c)Lidar pixelclassi˝cationby Area-Net trainedonlidarrangeimages. 17 4.2SphereCenterEstimationfromPointCloud Oncelidarpixelsonthespherehavebeenclassi˝ed,thesphere's3Dlocationisestimatedby ˝ttingasphereofknownradiustothepixels'3Dpositions.Forthisweusethemethodin[25,26] which˝tsaspherebyminimizingthedirectionalerror,aswellasaccountingforraysthatgrazethe sphere.Forcompleteness,werewritetheirequationsinournotationasfollows.Let % 9 bethe3D locationofpixel 9 relativetotheorigin $ ,atthelidar,andlet ^ % 9 betheintersectionofthepixel's ray, $% 9 ,withthesphere.Whentheraydoesnotintersectwiththesphere, ^ % 9 isthepointonthe sphereclosesttotheray.This˝ttingisillustratedinFig.4.2.The˝ttingerroristhemagnitudeof thedistancebetween % 9 and ^ % 9 ,andfor " pixelsonaspherethetotalcostis: ˆ ¹ ( ¹ ; º 8 º = " Õ 9 8 > > >< > > > : j ) 9 jj % 9 j q A 2 @ 2 9 2 if @ 9 A ¹j ) 9 jj % 9 jº 2 ¸¹ @ 9 A º 2 if @ 9 ¡A (4.1) Here ! 2 normsareused, A isthesphereradius, ) 9 isthepointonray $% 9 closesttothesphere center,and @ 9 thedistancebetween ) 9 andthespherecenter.TheseareillustratedinFig.4.2,and givenby: ) 9 = ( ¹ ; º 8 % 9 % 9 % 9 % 9 Œ@ 9 = j ( ¹ ; º 8 ) 9 j Ł (4.2) Thespherecenterisobtainedby˝nding ( ¹ ; º 8 thatminimizesEq.(4.1). Figure4.2:IllustrationofthecostEq.(4.1),usedin˝ttingasphereofknownradiustolidarpixels detectedinFig.4.1. 18 4.3CovarianceModelforLidar Thecovarianceofthespherecenter,asmeasuredbythelidarraydistance,canbederivedfrom Eq.(4.1).Wemodeleachlidarraywiththerange < 9 = j % 9 j andradialerrorswithstandarddeviation f < .A˝rstorderTaylorexpansionofthisleadstothefollowingequationforthecovariance: + ¹ ; º 8 = f 2 < © « " Õ 9 r 2 ( ˆ 89 ª ® ¬ 1 © « " Õ 9 r (< 9 ˆ 89 r ) (< 9 ˆ 89 ª ® ¬ © « " Õ 9 r 2 ( ˆ 89 ª ® ¬ ) (4.3) Empirically,weestimate f < = 2 2< .ThedetailsofthederivationisprovidedintheSectionB. AsEq.(4.3)leadstoacomplexexpression,wealsoinsteadmakeasimplifyingapproximation thatthespherecentercovarianceisthecovarianceofthemeanofthelidarhitsonthesphere.If eachlidarrayhasacovarianceof f 2 < ˚ ,where ˚ isthe 3 3 identitymatrix,thenthisleadstothe followingexpressionforthecovarianceofspherecenter 8 : + ¹ ; º 8 = f 2 < " ˚Ł (4.4) Atlongerrangewithfewerhits, " ,thespherecovarianceincreases. 19 CHAPTER5 SOLVEFOREXTRINSICPARAMETERS Asphereistranslatedinthejoint˝eldofviewofthelidarandcameraandobservedat # locationsbybothsensors.Thesphere-centerlocationrelativetothelidaratthe 8 'thmeasurement indicatedby ( ¹ ; º 8 andrelativetothecameraby ( ¹ 2 º 8 .Weseekarotation ' andtranslation C that transformsthelidarcentertocameracoordinates: ( ¹ 2 º 8 = '( ¹ ; º 8 ¸ CŁ (5.1) Astraightforwardideato˝ndtheextrinsicparametersistominimizetheEuclideanpoint-to-point distance.Anotherapproachtoestimatetheextrinsicparametersistominimizethepoint-to-ray distancewhichisusedin[16,17].Weproposetoderivetheextrinsicparametersbyminimizing ourproposedpoint-to-pointdistancewithcovariancemodels.Thesethreeapproacheswillbe introducedinthefollowingsections. 5.1Point-to-PointDistance ThedistancebetweenallmeasurementsfromsensorsistheEuclideanpoint-to-pointdistance, whichisde˝nedasthefollowingEq.5.2. ˆ ¹ 'ŒC º = 1 # # Õ 8 ¹ '( ¹ ; º 8 ¸ C ( ¹ 2 º 8 º ) ¹ '( ¹ ; º 8 ¸ C ( ¹ 2 º 8 º Ł (5.2) ThedistancemeasureinEq.5.2dependsonthetypesofobservationswithin ( ¹ 2 º 8 and ( ¹ ; º 8 .Here, werepresenttheobservationsfromthecameraandlidarasa3Dpoint. Thiscostfunctionhasaclosed-formlinearsolutionbasedonSVD.Tocomputethecentroids ofbothpointsets: ( ¹ ; º = 1 # # Õ 8 ( ¹ ; º 8 Œ ( ¹ 2 º = 1 # # Õ 8 ( ¹ 2 º 8 Ł (5.3) To˝ndtheoptimalrotation,we˝rstre-centerbothdatasetssothatbothcentroidsareatthe origin.Thisremovesthetranslationcomponent,leavingtherotationtoconsider.Thenextstep 20 involvesaccumulatingamatrix,called ˛ ,and˝ndstherotation ' andtranslation C bysingular valuedecomposition(SVD)asfollows: ˛ = ¹ ( ¹ ; º ( ¹ ; º º¹ ( ¹ 2 º 8 ( ¹ 2 º º ) Œ » *Œ˝Œ+ ¼ = (+ˇ ¹ ˛ º Ł (5.4) Wehavesolvedfortherotation ' ;nowwecansolvefortranslationbypluggingthecentroidsinto theEq.5.1 ' = +* ) ŒC = ( ¹ 2 º ' ( ¹ ; º Ł (5.5) 5.2Point-to-RayDistance Thesecondapproachtosolvingfortherotationandtranslationisminimizingthepoint-to-ray distance[16,17].Thepoint-to-pointdistancedoesnotmodelthevariationindepthuncertainty. Soonewaytoavoidthisproblemistominimizepoint-to-raydistance.Acameraobservationcan beinterpretedasaraywhenonlyusingthe2Dprojectionofthesphere'scenter.Arayisde˝ned bythespherecenter ( ¹ 2 º 8 relativetothecameraanditsunitvector E .Theclosestpointontheray tothespherecenter ( ; 8 relativetolidaris - 8 .Thusthepoint-to-raydistanceisthedistanceofthe point ( ¹ @ º 8 totheray - 8 ,where ( ¹ @ º 8 = '( ¹ ; º 8 ¸ C . Theequationforthepoint-to-raylossisasfollows: ˆ ¹ 'ŒC º = 1 # # Õ 8 j ( ¹ @ º 8 - 8 j 2 (5.6) Therotationandtranslationissolvedbyusingthelibraryof scipy.minimize [30],giventhe initialvaluesestimatedin5.1. 5.3Point-to-PointDistancewithCovarianceModels TheproblemwithEq.(5.2)isthatsomeofthespherecenter3Dcoordinatesareaccurate, whileothershavelargeuncertaintyanderror.Ideally,the˝ttederrorforspherecentershouldbe weighedwithitsinversecovariance.Let + ¹ 2 º 8 and + ¹ ; º 8 indicatethecovarianceofthe 8 'thsphere centerrelativetothecameraandlidar,respectively.Then,theoptimal˝tof3Dpoints ^ ( 8 inlidar 21 coordinatesandextrinsicparameters ' and C willminimizethefollowingloss: ˆ < ¹ 'ŒCŒ ^ ( 1 ŒŁŁŁ ^ ( # º = 1 # # Õ 8 ¹ ^ ( 8 ( ¹ ; º 8 º ) + ¹ ; º 8 1 ¹ ^ ( 8 ( ¹ ; º 8 º ¸ 1 # # Õ 8 ¹ ' ^ ( 8 ¸ C ( ¹ 2 º 8 º ) + ¹ 2 º 8 1 ¹ ' ^ ( 8 ¸ C ( ¹ 2 º 8 º Ł (5.7) Here,the˝tted3Dpoints ^ ( 8 arenuisanceparametersanddiscarded,leavingtheestimatedpose ' and C .Therotationandtranslationarealsosolvedbyusingthelibraryof scipy.minimize [30],given theinitialvaluesestimatedin5.1. 22 CHAPTER6 EXPERIMENTSANDRESULTS 6.1Image-BasedSphereDetectionand3DEstimation Tovalidatetheproposedspheredetectionandestimationperformance,wecollectaimage datasetcontaining548labeledsphereimagesat 800 600 resolutioninvariouslightingconditions. Labelingisdonebymarkingseveralpointsonthesphereboundaryand˝ttinganellipse.Sample labelsareshowninFig.3.3(b)and(d).Thedatasetisdividedintothetraining,validationand testingsetswith367,103and78images,respectively. Asampleofthebinaryclassi˝cation(areaestimation),centerestimation,andcomparisonswith groundtruthsinthetestsetareshowninFig.6.3.ThequantitativeCNNperformancemeasureson 103imagesatvariousdepthvaluesaresummarizedinTable6.1.TheCentroidof Area-Net achieves betterresultsthanthe Center-Net method,thusthefollowingresultsoftheprojectedspherecenter arebasedonthemethodofthecentroidof Area-Net . Table6.1:Estimationaccuracyforspherearea(A)andcenter ¹ G 2 ŒH 2 º measuredinpixelsat di˙eringdepthsmeasuredonmeanabsoluteerror(MAE)andstandarddeviation( f ). Depth/m A/pixel Center-Net Centroidof Area-Net MAE f G 2 /pixel H 2 /pixel G 2 /pixel H 2 /pixel MAE f MAE f MAE f MAE f [2.0,3.5) 59.0 71.7 0.7 0.7 0.8 0.7 0.5 0.6 0.4 0.5 [3.5,4.5) 64.2 76.5 0.8 0.8 0.7 0.6 0.5 0.6 0.4 0.5 [4.5,5.5) 35.3 46.5 0.6 0.6 0.6 0.8 0.4 0.5 0.4 0.5 [5.5,6.5) 30.0 35.5 0.7 0.8 0.5 0.7 0.4 0.5 0.3 0.4 [6.5,7.5) 33.4 42.0 0.8 0.8 0.7 0.7 0.3 0.4 0.3 0.5 23 6.2Lidar-BasedSphereDetectionand3DEstimation Wecreatealabeleddatasetof470scansat 1024 64 resolutionusinganOusterLidar.They aredividedintotraining,validationandtestsetscontaining289,103and78imagesrespectively.A croppedregionaroundasphereisshowninFig.4.1,alongwithlabelandestimatedspherepixels. OurCNNtodeterminedon-spherelidarpixelsevaluatedon103rangeimageswithanaverage precisionof0.988.Asampleofthebinaryclassi˝cation,andcomparisonswithgroundtruthsin thetestsetforlidarareshowninFig.6.4. 6.3CalibrationResults Extrinsiccalibrationparameters, 'ŒC ,areobtainedbyminimizingEq.(5.7)andusingthe cameracovariancemodelinEq.(3.7)andlidarcovariancemodelinEq.(4.4)with140pairsof lidarandcameraframes.Fig.1.3showstheposeestimationalongwithasubsetofthespheresused inthealignment. 6.4MethodValidation Ifwecouldgetthegroundtruthfortherelativepositionsofthesensors,itwouldallowthe comparisonoftheestimatedtransformationwiththeexacttransformationamongsensors.However, itisverydi˚culttoguaranteewithprecisioniftheobtainedcalibrationiscorrect,giventhedi˚culty tomeasureexactlythecorrectposebetweenpairsofsensors.Thisproblemisparticularlyobvious fortherotations,sincesmallerrorinrotationmayresultinlargeerrorinthe˝nalregisterederror. Werecordimageandlidardataofasphereatasetof26knownpositionsat0.25mintervals usingtworailsmountedonawall,asillustratedinFig.6.1.Wedeterminethelidarlocationrelative tothisgridby˝ttingitsestimated3Dspherecenterswiththeknown3Dcenterlocations.Then usingourestimatedextrinsicparameterswecanprojectthegridcentersintothecameraimage planeand˝ndtheerrorbetweenthesepointsandthehand-labeledspherecenters.Fig.6.2illustrate thevalidationmethod.Thiserrorwillbeanupperboundontheindirectlyextrinsiccalibration error,asitincludessomeerrorin˝ttingthe3Dgrid. 24 Table6.2:ThemeanprojectederrorinpixelsforagridpatterninFig6.1usedtocomparealignment methods.Extrinsiccalibrationerrorsresultinprojectedgridalignmenterrors.Thespherecenter estimationisbasedoncostEq.(4.1).Cov1isbasedonthelidarcovariancemodelinEq.(4.3)and Cov2isbasedonEq.(4.4).P2R isthepoint-to-raydistanceusedin[16,17]. Depth/m SVD P2R Ours Cov1 Cov2 [4.00,4.15) 1.99 1.81 1.48 1.46 [5.15,5.30) 2.36 1.53 1.29 1.24 [6.15,6.30) 2.68 1.85 1.46 1.45 Figure6.1:Extrinsiccalibrationaccuracyisvalidatedusingaspheremovedthroughasetofknown locationsintheworld. UsingthisprocedurewecomparethreealignmentmethodsinTable6.2.The˝rstcolumnis Euclideanpoint-to-pointalignmentthroughminimizingEq.(5.2).Thesecondminimizesthe3D lidarpoint-to-raydistance,asusedin[16,17].ThirdisourmethodthatminimizestheMahalanobis distance(3.7)andobtainsthebestresult.Thesimpli˝edlidarcovariancemodelofthemeanofthe lidarhitsonthesphereinEq.(4.4)achievesbetterresultsthanthecovarianceofthespherecenter inEq.(4.3). 25 Figure6.2:Methodvalidationillustration. (a) (b) (c) (d) (e) Figure6.3:Asampleofthebinaryclassi˝cation(areaestimation),centerestimation,andcompar- isonswithgroundtruthsforthecamera.(a)RGBimage.(b)Binarylabel.(c)Binaryclassi˝cation. (d)Centerlabel.(e)CenterEstimation 26 (a) (b) (c) Figure6.4:Asampleofthebinaryclassi˝cation,andcomparisonswithgroundtruthsforthelidar. (a)Rangeimage/m.(b)Binarylabel.(c)Binaryclassi˝cation 27 CHAPTER7 CONCLUSIONANDFUTUREWORK Weproposeasimple-to-perform,fast,robustandaccurateextrinsiccalibrationmethodthat estimatesrelativeposebetweenacameraandlidar.Theapparatusisportableandinexpensive:a hand-held,paintedacrylicsphere.Nomanualannotationisrequiredduringcalibrationasweuse pre-trainedconvolutionalneuralnetworkstoautomaticallydetectthesphereinboththecamera imagesandlidarscans.Ourmethodestimatesthe3Dpositionsofthesphererelativetoboththe cameraandlidaraswellasthepositioncovarianceincameraandlidarcovariances.Byoptimizing acovariance-weightedcost,wedeterminetherelativeposeofthesensors.Webelievethatthe combinationoflow-cost,easeofdatacollection,automatedprocessingandaccurateresultswill makethisacompellingtoolforautomotiveandotherroboticapplicationsthatcombine3Dand2D sensors. 28 APPENDICES 29 APPENDIXA THECAMERACOVARIANCEMODEL Togetthecovariancemodel,weneedtousetheperturbations.Assuming - isanunbiased variable,weneedtoestimate - = - ¸ - . - isasmallerrorwhichistheperturbation. - isthe truevalue.Thentheexpectationof - is: ˆ f - g = ˆ f - ¸ - g = - ¸ ˆ f - g = - (A.1) Thevarianceof - is: +0A ¹ - º = ˆ f - - ) g Œ (A.2) OurCNNestimatestheprojectedspherecenter f G 2 ŒH 2 g andarea ,inanimage,andfrom thesewecalculatethe3Dspherecenterrelativetothecamera.Assumingthe3Dspherecenter relativetocamerais ¹ - 2 Œ. 2 Œ/ 2 º .Thecovariancematrixmodelisasfollowings: + ¹ 2 º = 2 6 6 6 6 6 6 6 6 4 f 2 - 2 f 2 - 2 . 2 f 2 - 2 / 2 f 2 . 2 - 2 f 2 . 2 f 2 . 2 / 2 f 2 / 2 - 2 f 2 . 2 / 2 f 2 / 2 3 7 7 7 7 7 7 7 7 5 (A.3) RecallEq.(3.2),weknowunit-focalplanecoordinatesoftheprojectedspherecenter ¹ D 2 ŒE 2 º where D 2 = ¹ G 2 G 0 ºš 5 G and E 2 = ¹ H 2 H 0 ºš 5 H .Forsimplicity,thefollowingderivationisbased on ¹ D 2 ŒE 2 º .We˝rstlycalculatethevarianceforarea andcenter ¹ D 2 ŒE 2 º . = ¸ ,where isthetrueareaand istheperturbation.Itisthesamefor D 2 = D 2 ¸ D 2 and E = E 2 ¸ E 2 .The varianceof is: f 2 = ˆ f ) g (A.4) Thevarianceof D 2 and E 2 is: f 2 D 2 = ˆ f D 2 D 2 g Œf 2 E 2 = ˆ f E 2 E 2 g Œ (A.5) SmallperturbationcanbeapproximatedusingtheTaylorexpansion.RecallEq.3.3,wehave / 2 = : q F 2 ,where F 2 = q D 2 2 ¸ E 2 2 ¸ 1 and : = A p c5 G 5 H . 30 DoTaylerexpansion: / 2 = / 2 ¸ / 2 = : r F 2 0 0 ¸ 3 ¹ : q F 2 º 3 ¸ 3 ¹ : q F 2 º 3D 2 D 2 ¸ 3 ¹ : q F 2 º 3E 2 E 2 ¸ ŁŁŁ = : r F 2 0 0 ¸ : p F 2 2 3 š 2 ¸ :D 2 2 p F 3 š 2 2 D 2 ¸ :E 2 2 p F 3 š 2 2 E 2 ¸ ŁŁŁ (A.6) So,wehave / 2 : / 2 = : p F 2 2 3 š 2 ¸ :D 2 2 p F 3 š 2 2 D 2 ¸ :E 2 2 p F 3 š 2 2 E 2 (A.7) As D 2 , E 2 and areuncorrelated,thevarianceof / is: f 2 / 2 = ˆ f / / ) g = : 2 4 3 ˆ f ) g¸ : 2 D 2 2 4 F 3 2 ˆ f D 2 D 2 ) g¸ : 2 E 2 2 4 F 3 2 ˆ f E 2 E 2 ) g = : 2 4 3 f 2 ¸ : 2 D 2 2 4 F 3 2 f 2 D 2 ¸ : 2 E 2 2 4 F 3 2 f 2 E 2 = / 2 2 f 2 4 2 ¸ / 2 2 D 2 2 f 2 D 2 4 F 4 2 ¸ / 2 2 E 2 2 f 2 E 2 4 F 4 2 (A.8) Recallthat - B = D 2 / 2 fromEq.3.4.DoTylerexpansion: - 2 = - 2 ¸ - 2 = D 2 / 2 ¸ D 2 / 2 ¸ D 2 / 2 (A.9) Thevarianceof - 2 is: f 2 - 2 = D 2 2 f 2 / ¸ / 2 f 2 D 2 (A.10) Similarity,thevarianceof . 2 is: f 2 . 2 = E 2 2 f 2 / ¸ / 2 f 2 E 2 (A.11) As D 2 and E 2 areuncorrelated, f 2 D 2 E 2 =0. f 2 - 2 . 2 = f 2 . 2 - 2 = ˆ f - . g = D 2 E 2 f 2 / 2 ¸ / 2 2 f 2 D 2 E 2 = D 2 E 2 f 2 / 2 (A.12) 31 Similarity,wehave: f 2 - 2 / 2 = f 2 / 2 - 2 = E 2 f 2 / 2 Œf 2 . 2 / 2 = f 2 / 2 . 2 = D 2 f 2 / 2 (A.13) 32 APPENDIXB THELIDARCOVARIANCEMODEL Thecovarianceofthe3Dspherecenter ( ,asmeasuredbythelidarhits.Wemodellidarrays withranges < = ¹ < 1 ŒŁŁŁŒ< " º havingradialerrorseachwithstandarddeviation f < .Assuming the3Dspherecenterat 8 C positionis ( whichisderivedbyminimizingacostfunction ˆ ¹ <Œ( º . Let r ( ˆ ¹ <Œ( º bethe ( gradientofthecost.Atthisminimumwecanwrite: r ( ˆ ¹ <Œ( º = 0 (B.1) DotheTaylorexpansionofEq.(B.1)andkeepthe˝rstorderterms: r 2 ( ˆ ¹ <Œ( º ( ¸r (< ˆ ¹ <Œ( º < = 0 (B.2) Sowehave: ( = ¹r 2 ( ˆ ¹ <Œ( ºº 1 r (< ˆ ¹ <Œ( º < (B.3) Thecovarianceofthespherecenter + ¹ ; º ismeasuredas: + ¹ ; º = ˆ f ( ( ) g = ¹r 2 ( ˆ ¹ <Œ( ºº 1 r (< ˆ ¹ <Œ( º ˆ f < < ) gr 2 (< ˆ ¹ <Œ( º¹r 2 ( ˆ ¹ <Œ( ºº ) = f 2 < ¹r 2 ( ˆ ¹ <Œ( ºº 1 r (< ˆ ¹ <Œ( ºr ) (< ˆ ¹ <Œ( º¹r 2 ( ˆ ¹ <Œ( ºº ) = f 2 < © « " Õ 9 r 2 ( ˆ 9 ª ® ¬ 1 © « " Õ 9 r (< 9 ˆ 9 r ) (< 9 ˆ 9 ª ® ¬ © « " Õ 9 r 2 ( ˆ 9 ª ® ¬ ) (B.4) Thisequationof + ¹ ; º isappliedtoEq.(4.3)andEq.(4.4). 33 BIBLIOGRAPHY 34 BIBLIOGRAPHY [1] KoyelBanerjeeetal.cameralidarfusionandobjectdetectiononhybriddatafor autonomousdrIn: 2018IEEEIntelligentVehiclesSymposium(IV) .IEEE.2018, pp. [2] FeihuZhang,DanielClarke,andAloisKnoll.ehicledetectionbasedonlidarandcam- erafusionIn: 17thInternationalIEEEConferenceonIntelligentTransportationSystems (ITSC) .IEEE.2014,pp. [3] VarunaDeSilva,JamieRoche,andAhmetKondoz.usionofLiDARandcamerasensor dataforenvironmentsensingindriverlessvIn:(2017). [4] GangqiangZhaoetal.usionof3D-LIDARandcameradataforsceneIn: Journal ofVisualCommunicationandImageRepresentation 25.1(2014),pp. [5] KihoKwaketal.insiccalibrationofasinglelinescanninglidarandaIn: 2011IEEE/RSJInternationalConferenceonIntelligentRobotsandSystems .IEEE.2011, pp. [6] JesseLevinsonandSebastianThrun.utomaticOnlineCalibrationofCamerasand In: Robotics:ScienceandSystems .Vol.2.2013,p.7. [7] GauravPandeyetal.utomatictargetlessextrinsiccalibrationofa3dlidarandcameraby maximizingmutualinforIn: AAAI .Citeseer.2012. [8] ZacharyTaylorandJuanNieto.Automaticcalibrationoflidarandcameraimagesusingnor- malizedmutualinformationIn: RoboticsandAutomation(ICRA),2013IEEEInternational Conferenceon .2013. [9] QilongZhangandRobertPless.insiccalibrationofacameraandlaserrange˝nder (improvescameraIn: 2004IEEE/RSJInternationalConferenceonIntelligent RobotsandSystems(IROS)(IEEECat.No.04CH37566) .Vol.3.IEEE.2004,pp. [10] AbdallahKassirandThierryPeynot.eliableautomaticcamera-laserIn: Proceedingsofthe2010AustralasianConferenceonRobotics&Automation .ARAA.2010. [11] AndreasGeigeretal.utomaticcameraandrangesensorcalibrationusingasingle In: 2012IEEEInternationalConferenceonRoboticsandAutomation .IEEE.2012,pp. 3943. [12] YoonsuParketal.betweencolorcameraand3DLIDARinstrumentswitha polygonalplanarIn: Sensors 14.3(2014),pp. [13] Z.Chai,Y.Sun,andZ.Xiong.NovelMethodforLiDARCameraCalibrationbyPlaneFit- In: 2018IEEE/ASMEInternationalConferenceonAdvancedIntelligentMechatronics (AIM) .2018,pp. 35 [14] VincentFremont,PhilippeBonnifait,etal.insiccalibrationbetweenamulti-layer lidarandaIn: 2008IEEEInternationalConferenceonMultisensorFusionand IntegrationforIntelligentSystems .IEEE.2008,pp. [15] MartinVel'asetal.ofrgbcamerawithvelodyneIn:(2014). [16] MarceloPereiraetal.calibrationofmultipleLIDARsandcamerasonautonomous vIn: RoboticsandAutonomousSystems 83(2016),pp. [17] JuliusKümmerle,TilmanKühner,andMartinLauer.Automaticcalibrationofmultiplecam- erasanddepthsensorswithasphericaltargetIn: 2018IEEE/RSJInternationalConference onIntelligentRobotsandSystems(IROS) .IEEE.2018,pp. [18] JuanCastorena,UlugbekSKamilov,andPetrosTBoufounos.utocalibrationoflidarand opticalcamerasviaedgeIn: 2016IEEEInternationalConferenceonAcoustics, SpeechandSignalProcessing(ICASSP) .IEEE.2016,pp. [19] MichaelA.Penna.calibration:Aquickandeasywaytodeterminethescalef In: IEEETransactionsonPatternAnalysis&MachineIntelligence 12(1991),pp. [20] HirohisaTeramotoandGangXu.calibrationbyasingleimageofballs:From conicstotheabsoluteIn: Proc.of5thAsianConferenceonComputerVision .2002, pp. [21] XianghuaYingandHongbinZha.icinterpretationsoftherelationbetweenthe imageoftheabsoluteconicandsphereimagIn: IEEETransactionsonPatternAnalysis andMachineIntelligence 28.12(2006),pp. [22] MotilalAgrawalandLarrySDavis.calibrationusingspheres:Asemi-de˝nite programmingapproacIn: null .IEEE.2003,p.782. [23] HuiZhang,KWongKwan-yee,andGuoqiangZhang.calibrationfromimagesof In: IEEETransactionsonPatternAnalysisandMachineIntelligence 29.3(2007), pp. [24] JunzhiGuanetal.insiccalibrationofcameranetworksusingaIn: Sensors 15.8(2015),pp. [25] ChristophWitzgall,GeraldineSCheok,andAnthonyJKearsley.ecoveringspheresfrom 3dpointIn: 35thIEEEAppliedImageryandPatternRecognitionWorkshop(AIPR'06) . IEEE.2006,pp. [26] MarekFranaszeketal.ittingspherestorangedatafrom3-DimagingsysIn: IEEE TransactionsonInstrumentationandMeasurement 58.10(2009),pp. [27] MinghaoRuanandDanielHuber.of3DsensorsusingasphericaltargIn: 20142ndInternationalConferenceon3DVision .Vol.1.IEEE.2014,pp. 36 [28] OlafRonneberger,PhilippFischer,andThomasBrox.Convolutionalnetworksfor biomedicalimageIn: InternationalConferenceonMedicalimagecomputing andcomputer-assistedintervention .Springer.2015,pp. [29] AdamPaszkeetal.orch:Animperativestyle,high-performancedeeplearninglibrar In: AdvancesinNeuralInformationProcessingSystems .2019,pp. [30] EJones,TOliphant,PPeterson,etal..minimize:Opensourcescienti˝ctoolsfor In:(2001). 37