Autonomous driving systems require accurate 3D object detection and tracking to achieve reliable path planning and navigation. For object detection, there have been significant advances in neural networks for single-modality approaches. However, it has been surprisingly difficult to train networks to use multiple modalities in a way that demonstrates gain over single-modality networks. In this dissertation, we first propose three networks for Camera-LiDAR and Camera-Radar fusion. For Camera-LiDAR fusion, CLOCs (Camera-LiDAR Object Candidates fusion) and Fast-CLOCs are presented. CLOCs fusion provides a multi-modal fusion framework that significantly improves the performance of single-modality detectors. CLOCs operates on the combined output candidates before Non-Maximum Suppression (NMS) of any 2D and any 3D detector, and is trained to leverage their geometric and semantic consistencies to produce more accurate 3D detection results. Fast-CLOCs can run in near real-time with less computational requirements compared to CLOCs. Fast-CLOCs eliminates the separate heavy 2D detector, and instead uses a 3D detector-cued 2D image detector (3D-Q-2D) to reduce memory and computation. For Camera-Radar fusion, we propose TransCAR, a Transformer-based Camera-And-Radar fusion solution for 3D object detection. The cross-attention layer within the transformer decoder can adaptively learn the soft-association between the radar features and vision queries instead of hard-association based on sensor calibration only. Then, we propose to solve the 3D multiple object tracking (MOT) problem for autonomous driving applications using a random finite set-based (RFS) Multiple Measurement Models filter (RFS-M3). In particular, we propose multiple measurement models for a Poisson multi-Bernoulli mixture (PMBM) filter in support of different application scenarios. Our RFS-M3 filter can naturally model these uncertainties accurately and elegantly. We combine learning-based detections with our RFS-M3 tracker by incorporating the detection confidence score into the PMBM prediction and update step. We have evaluated our CLOCs, Fast-CLOCs and TransCAR fusion-based 3D detector and RFS-M3 3D tracker using challenging datasets including KITTI, nuScenes, Argoverse and Waymo that are released by academia and industry leaders. Superior experimental results demonstrated the effectiveness of the proposed approaches.
Read
