Accurate 3D tracking using visual and depth data

Gedik, Osman Serdar
3D tracking of objects is essential in many applications, such as robotics and aug- mented reality (AR), and availability of accurate pose estimates increases reliability in robotic applications whereas decreases jitter in AR scenarios. As a result of the recent advances in the sensor technology, it is possible to capture synchronous high frame rate RGB and depth data. With this motivation, an automated and high accurate 3D tracking algorithm based on simultaneous utilization of visual and depth sensors is presented. The depth sensor data is utilized both in raw format and in the form of Shape Index Map (SIM), after the observation that the latter transformation empha- sizes structural details and provides a proper basis to jointly utilize both sensor data. As the object model, the initial object colored point cloud is utilized, which elimi- nates dependency on any offline generated Computer Aided Design (CAD) models that might limit the application areas. A typical 3D tracking algorithm composes of the following stages: Feature selection, feature association between consecutive frames and 3D pose estimation by the feature correspondences. Since the main aim is to perform highly accurate 3D tracking of any user selected object, data from both sensors is exploited in every stage of the process for improving accuracy, as well as robustness. First of all, a novel feature selection method, which localizes features with high textural and spatial cornerness properties, is proposed. In this method, in order to increase the spatial spread of features around the object, the region of interest is divided into regular grids. Within each grid a single feature with maximum cornerness measure in terms of both intensity and SIM data is selected. Imposing spatial-textural constraints jointly selects more discrimina- tive features, whereas a regular grid-based approach decreases bias on pose estimates. Then, the selected features are associated between consecutive frames by a new fea- ture tracking approach, which tracks each feature independently and simultaneously on intensity and SIM data for improving 3D tracking performance. The method de- cides on the final feature association based on the reliabilities of individual trackers estimated online. Such a parallel approach is observed to increase robustness against sensor noise and individual tracker failures. Finally, RGB and depth measurements of localized features are fused in a well-known Extended Kalman Filter (EKF) frame- work. In this framework, we propose a novel measurement weighting scheme, based on the manipulation of Kalman gain term, which favors high quality features and pro- vides robustness against measurement errors. This scheme, establishing a connection between computer vision and Bayes filtering disciplines, eliminates sole dependency on predefined sensor noise parameters and identical measurement noise assumption. The increase in 3D tracing accuracy due to each proposed sub-system is shown via experimental results. Furthermore, the accuracy of the proposed 3D tracking method is tested against a number of well-known techniques from the literature and supe- rior performance is observed against such approaches. Finally, the resulting pose estimates of the proposed algorithm is utilized to obtain 3D maps after combining colored point clouds at consecutive time instants. We observe that, although loop clo- sure or post-processing algorithms are not exploited, significant number of 3D point clouds are combined with a quite high accuracy.


Fusing 2D and 3D Clues for 3D Tracking Using Visual and Range Data
Gedik, O. Serdar; Alatan, Abdullah Aydın (2013-07-12)
3D tracking of rigid objects is required in many applications, such as robotics or augmented reality (AR). The availability of accurate pose estimates increases reliability in robotic applications and decreases jitter in AR scenarios. Pure vision-based 3D trackers require either manual initializations or offline training stages, whereas trackers relying on pure depth sensors are not suitable for AR applications. In this paper, an automated 3D tracking algorithm, which is based on fusion of vision and depth ...
AKSOY, Yagiz; Alatan, Abdullah Aydın (2014-10-30)
Most of the mobile applications require efficient and precise computation of the device pose, and almost every mobile device has inertial sensors already equipped together with a camera. This fact makes sensor fusion quite attractive for increasing efficiency during pose tracking. However, the state-of-the-art fusion algorithms have a major shortcoming: lack of well-defined uncertainty introduced to the system during the prediction stage of the fusion filters. Such a drawback results in determining covarian...
Case studies on the use of neural networks in eutrophication modeling
Karul, C; Soyupak, S; Cilesiz, AF; Akbay, N; Germen, E (2000-10-30)
Artificial neural networks are becoming more and more common to be used in development of prediction models for complex systems as the theory behind them develops and the processing power of computers increase. A three layer Levenberg-Marquardt feedforward learning algorithm was used to model the eutrophication process in three water bodies of Turkey (Keban Dam Reservoir, Mogan and Eymir Lakes). Despite the very complex and peculiar nature of Keban Dam, a relatively good correlation (correlation coefficient...
Design and implementation of a head tracking controlled pan and tilt vision system
Ölmez, Hasan; Konukseven, Erhan İlhan; Koku, Ahmet Buğra; Department of Mechanical Engineering (2013)
Head tracking applications are getting widely used in robotics implementations such as in military industry for surveillance applications of remote controlled unmanned vehicles, in medical applications for heart and vessel surgeries, in entertainment and game applications for virtual reality. It can be used for helping the operator of teleoperated unmanned vehicle with improving his performance by decreasing workload and managing stress. Head tracking is a challenging application due to real time tracking a...
3-D Rigid Body Tracking Using Vision and Depth Sensors
Gedik, O. Serdar; Alatan, Abdullah Aydın (Institute of Electrical and Electronics Engineers (IEEE), 2013-10-01)
In robotics and augmented reality applications, model-based 3-D tracking of rigid objects is generally required. With the help of accurate pose estimates, it is required to increase reliability and decrease jitter in total. Among many solutions of pose estimation in the literature, pure vision-based 3-D trackers require either manual initializations or offline training stages. On the other hand, trackers relying on pure depth sensors are not suitable for AR applications. An automated 3-D tracking algorithm,...
Citation Formats
O. S. Gedik, “Accurate 3D tracking using visual and depth data,” Ph.D. - Doctoral Program, Middle East Technical University, 2014.