Multimodal Deep Learning for Heterogeneous GNSS-R Data Fusion and Ocean Wind Speed Retrieval

• Published in: IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing
• Main Authors: Chu, Xiaohan, He, Jie, Song, Hongqing, Qi, Yue, Sun, Yueqiang, Bai, Weihua, Li, Wei, Wu, Qiwu
• Format: Report
• Language: English
• Published: 2020
• Subjects: Feature extraction; Wind speed; Data models; Sea surface; Sea measurements; Machine learning; Delay-Doppler map (DDM); GNSS-reflectometry (GNSS-R); multimodal deep learning; ocean surface wind retrieval; SEA-ICE; SCATTERING; Sea ice
• Online Access: http://ir.nssc.ac.cn/handle/122/7580; https://doi.org/10.1109/JSTARS.2020.3010879

The comprehensiveness of the raw input data and the effectiveness of feature engineering are two key factors affecting the performance of machine learning. To improve the data comprehensiveness for Global Navigation Satellite System Reflectometry (GNSS-R) ocean wind speed retrieval, this article introduces a new input data structure, which is composed of Delay-Doppler maps (DDM) and all satellite receiver status (SRS) parameters. Then, to overcome the difficulty of handcrafted feature engineering and effectively fusion the information of DDM and SRS, we presented a heterogeneous multimodal deep learning (HMDL) method to retrieve the wind speed according to the heterogeneity of the input data. The proposed model is verified by the performance evaluation of realistic data sets obtained from TDS-1. The new input data structure improves the prediction accuracy at 13.5% to 30.7% on mean absolute error (MAE) at 10.6% to 29.5% on the root mean square error (RMSE). The HMDL improves the prediction accuracy at 7.7% on MAE and 7.1% on RMSE. The whole proposed solution improves the prediction accuracy at 36.3% on MAE and 36.8% on RMSE, comparing with the traditional neural network-based solution. The results clearly show that both the introduction of the new input data structure and HMDL effectively improve the accuracy and robustness of GNSS-R wind speed retrieval.