China Aero Geophysical Survey and Remote Sensing Center for Natural ResourcesHost
地质出版社Publish
Advanced Search
Previous Article Next Article
2023 Vol. 35, No. 1
Article Contents
Article navigation > Remote Sensing for Natural Resources > 2023 > 35(1): 1-14
Next Article Previous Article

HU Jianwen, WANG Zeping, HU Pei. 2023. A review of pansharpening methods based on deep learning. Remote Sensing for Natural Resources, 35(1): 1-14. doi: 10.6046/zrzyyg.2021433
Citation: HU Jianwen, WANG Zeping, HU Pei. 2023. A review of pansharpening methods based on deep learning. Remote Sensing for Natural Resources, 35(1): 1-14. doi: 10.6046/zrzyyg.2021433
shu

A review of pansharpening methods based on deep learning

  • School of Electrical and Information Engineering, Changsha University of Science and Technology, Changsha 410114, China

Received Date:  13 December 2021
Revised Date:  15 March 2023
Publish Date:  20 March 2023
    Abstract
  • With the fast development and wide application of remote sensing technology, remote sensing images with higher quality are needed. However, it is difficult to directly acquire high-resolution, multispectral remote sensing images. To obtain high-quality images by integrating the information from different imaging sensors, pansharpening technology emerged. Pansharpening is an effective method used to obtain multispectral images with high spatial resolution. Many scholars have studied this method and obtained fruitful achievements. In recent years, deep learning theory has developed rapidly and has been widely applied in pansharpening. This study aims to systematically introduce the progress in pansharpening and promote its development. To this end, this study first introduced the traditional, classical pansharpening methods, followed by commonly used remote sensing satellites. Then, this study elaborated on the pansharpening methods based on deep learning from the perspective of supervised learning, unsupervised learning, and semi-supervised learning. After that, it described and analyzed loss functions. To demonstrate the superiority of the pansharpening methods based on deep learning and analyze the effects of loss functions, this study conducted remote sensing image fusion experiments. Finally, this study presented the future prospects of the pansharpening methods based on deep learning.
    • FullText(HTML)
  • 加载中
    • References
  • [1] 肖亮, 刘鹏飞, 李恒. 多源空—谱遥感图像融合方法进展与挑战[J]. 中国图象图形学, 2020, 25(5):851-863.

    Google Scholar

    [2] Xiao L, Liu P F, Li H. Progress and challenges in the fusion of multisource spatial-spectral remote sensing images[J]. Journal of Image and Graphics, 2020, 25(5):851-863.

    Google Scholar

    [3] Ghassemian H. A review of remote sensing image fusion methods[J]. Information Fusion, 2016, 32:75-89.

    Google Scholar

    [4] 张丽霞, 曾广平, 宣兆成. 多源图像融合方法的研究综述[J]. 计算机工程与科学, 2021:1-14.

    Google Scholar

    [5] Zhang L X, Zeng G P, Xuan Z C. A survey of fusion methods for multi-source image[J]. Computer Engineering and Science, 2021:1-14.

    Google Scholar

    [6] 杨丽萍, 马孟, 谢巍, 等. 干旱区Landsat8全色与多光谱数据融合算法评价[J]. 国土资源遥感, 2019, 31(4):11-19.doi:10.6046/gtzyyg.2019.04.02.

    Google Scholar

    [7] Yang L P, Ma M, Xie W, et al. Fusion algorithm evaluation of Landsat8 panchromatic and multispetral images in arid regions[J]. Remote Sensing for Land and Resources, 2019, 31(4):11-19.doi:10.6046/gtzyyg.2019.04.02.

    Google Scholar

    [8] Weiss M, Jacob F, Duveiller G. Remote sensing for agricultural applications:A meta-review[J]. Remote Sensing of Environment, 2020, 236:111402-111420.

    Google Scholar

    [9] 刘万军, 高健康, 曲海成, 等. 多尺度特征增强的遥感图像舰船目标检测[J]. 自然资源遥感, 2021, 33(3):97-106.doi:10.6046/zrzyyg.20200372.

    Google Scholar

    [10] Liu W J, Gao J K, Qu H C, et al. Ship detection based on multi-scale feature enhancement of remote sensing images[J]. Remote Sensing for Natural Resources, 2021, 33(3):97-106.doi:10.6046/zrzyyg.20200372.

    Google Scholar

    [11] 王琳, 李迅, 包云轩, 等. 遥感技术在交通气象灾害监测中的应用进展[J]. 国土资源遥感, 2018, 30(4):1-7.doi:10.6046/gtzyyg.2018.04.01.

    Google Scholar

    [12] Wang L, Li X, Bao Y X, et al. Research progress of remote sensing application on transportation meteorological disasters[J]. Remote Sensing and for Land and Resources, 2018, 30(4):1-7.doi:10.6046/gtzyyg.2018.04.01.

    Google Scholar

    [13] 王洪斌, 肖嵩, 曲家慧, 等. 基于多分支CNN高光谱与全色影像融合处理[J]. 光学学报, 2021, 41(7):55-63.

    Google Scholar

    [14] Wang H B, Xiao S, Qu J H, et al. Pansharpening based on multi-branch CNN[J]. Acta Optica Sinica, 2021, 41(7):55-63.

    Google Scholar

    [15] 张良培, 沈焕锋. 遥感数据融合的进展与前瞻[J]. 遥感学报, 2016, 20(5):1050-1061.

    Google Scholar

    [16] Zhang L P, Shen H F. Progress and future of remote sensing data fusion[J]. Journal of Remote Sensing, 2016, 20(5):1050-1061.

    Google Scholar

    [17] 李树涛, 李聪妤, 康旭东. 多源遥感图像融合发展现状与未来展望[J]. 遥感学报, 2021, 25(1):148-166.

    Google Scholar

    [18] Li S T, Li C Y, Kang X D. Development status and future prospects of multi-source remote sensing image fusion[J]. National Remote Sensing Bulletin, 2021, 25(1):148-166.

    Google Scholar

    [19] 张立福, 彭明媛, 孙雪剑, 等. 遥感数据融合研究进展与文献定量分析(1992—2018)[J]. 遥感学报, 2019, 23(4):603-619.

    Google Scholar

    [20] Zhang L F, Peng M Y, Sun X J, et al. Progress and bibliometric analysis of remote sensing data fusion methods (1992—2018)[J]. Journal of Remote Sensing, 2019, 23(4):603-619.

    Google Scholar

    [21] Meng X, Shen H, Li H, et al. Review of the pansharpening methods for remote sensing images based on the idea of meta-analysis:Practical discussion and challenges[J]. Information Fusion, 2019, 46:102-113.

    Google Scholar

    [22] Javan F D, Samadzadegan F, Mehravar S, et al. A review of image fusion techniques for pan-sharpening of high-resolution satellite imagery[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2021, 171:101-117.

    Google Scholar

    [23] Vivone G, Mura M D, Garzelli A, et al. A new benchmark based on recent advances in multispectral pansharpening:Revisiting pansharpening with classical and emerging pansharpening methods[J]. IEEE Geoscience and Remote Sensing Magazine, 2020, 9(1):53-81.

    Google Scholar

    [24] Huang W, Xiao L, Wei Z, et al. A new pan-sharpening method with deep neural networks[J]. IEEE Geoscience and Remote Sensing Letters, 2015, 12(5):1037-1041.

    Google Scholar

    [25] Masi G, Cozzolino D, Verdoliva L, et al. Pansharpening by convolutional neural networks[J]. Remote Sensing, 2016, 8(7):594-615.

    Google Scholar

    [26] Shahdoosti H R, Ghassemian H. Combining the spectral PCA and spatial PCA fusion methods by an optimal filter[J]. Information Fusion, 2016, 27:150-160.

    Google Scholar

    [27] Laben C A, Brower B V. Process for enhancing the spatial resolution of multispectral imagery using pan-sharpening:U.S.,US09/069232[P]. 2000-01-04.

    Google Scholar

    [28] Ghahremani M, Ghassemian H. Nonlinear IHS:A promising method for pan-sharpening[J]. IEEE Geoscience and Remote Sensing Letters, 2016, 13(11):1606-1610.

    Google Scholar

    [29] Garzelli A, Nencini F, Capobianco L. Optimal MMSE pan sharpening of very high resolution multispectral images[J]. IEEE Transactions on Geoscience and Remote Sensing, 2007, 46(1):228-236.

    Google Scholar

    [30] Aiazzi B, Alparone L, Baronti S, et al. MTF-tailored multiscale fusion of high-resolution MS and pan imagery[J]. Photogrammetric Engineering and Remote Sensing, 2006, 72(5):591-596.

    Google Scholar

    [31] Vivone G, Restaino R, Chanussot J. Full scale regression-based injection coefficients for panchromatic sharpening[J]. IEEE Transactions on Image Processing, 2018, 27(7):3418-3431.

    Google Scholar

    [32] Khan M M, Chanussot J, Condat L, et al. Indusion:Fusion of multispectral and panchromatic images using the induction scaling technique[J]. IEEE Geoscience and Remote Sensing Letters, 2008, 5(1):98-102.

    Google Scholar

    [33] Dong L, Yang Q, Wu H, et al. High quality multi-spectral and panchromatic image fusion technologies based on curvelet transform[J]. Neurocomputing, 2015, 159:268-274.

    Google Scholar

    [34] El-Mezouar M C, Kpalma K, Taleb N, et al. A pan-sharpening based on the non-subsampled contourlet transform:Application to WorldView-2 imagery[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2014, 7(5):1806-1815.

    Google Scholar

    [35] 曹义亲, 杨世超, 谢舒慧. 基于NSST的PCNN-SR卫星遥感图像融合方法[J]. 航天控制, 2020, 38(2):44-50.

    Google Scholar

    [36] Cao Y Q, Yang S C, Xie S H. Satellite remote sensing image fusion method based on NSST and PCNN-SR[J]. Aerospce Control, 2020, 38(2):44-50.

    Google Scholar

    [37] 吴一全, 王志来. 混沌蜂群优化的NSST域多光谱与全色图像融合[J]. 遥感学报, 2017, 21(4):549-557.

    Google Scholar

    [38] Wu Y Q, Wang Z L. Multispectral and panchromatic image fusion using chaotic Bee Colony optimization in NSST domain[J]. Journal of Remote Sensing, 2017, 21(4):549-557.

    Google Scholar

    [39] Li S, Yang B. A new pan-sharpening method using a compressed sensing technique[J]. IEEE Transactions on Geoscience and Remote Sensing, 2010, 49(2):738-746.

    Google Scholar

    [40] Yin H. PAN-guided cross-resolution projection for local adaptive sparse representation-based pansharpening[J]. IEEE Transactions on Geoscience and Remote Sensing, 2019, 57(7):4938-4950.

    Google Scholar

    [41] 李成轶, 田淑芳. 基于字典学习的遥感影像超分辨率融合方法[J]. 国土资源遥感, 2017, 29(1):50-56.doi:10.6046/gtzyyg.2017.01.08.

    Google Scholar

    [42] Li C Y, Tian S F. Super-resolution fusion method for remote sensing image based on dictionary learning[J]. Remote Sensing and for Land and Resources, 2017, 29(1):50-56.doi:10.6046/gtzyyg.2017.01.08.

    Google Scholar

    [43] Yin H. Sparse representation based pansharpening with details injection model[J]. Signal Processing, 2015, 113:218-227.

    Google Scholar

    [44] Li S, Yin H, Fang L. Remote sensing image fusion via sparse representations over learned dictionaries[J]. IEEE Transactions on Geoscience and Remote Sensing, 2013, 51(9):4779-4789.

    Google Scholar

    [45] Scarpa G, Vitale S, Cozzolino D. Target-adaptive CNN-based pansharpening[J]. IEEE Transactions on Geoscience and Remote Sensing, 2018, 56(9),5443-5457.

    Google Scholar

    [46] Wei Y, Yuan Q, Shen H, et al. Boosting the accuracy of multispectral image pansharpening by learning a deep residual network[J]. IEEE Geoscience and Remote Sensing Letters, 2017, 14(10),1795-1799.

    Google Scholar

    [47] Yang J, Fu X, Hu Y, et al. PanNet:A deep network architecture for pan-sharpening[C]// Proceedings of the IEEE International Conference on Computer Vision.IEEE,Venice,Italy, 2017:5449-5457.

    Google Scholar

    [48] Zhang H, Ma J. GTP-PNet:A residual learning network based on gradient transformation prior for pansharpening[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2021, 172:223-239.

    Google Scholar

    [49] Yang Y, Tu W, Huang S, et al. PCDRN:Progressive cascade deep residual network for pansharpening[J]. Remote Sensing, 2020, 12(4):676.

    Google Scholar

    [50] Huang W, Feng J, Wang H, et al. A new architecture of densely connected convolutional networks for pan-sharpening[J]. ISPRS International Journal of Geo-Information, 2020, 9(4):242.

    Google Scholar

    [51] Peng J, Liu L, Wang J, et al. PSMD-Net:A novel pan-sharpening method based on a multiscale dense network[J]. IEEE Transactions on Geoscience and Remote Sensing, 2020, 59(6):4957-4971.

    Google Scholar

    [52] Luo S, Zhou S, Qi Y. CSAFNET:Channel similarity attention fusion network for multispectral pansharpening[J]. IEEE Geoscience and Remote Sensing Letters, 2020.

    Google Scholar

    [53] 孔爱玲, 张承明, 李峰, 等. 基于知识引导的遥感影像融合方法[J]. 自然资源遥感, 2022, 34(2):47-55.doi:10.6046/zrzyyg.2021179.

    Google Scholar

    [54] Kong A L, Zhang C M, Li F, et al. Knowledge-based remote sensing imagery fusion method[J]. Remote Sensing for Natural Resources, 2022, 34(2):47-55.doi:10.6046/zrzyyg.2021179.

    Google Scholar

    [55] Jiang M, Shen H, Li J, et al. A differential information residual convolutional neural network for pansharpening[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2020, 163:257-271.

    Google Scholar

    [56] Lei D, Chen H, Zhang L, et al. NLRNet:An efficient nonlocal attention resnet for pansharpening[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60:5401113.

    Google Scholar

    [57] Shao Z, Cai J. Remote sensing image fusion with deep convolutional neural network[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2018, 11(5),1656-1669.

    Google Scholar

    [58] Liu X, Liu Q, Wang Y. Remote sensing image fusion based on two-stream fusion network[J]. Information Fusion, 2020,55,1-15.

    Google Scholar

    [59] Fu S, Meng W, Jeon G, et al. Two-path network with feedback connections for pan-sharpening in remote sensing[J]. Remote Sensing, 2020, 12(10),1674.

    Google Scholar

    [60] He L, Xi D, Li J, et al. A spectral-aware convolutional neural network for pansharpening[J]. Applied Sciences, 2020, 10(17),5809.

    Google Scholar

    [61] Zhang Y, Liu C, Sun M, et al. Pan-sharpening using an efficient bidirectional pyramid network[J]. IEEE Transactions on Geoscience and Remote Sensing, 2019, 57(8),5549-5563.

    Google Scholar

    [62] Fang S, Wang X, Zhang J, et al. Pan-sharpening based on parallel pyramid convolutional neural network[C]// IEEE International Conference on Image Processing (ICIP), 2020:453-457.

    Google Scholar

    [63] 方帅, 方赛华, 姚宏亮. 基于深度金字塔网络的Pan-Sharpening算法[J]. 计算机辅助设计与图形学学报, 2019, 31(10):1831-1837.

    Google Scholar

    [64] Fang S, Fang S H, Yao H L. Pan-sharpening based on a deep pyramid network[J]. Journal of Computer-Aided Design and Computer Graphics, 2019, 31(10):1831-1837.

    Google Scholar

    [65] Ronneberger O, Fischer P, Brox T. U-Net:Convolutional networks for biomedical image segmentation[C]// International Conference on Medical Image Computing and Computer-Assisted Intervention.Springer,Cham, 2015:234-241.

    Google Scholar

    [66] Yao W, Zeng Z, Lian C, et al. Pixel-wise regression using U-Net and its application on pansharpening[J]. Neurocomputing, 2018, 312:364-371.

    Google Scholar

    [67] Wang W, Zhou Z, Liu H, et al. MSDRN:Pansharpening of multispectral images via multi-scale deep residual network[J]. Remote Sensing, 2021, 13(6):1200.

    Google Scholar

    [68] Lai Z, Chen L, Jeon G, et al. Real-time and effective pan-sharpening for remote sensing using multi-scale fusion network[J]. Journal of Real-Time Image Processing, 2021:1-17.

    Google Scholar

    [69] Benzenati T, Kallel A, Kessentini Y. Two stages pan-sharpening details injection approach based on very deep residual networks[J]. IEEE Transactions on Geoscience and Remote Sensing, 2020, 59(6),4984-4992.

    Google Scholar

    [70] He K, Zhang X, Ren S, et al. Deep residual learning for image reco-gnition[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016:770-778.

    Google Scholar

    [71] Hu J, Du C, Fan S. Two-stage pansharpening based on multi-level detail injection network[J]. IEEE Access, 2020, 8:156442-156455.

    Google Scholar

    [72] Zhang L, Zhang J, Ma J, et al. SC-PNN:Saliency cascade convolutional neural network for pansharpening[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021:1-19.

    Google Scholar

    [73] Li W, Liang X, Dong M. MDECNN:A multiscale perception dense encoding convolutional neural network for multispectral pan-sharpening[J]. Remote Sensing, 2021, 13(3):535.

    Google Scholar

    [74] Yuan Q, Wei Y, Meng X, et al. A multiscale and multidepth convolutional neural network for remote sensing imagery pan-sharpening[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2018, 11(3):978-989.

    Google Scholar

    [75] Hu J, Hu P, Kang X, et al. Pan-sharpening via multiscale dynamic convolutional neural network[J]. IEEE Transactions on Geoscience and Remote Sensing, 2020, 59(3):2231-2244.

    Google Scholar

    [76] Guo Y, Ye F, Gong H. Learning an efficient convolution neural network for pansharpening[J]. Algorithms, 2019, 12(1):16.

    Google Scholar

    [77] Hu J, Hu P, Wang Z P, et al. Spatial dynamic selection network for remote-sensing image fusion[J]. IEEE Geoscience and Remote Sensing Letters, 2022, 19:8013205.

    Google Scholar

    [78] Hu J, He Z, Wu J. Deep self-learning network for adaptive pansharpening[J]. Remote Sensing, 2019, 11(20):2395.

    Google Scholar

    [79] Liu J, Feng Y, Zhou C, et al. PWnet:An adaptive weigh network for the fusion of panchromatic and multispectral images[J]. Remote Sensing, 2020, 12(17):2804.

    Google Scholar

    [80] Deng L J, Vivone G, Jin C, et al. Detail injection-based deep convolutional neural networks for pansharpening[J]. IEEE Transactions on Geoscience and Remote Sensing, 2020:1-16.

    Google Scholar

    [81] Goodfellow I, Pouget-Abadie J, Mirza M, et al. Generative adversarial networks[J]. Communications of the ACM, 2020, 63(11):139-144.

    Google Scholar

    [82] Ma J, Yu W, Chen C, et al. Pan-GAN:An unsupervised pan-sharpening method for remote sensing image fusion[J]. Information Fusion, 2020, 62:110-120.

    Google Scholar

    [83] Zhou C, Zhang J, Liu J, et al. PercepPan:Towards unsupervised pan-sharpening based on perceptual loss[J]. Remote Sensing, 2020, 12(14):2318.

    Google Scholar

    [84] Xiong Z, Guo Q, Liu M, et al. Pan-sharpening based on convolutional neural network by using the loss function with no-reference[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2020, 14:897-906.

    Google Scholar

    [85] Luo S, Zhou S, Feng Y, et al. Pansharpening via unsupervised convolutional neural networks[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2020, 13:4295-4310.

    Google Scholar

    [86] 杜晨光, 胡建文, 胡佩. 半监督卷积神经网络遥感图像融合[J], 电子测量与仪器学报, 2021, 35(6):63-70.

    Google Scholar

    [87] Du C G, Hu J W, Hu P. Semi-supervised convolutional neural network remote sensing image fusion[J]. Journal of Electronic Measurement and Instrumentation, 2021, 35(6):63-70.

    Google Scholar

    [88] Vitale S, Scarpa G. A detail-preserving cross-scale learning strategy for CNN-based pansharpening[J]. Remote Sensing, 2020, 12(3):348.

    Google Scholar

    [89] 黄珊珊, 江倩, 金鑫, 等. 结合双胞胎结构与生成对抗网络的半监督遥感图像融合[J]. 计算机辅助设计与图形学学报, 2021, 33(1):92-105.

    Google Scholar

    [90] Huang S S, Jiang Q, Jin X, et al. Semi-supervised remote sensing image fusion method combining siamese structure with generative adversarial networks[J]. Journal of Computer-Aided Design and Computer Graphics, 2021, 33(1):92-105.

    Google Scholar

    [91] Arjovsky M, Chintala S, Bottou L. Wasserstein generative adversarial networks[C]// International Conference on Machine Learning.PMLR, 2017:214-223.

    Google Scholar

    [92] Mirza M, Osindero S. Conditional generative adversarial nets[J]. Computer Science, 2014:2672-2680.

    Google Scholar

    [93] Liu X, Deng C, Zhao B, et al. Feature-level loss for multispectral pan-sharpening with machine learning[C]// IEEE International Geoscience and Remote Sensing Symposium(IGARSS).IEEE, 2018:8062-8065.

    Google Scholar

    [94] Xu H, Ma J, Shao Z, et al. SDPNet:A deep network for pan-sharpening with enhanced information representation[J]. IEEE Transactions on Geoscience and Remote Sensing, 2020, 59(5):4120-4134.

    Google Scholar

    [95] Choi J S, Kim Y, Kim M. S3:A spectral-spatial structure loss for pan-sharpening networks[J]. IEEE Geoscience and Remote Sensing Letters, 2019, 17(5):829-833.

    Google Scholar

    [96] Eghbalian S, Ghassemian H. Multi spectral image fusion by deep convolutional neural network and new spectral loss function[J]. International Journal of Remote Sensing, 2018, 39(12):3983-4002.

    Google Scholar

    [97] Johnson J, Alahi A, Li F F. Perceptual losses for real-time style transfer and super-resolution[C]// European Conference on Computer Vision.Springer,Cham, 2016:694-711.

    Google Scholar

    [98] Bello J L G, Seo S, Kim M. Pan-sharpening with color-aware perceptual loss and guided re-colorization[C]// IEEE International Conference on Image Processing (ICIP).IEEE, 2020:908-912.

    Google Scholar

    [99] Vivone G, Alparone L, Chanussot J, et al. A critical comparison among pansharpening algorithms[J]. IEEE Transactions on Geoscience and Remote Sensing, 2014, 53(5):2565-2586.

    Google Scholar

    [100] Alparone L, Aiazzi B, Baronti S, et al. Multispectral and panchromatic data fusion assessment without reference[J]. Photogrammetric Engineering and Remote Sensing, 2008, 74(2):193-200.

    Google Scholar

    • Relative Articles
  • Cited by
  • Supplements
    • Proportional views
  • 加载中
通讯作者: 陈斌, bchen63@163.com
  • 1. 

    沈阳化工大学材料科学与工程学院 沈阳 110142

  1. 本站搜索
  2. 百度学术搜索
  3. 万方数据库搜索
  4. CNKI搜索
Export Citation
PDF
XML

Article Metrics

Article views(3498) PDF downloads(2008) Cited by(0)

Access History

Other Articles By Authors

Catalog

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    版权所有：中国地质调查局发展研究中心 copyright @2020