Mobile3DScanner: An Online 3D Scanner for High-quality Object Reconstruction with a Mobile Device

Part 1 Ïà¹ØÊÂÇé

ÏÖÔÚÖ÷ÒªµÄ¾²Ì¬ÎïÌåÖØÐÞÒªÁì¿ÉÒÔ·ÖΪÁ½Àࣺ»ùÓÚRGBDÏà»úµÄÈýάɨÃèÖØÐÞÓë»ùÓÚͼÏñµÄ¶àÊÓͼÖØÐÞ¡£¡£¡£¡£¡£¡£¡£

1.1 »ùÓÚRGBDÏà»úµÄÈýάɨÃèÖØÐÞ

´ËÀàÒªÁìÒÔKinectFusion[3]Ϊ´ú±í£¬£¬£¬£¬£¬£¬¶ÔÊäÈëµÄÉî¶ÈͼʹÓÃICPÇó½âÏà»úλ×Ë£¬£¬£¬£¬£¬£¬²¢½«Éî¶ÈͼÈںϳÉÒ»¸öÈ«¾ÖµÄTSDFÄ£×Ó£¬£¬£¬£¬£¬£¬ÀàËƵÄÊÂÇéÉÐÓÐInfiniTAM[2]ÒÔ¼°BundleFusion[1]£¬£¬£¬£¬£¬£¬µ«ÓÉÓÚÅÌËãÖØƯºóÒÔ¼°ÄÚ´æÕ¼ÓõÄÔµ¹ÊÔ­ÓÉ£¬£¬£¬£¬£¬£¬ËüÃÇÖ»ÄÜÔÚPC¶ËÔËÐС£¡£¡£¡£¡£¡£¡£±ðµÄ£¬£¬£¬£¬£¬£¬ÉÐÓÐ[8][9][10]µÈ½áÊÊÓû§ÊÖ²Ù×÷µÄ½»»¥Ê½É¨ÃèÖØÐÞÒªÁìÒÔ¼°»ùÓڷǸÕÐÔ×¢²áµÄÎïÌåɨÃèÖØÐÞ[11]£¬£¬£¬£¬£¬£¬ÕâÀàÒªÁì´ó¶àÊÊÓÃÓÚСÎïÌåÖØÐÞ¡£¡£¡£¡£¡£¡£¡£

1.2 »ùÓÚͼÏñµÄ¶àÊÓͼÖØÐÞ

´ËÀàÒªÁìÖÂÁ¦ÓÚÔÚÒƶ¯×°±¸ÉÏʵÏÖ»ùÓÚµ¥Ä¿RGBÊÓƵµÄÎïÌåɨÃèÖØÐÞ£¬£¬£¬£¬£¬£¬ËûÃÇʹÓöàÊÓͼÁ¢ÌåÆ¥ÅäµÄÒªÁì»Ö¸´³öµ¥Ö¡µÄÉî¶È²¢½«Éî¶ÈÈںϳÉTSDFÄ£×Ó[12][13]»òÕßSurfelÄ£×Ó[14][15]¡£¡£¡£¡£¡£¡£¡£×ÜÌå¶øÑÔ£¬£¬£¬£¬£¬£¬ÕâÀàÒªÁìÖØÐÞ³öµÄÄ£×ÓÔÚÄ£×Ó¾«¶ÈÒÔ¼°ÍêÕû¶ÈÉÏÓë»ùÓÚRGBDÏà»úµÄÖØÐÞÒªÁìÏà±ÈÉÐÓÐÒ»¶¨²î±ð¡£¡£¡£¡£¡£¡£¡£

Part 2 ÒªÁìÐÎò

2.1 ϵͳ¸ÅÊö

Èçͼ1Ëùʾ£¬£¬£¬£¬£¬£¬¹þ¹þ(haha)ÌåÓýÖØÐÞϵͳҪÇóÓû§Å䱸һ¸ö´øÓÐRGBDÉãÏñÍ·µÄÒƶ¯ÅÌËã×°±¸£¬£¬£¬£¬£¬£¬²¢½«´ýɨÃèµÄÎïÌå°²ÅÅÔÚÒ»¸öˮƽÃæÉÏ¡£¡£¡£¡£¡£¡£¡£Óû§×îÏÈɨÃèºó£¬£¬£¬£¬£¬£¬ÏµÍ³½«×Ô¶¯Ö§½â³öÎïÌåÇøÓò²¢ÊµÊ±Çó½âÏà»úλ×Ë£¬£¬£¬£¬£¬£¬Éî¶È´«¸ÐÆ÷»ñÈ¡µ½µÄÉî¶ÈÊý¾Ý½«±»ÈںϳÉTSDFÄ£×Ó²¢äÖȾ¹©Óû§×÷ʵʱԤÀÀ¡£¡£¡£¡£¡£¡£¡£É¨Ã迢ʺ󣬣¬£¬£¬£¬£¬ÏµÍ³½«¶ÔËùÓеÄÒªº¦Ö¡¾ÙÐÐÈ«¾ÖBA²¢ÍŽá¶àÊÓͼÁ¢ÌåÆ¥ÅäÒªÁìÀ´ÓÅ»¯Òªº¦Ö¡µÄÉî¶È£¬£¬£¬£¬£¬£¬×îÖÕÈںϳöµÄTSDFÄ£×Ó¾­Óɲ´ËÉÖØÐÞ[16]¡¢Shape-from-ShadingÒÔ¼°ÎÆÀíÌùͼºó£¬£¬£¬£¬£¬£¬»ñµÃ×îÖÕ´øÎÆÀíµÄÖØÐÞÄ£×Ó¡£¡£¡£¡£¡£¡£¡£

ÎÒÃÇÓà Semantic Field À´ÊµÏÖ¶ÔÈËÁ³ÌØÕ÷Ò»Á¬ÇÒ¸ßϸÁ£¶È¿É¿ØµÄ±à¼­¡£¡£¡£¡£¡£¡£¡£¶Ô»°¹¦Ð§ÔòÓÉ Language EncoderºÍTalkÄ£¿£¿£¿£¿£¿£¿£¿£¿éÀ´ÊµÏÖ¡£¡£¡£¡£¡£¡£¡£½ÓÏÂÀ´ÎÒÃÇÚ¹Ê͸÷¸öÄ£¿£¿£¿£¿£¿£¿£¿£¿éµÄÊÂÇéÔ­ÀíºÍЧ¹û¡£¡£¡£¡£¡£¡£¡£

2.2 ICPÓëIMUËÉñîºÏ

Òƶ¯Æ½Ì¨ËãÁ¦ÓÐÏÞ£¬£¬£¬£¬£¬£¬ÎÒÃÇÌá³ö½«ICPÓëIMU¾ÙÐÐËÉñîºÏÀ´ÌáÉýÏà»úλ×ËÇó½âµÄ³°ôÐÔÓ뾫¶È¡£¡£¡£¡£¡£¡£¡£ÎÒÃÇÊ×ÏÈʹÓóõʼÁ½Ö¡µÄICPЧ¹û¶ÔIMUÄ£¿£¿£¿£¿£¿£¿£¿£¿é¾ÙÐгõʼ»¯£¬£¬£¬£¬£¬£¬IMU³õʼ»¯Íê³Éºó¼´¿ÉÒÀ¾ÝÊäÈëµÄIMUÊý¾ÝΪĿ½ñÖ¡Ìṩλ×ËÕ¹Íû£¬£¬£¬£¬£¬£¬ÎÒÃÇÔÙ½«IMUµÄÕ¹ÍûÖµ¼ÓÈëICPÄÜÁ¿ÏîÀ´Çó½âÄ¿½ñÖ¡µÄÏà»úλ×Ë£¬£¬£¬£¬£¬£¬Ä¿½ñ֡λ×ËÇó½âÍê³Éºó£¬£¬£¬£¬£¬£¬ÆäЧ¹û½«×÷ΪԼÊøÀ´½øÒ»²½ÓÅ»¯IMUµÄ״̬Á¿[17]¡£¡£¡£¡£¡£¡£¡£Ô­±¾µÄICPÄÜÁ¿Ïî[4]˼Á¿ÁËÁ½Ö¡Ö®¼äµÄÑÕÉ«²îºÍÉî¶È²î£º

ÔÚ´Ë»ù´¡ÉÏÎÒÃǼÓÈëÁËIMUµÄÐýתԼÊøÒÔ¼°ÖØÁ¦Ô¼Êø£º

¹¹½¨×îÖÕµÄÄÜÁ¿·½³Ì

ͼ2չʾÁËÎÒÃÇÒªÁìµÄÓÐÓÃÐÔ¡£¡£¡£¡£¡£¡£¡£

ͼ2.(a)Open3D[4] ICP. (b)ÒýÈëIMU. (c)(d)¼ÓÈëLBAÒÔ¼°Loop Closure.

2.3 ×Ô˳Ӧµ÷ÀíÌåËصÄTSDF

Òƶ¯Æ½Ì¨ÉÏÄÚ´æÓÐÏÞ£¬£¬£¬£¬£¬£¬±ØÐèÔ¼ÊøTSDFÄ£×ӵĴóС£¡£¡£¡£¡£¡£¡£¬£¬£¬£¬£¬£¬ÎÒÃÇÌá³öÒ»ÖÖ×Ô˳Ӧµ÷ÀíÌåËصÄTSDFÒªÁ죬£¬£¬£¬£¬£¬µ±ÆäÄÚ´æÕ¼ÓÃÁ¿µÖ´ïãÐֵʱ£¬£¬£¬£¬£¬£¬ÏµÍ³½«×Ô¶¯½¨ÉèÒ»¸öеÄTSDF£¬£¬£¬£¬£¬£¬ÆäÌåËؾÞϸΪ¾ÉÌåËصÄ1.5±¶£¬£¬£¬£¬£¬£¬ÐÂÌåËصķûºÅ¾àÀëÒÔ¼°È¨ÖØÎÞÐèÖØÐÂÅÌË㣬£¬£¬£¬£¬£¬¿Éͨ¹ý¾ÉÌåËØÈýÏßÐÔ²åÖµ¸ßЧ»ñÈ¡¡£¡£¡£¡£¡£¡£¡£Õâ¸öÕ½ÂÔ¿É°ü¹ÜTSDFÄ£×ÓµÄÄÚ´æÕ¼Óò»»áÁè¼ÝÓû§É趨µÄãÐÖµ£¬£¬£¬£¬£¬£¬´Ó¶øÔÚÒƶ¯×°±¸ÉÏÄÜÖ§³Ö´óÎïÌåµÄɨÃèÖØÐÞ¡£¡£¡£¡£¡£¡£¡£

2.4 ÍŽáÉî¶ÈÏÈÑéµÄ¶àÊÓͼÁ¢ÌåÆ¥Åä

ÏûºÄ¼¶µÄÉî¶È´«¸ÐÆ÷ÀýÈçdToF£¬£¬£¬£¬£¬£¬Æä»ñÈ¡µÄÉî¶È±£´æ²î±ðˮƽµÄ¹ýʧÒÔ¼°¹ýƽ»¬ÎÊÌ⣬£¬£¬£¬£¬£¬ÎÒÃÇÌá³öÒ»ÖÖÍŽáÉî¶ÈÏÈÑéµÄSGMÒªÁ죬£¬£¬£¬£¬£¬Ê¹ÓÃÊÓ¾õÐÅÏ¢À´ÌáÉýÉî¶ÈͼµÄ¾«¶È£¬£¬£¬£¬£¬£¬ÆäЧ¹ûÈçͼ3Ëùʾ¡£¡£¡£¡£¡£¡£¡£

ͼ3.(a)Òªº¦Ö¡. (b)dToFµ¥Ö¡Éî¶ÈÒÔ¼°ÈںϺóµÄÄ£×Ó. (c)SGM[6]µ¥Ö¡Éî¶È. (d)ÍŽáÉî¶ÈÏÈÑéµÄSGM¼°ÆäÄ£×Ó.

2.5 ¸ßЧµÄShape-from-Shading

Shape-from-Shading¿ÉÒÔÌáÉýÖØÐÞÄ£×ӵļ¸ºÎϸ½Ú£¬£¬£¬£¬£¬£¬µ«ÏÖÔÚÒÑÓеÄʵÏּƻ®ÎÞÂÛÊÇÅÌËãÖØƯºóÕÕ¾ÉÄÚ´æÕ¼Óö¼Ô¶ÎÞ·¨ÔÚÒƶ¯Æ½Ì¨ÉÏʵÑé¡£¡£¡£¡£¡£¡£¡£²î±ðÓÚÏÖÓмƻ®£¬£¬£¬£¬£¬£¬ÎÒÃÇÖ±½ÓÔÚÖØÐ޺õÄÄ£×ÓÉÏÓÅ»¯Èý½ÇÃæƬµÄ·¨ÏòÁ¿£¬£¬£¬£¬£¬£¬ÔÙÍŽáÒªÁì[5]¸üÐÂÄ£×Ó¼«µã¡£¡£¡£¡£¡£¡£¡£ÒÔͼ4Öеġ°David¡±ÎªÀý£¬£¬£¬£¬£¬£¬¹þ¹þ(haha)ÌåÓýÒªÁìÔÚiPad Pro2020 CPUÉϺÄʱ3.24Ã룬£¬£¬£¬£¬£¬ÓÅ»¯ºóÄ£×Ó¾«¶ÈRMSE/MAEΪ3.12mm/2.473mm¡£¡£¡£¡£¡£¡£¡£¶øIntrinsic3D[7]ÔÚPC¶ËIntel Core i7 7700K CPUÉϺÄʱ81·ÖÖÓ»ñµÃÂÔÓŵÄЧ¹û£¬£¬£¬£¬£¬£¬RMSE/MAEΪ2.89mmÓë2.16mm¡£¡£¡£¡£¡£¡£¡£

ͼ4.(a)³õʼģ×Ó. (b)SFSÓÅ»¯ºóµÄÄ£×Ó

Part 3 ʵÑéЧ¹û

3.1 Ä£×Ó¾«¶È±ÈÕÕ

ÎÒÃÇÓÃiPad Pro2020¼ÖÆÁËËÄ×é²âÊÔÊý¾ÝÀ´±ÈÕÕ¹þ¹þ(haha)ÌåÓý¼Æ»®ÓëÏÖÓеļƻ®ÔÚÄ£×Ó¾«¶ÈÉϵIJî±ð£¬£¬£¬£¬£¬£¬ground truthÓÉÉÌҵɨÃèÒÇ»ñÈ¡¡£¡£¡£¡£¡£¡£¡£±í¸ñ1ÒÔ¼°Í¼5ÏÔʾ¹þ¹þ(haha)ÌåÓýÒªÁìÔÚÄ£×Ó¾«¶ÈÒÔ¼°ÍêÕû¶ÈÉ϶¼È¡µÃÁË×îÓÅЧ¹û¡£¡£¡£¡£¡£¡£¡£

±í¸ñ1.Ä£×Ó¾«¶È±ÈÕÕ£¨RMSE/MAE£©

ͼ5.ÓëÏÖÓÐSOTAÒªÁìµÄÄ£×Ó±ÈÕÕ.

(b)Open3D[4].(c)KinectFusion[3].(d)InfiniTAM[2].(e)BundleFusion[1].(f)3D Scanner App.(g)¹þ¹þ(haha)ÌåÓýÒªÁì.(h)GT.

3.2 ºÄʱͳ¼Æ

±í¸ñ2ͳ¼ÆÁËËã·¨¸÷¸ö°ì·¨ÔÚiPad Pro2020ÉϵĺÄʱ£¬£¬£¬£¬£¬£¬¹þ¹þ(haha)ÌåÓýÒªÁìÔÚ´óÎïÌå¡°worker¡±ÉÏÒ²µÖ´ïÁËʵʱ¡£¡£¡£¡£¡£¡£¡£

±í¸ñ2£ºË㷨ģ¿£¿£¿£¿£¿£¿£¿£¿éºÄʱͳ¼Æ

3.3 ¸ü¶àЧ¹û

Reference:

[1] Dai, M. Nie?ner, M. Zollho ?fer, S. Izadi, and C. Theobalt. BundleFusion: Real-time globally consistent 3D reconstruction using on-the-fly surface re-integration. ACM Transactions on Graphics, 36(4):1, 2017.

[2] O. Kahler, V. Prisacariu, C. Ren, X. Sun, P. Torr, and D. Murray. Very high frame rate volumetric integration of depth images on mobile devices. IEEE Transactions on Visualization and Computer Graphics, 21(11):1¨C1, 2015.

[3] R. A. Newcombe, S. Izadi, O. Hilliges, D. Molyneaux, and A. W. Fitzgibbon. KinectFusion: Real-time dense surface mapping and tracking. In IEEE International Symposium on Mixed and Augmented Reality, 2011.

[4] Q. Zhou, J. Park, and V. Koltun. Open3D: A modern library for 3D data processing. arXiv:1801.09847, 2018.

[5] H. Zhao, K. Su, C. Li, B. Zhang, L. Yang, N. Lei, X. Wang, S. J. Gortler, and X. Gu. Mesh parametrization driven by unit normal flow. Computer Graphics Forum, 39(1):34¨C49, 2020.

[6] H. Hirschmu ?ller. Accurate and efficient stereo processing by semi-global matching and mutual information. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 2, pp. 807¨C814, 2005.

[7] R. Maier, K. Kim, D. Cremers, J. Kautz, and M. Nie?ner. Intrinsic3D: High-quality 3D reconstruction by joint appearance and geometry optimization with spatially-varying lighting. In IEEE International Conference on Computer Vision, 2017.

[8] S. Rusinkiewicz, O. Hall-Holt, and M. Levoy. Real-time 3D model acquisition. ACM Transactions on Graphics, 21(3):438¨C446, 2002.

[9] D. Tzionas and J. Gall. 3D object reconstruction from hand-object interactions. In IEEE International Conference on Computer Vision, pp. 729¨C737, 2015.

[10] T. Weise, B. Leibe, and L. Van Gool. Accurate and robust registration for in-hand modeling. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 1¨C8, 2008.

[11] J. Xu, W. Xu, Y. Yang, Z. Deng, and H. Bao. Online global non-rigid registration for 3D object reconstruction using consumer-level depth cameras. 37(7):1¨C12, 2018.

[12] P. Ondru ?s¡¦ka, P. Kohli, and S. Izadi. MobileFusion: Real-time volumetric surface reconstruction and dense tracking on mobile phones. IEEE Trans- actions on Visualization and Computer Graphics, 21(11):1251¨C1258, 2015.

[13] V. Pradeep, C. Rhemann, S. Izadi, C. Zach, M. Bleyer, and S. Bathiche. MonoFusion: Real-time 3D reconstruction of small scenes with a single web camera. In IEEE International Symposium on Mixed and Augmented Reality, 2013.

[14] P. Tanskanen, K. Kolev, L. Meier, F. Camposeco, O. Saurer, and M. Polle- feys. Live metric 3D reconstruction on mobile phones. In IEEE Interna- tional Conference on Computer Vision, pp. 65¨C72, 2013.

[15] K. Kolev, P. Tanskanen, P. Speciale, and M. Pollefeys. Turning mobile phones into 3D scanners. In IEEE Conference on Computer Vision and Pattern Recognition, 2014.

[16] M. Kazhdan. Poisson surface reconstruction. In Eurographics Symposium on Geometry Processing, 2006.

[17] T. Qin, P. Li, and S. Shen. Vins-mono: A robust and versatile monocular visual-inertial state estimator.