Development of a Semantic Segmentation System for Dynamic Occlusion Handling in Mixed Reality for Landscape Simulation

Development of a Semantic Segmentation System for Dynamic Occlusion Handling in Mixed Reality for Landscape Simulation

Kido, Daiki; Fukuda, Tomohiro; Yabuki, Nobuyoshi

Article:

The use of mixed reality (MR) for landscape simulation has attracted attentionrecently. MR can produce a realistic landscape simulation by merging athree-dimensional computer graphic (3DCG) model of a new building on a realspace. One challenge with MR that remains to be tackled is occlusion. Properlyhandling occlusion is important for the understanding of the spatial relationshipbetween physical and virtual objects. When the occlusion targets move or thetarget's shape changes, depth-based methods using a special camera have beenapplied for dynamic occlusion handling. However, these methods have alimitation of the distance to obtain depth information and are unsuitable foroutdoor landscape simulation. This study focuses on a dynamic occlusionhandling method for MR-based landscape simulation. We developed a real-timesemantic segmentation system to perform dynamic occlusion handling. Wedesigned this system for use in mobile devices with client-server communicationfor real-time semantic segmentation processing in mobile devices. Additionally,we used a normal monocular camera for practice use.

The use of mixed reality (MR) for landscape simulation has attracted attentionrecently. MR can produce a realistic landscape simulation by merging athree-dimensional computer graphic (3DCG) model of a new building on a realspace. One challenge with MR that remains to be tackled is occlusion. Properlyhandling occlusion is important for the understanding of the spatial relationshipbetween physical and virtual objects. When the occlusion targets move or thetarget's shape changes, depth-based methods using a special camera have beenapplied for dynamic occlusion handling. However, these methods have alimitation of the distance to obtain depth information and are unsuitable foroutdoor landscape simulation. This study focuses on a dynamic occlusionhandling method for MR-based landscape simulation. We developed a real-timesemantic segmentation system to perform dynamic occlusion handling. Wedesigned this system for use in mobile devices with client-server communicationfor real-time semantic segmentation processing in mobile devices. Additionally,we used a normal monocular camera for practice use.

Palavras-chave:

DOI: 10.5151/proceedings-ecaadesigradi2019_117

Referências bibliográficas

• [1]

Como citar:

Kido, Daiki; Fukuda, Tomohiro; Yabuki, Nobuyoshi; "Development of a Semantic Segmentation System for Dynamic Occlusion Handling in Mixed Reality for Landscape Simulation", p-641-648. In: Proceedings of 37 eCAADe and XXIII SIGraDi Joint Conference, “Architecture in the Age of the 4Th Industrial Revolution”, Porto 2019, Sousa, José Pedro; Henriques, Gonçalo Castro; Xavier, João Pedro (eds.). São Paulo: Blucher, 2019.
ISSN 23186968, DOI 10.5151/proceedings-ecaadesigradi2019_117