Dynamic projection mapping (DPM) ideally needs to cover an entire scene by projection in a markerless way. A simple but effective solution is to obtain the depth map of the scene with high accuracy at a high speed, apply millisecond-order markerless tracking, separate the scene into a UV-textured area and other areas, render the augmented appearance for each area, and project the image by mixing them with low latency. We call this approach the depth-aware DPM. However, it is difficult for a system with an off-the-shelf consumer sensor to perform such a DPM because it cannot realize depth capturing and 3D tracking at a speed that meets the requirements of the DPM.
In this research, we report a new DPM system that combines a 947-fps 24-bit high-speed RGB projector, 500-fps high-speed infrared (IR) camera, and newly developed 2,880-fps 8-bit IR high-speed projector. In addition, we have developed a high-speed model-based rigid-body tracking method that uses a depth map captured by a high-speed structured-light method. Tracking can be performed at a 0.4-ms processing time by leveraging a small inter-frame motion. Our system demonstrates flexible, depth-aware DPM on the entire scene with a latency of approximately 8 ms.
The demonstration video will be soon released. We also plan to make the system compact by introducing the newly developed high-speed RGB+IR projector.
References
- Sora Hisaichi, Kiwamu Sumino, Kunihiro Ueda, Hidenori Kasebe, Tohru Yamashita, Takeshi Yuasa, Uwe Lippmann, Petra Aswendt, Roland Höfling, and Yoshihiro Watanabe. 2021. Depth-Aware Dynamic Projection Mapping using High-speed RGB and IR Projectors. In SIGGRAPH Asia 2021 Emerging Technologies (SA ’21 Emerging Technologies:), December 14-17, 2021. Tokyo, Japan