The computational display of material appearance such as metal, plastic, skin, and so on has attracted attention as an important technology in the fields of product design, education and entertainment. Using projection mapping, this type of a material display can be realized. The images projected by the projection mapping can replace the original surface of an existing normal object in the real world with new material appearance virtually.
On other hand, in this type of display, it is important to change the appearance dynamically according to the motion of object. For example, when observing a specular object, the highlight point is changed according to its arrangement, viewpoint, and light. This dynamic appearance change plays a substantial role in reproducing high-quality user experiences.
However, the conventional projector-camera system is difficult to represent this phenomenon because of its low-speed performance. Also, in the applications using such material display, the interactions where the user can touch and move the object is essentially important. In order to achieve this goal, occlusion-robust and high-speed 3D tracking technology is newly required.
Based on this background, we propose a new marker-based 3D tracking method achieved by only using a monocular monochrome image. Our tracking method achieves both high robustness and high-speed performance even when there is a considerable amount of occlusion. Also we built a computational display of material appearance based on dynamic projection mapping using the proposed tracking technology.
According to the recognized 3D position and pose, the graphics of the virtual material are generated and projected by a high-speed projector. As shown in the figure, the appearance of a white plaster object is changed to the metal-like one and the reproduced specular distribution is consistently changed without delay. Our experiment confirmed that it was possible to successfully merge the real-world geometry and the material appearance consistently in a dynamic environment. Furthermore, dynamic changes as a consequence of user manipulation of the object could drastically enrich the experience compared with observation of a static object.
Although we show only the example of the metal display in this demonstration, different material can also be realized. In addition, the different materials can be arranged on the surface spatially. We also extend this work to the color material display . And we can make the marker invisible by using the infrared ink.
References
- Yoshihiro Watanabe, Toshiyuki Kato, Masatoshi Ishikawa: Extended Dot Cluster Marker for High-speed 3D Tracking in Dynamic Projection Mapping, IEEE International Symposium on Mixed and Augmented Reality (ISMAR2017) (Nantes, 2017.11.11), pp. 52-61, 2017.