The last two decades have seen the emergence and steady development of tangible user interfaces. While most of these interfaces are applied for input—with output still on traditional computer screens—the goal of programmable matter and actuated shape-changing materials is to directly use the physical objects for visual or tangible feedback. Advances in material sciences and flexible display technologies are investigated to enable such reconfigurable physical objects. While existing solutions aim for making physical objects more controllable via the digital world, we propose an approach where "holograms" (virtual objects) in a mixed reality environment are augmented with physical variables such as shape, texture or temperature. As such, the support for mobility forms an important contribution of the proposed solution since it enables users to freely move within and across environments. Furthermore, our physically augmented virtual objects can co-exist in a single mixed-media environment with programmable matter and other actuated shape-changing solutions.

We are currently developing a system for simulating reconfigurable physical objects via tangible "holograms". The combination of holograms, created by a Microsoft HoloLens, and a pair of wearable robotic arms with spherical controllers (see Figure 1) enables innovative ways of interacting with tangible holograms and experiencing their numerous physical variables (e.g. shape, texture or temperature).

The proposed tangible hologram (TangHo) solution differs from other systems for simulating programmable matter in the sense that we offer a self-contained and mobile solution that can be used anywhere, within as well as across environments. The automatic repositioning of the spheres based on a user's hand movements and the underlying digital model in combination with the superimposed projected holograms further enables the simulation of arbitrary physical shapes and features. We see various opportunities how our tangible hologram system can be used in new forms of dynamic data physicalisation.

The central idea is to physically augment holograms in a mixed reality environment. In order to do so, the spheres on the two robotic arms are aligned with specific parts of the holograms in the environment based on the tracking of the cameras in the glasses. A first-person view mockup of a user interacting with a graph hologram is shown in Figure 3. The user's hands are positioned on the spheres and by moving the spheres the user can interact with the graph. Of course the user is free to move around in the room and can interact with any other parts of the graph since continuous tracking ensures that the spheres are re-aligned with other parts of the graph in real time. The spheres can not only be used as input devices but the robotic arms may also apply some directional force to the spheres in order to provide some additional computer-generated haptic force feedback based on the underlying digital model. In addition to the haptic feedback via the robot arms, the spheres can provide non-visual supplemental feedback about the underlying digital model or data via physical variables such as shape, texture or temperature as detailed later. Furthermore, other users wearing the same system might join the scene and collaboratively interact with the graph hologram. The interaction of one user may change the underlying digital model and thereby result in visual as well as physical feedback for the other users interacting with the same graph. Note that this represents a classical model-view-control (MVC) design pattern with the peculiarity that views can no longer only be digital but also have a physical manifestation.

A potential use case of tangible holograms is in automotive design. Let us assume that two automotive designers are working on a 1/4 scale physical clay model of a car and wish to explore some refinements and design alternatives at full scale. Both designers are wearing our tangible hologram system and one makes a spatial mapping of the model using the head mounted unit, which is then presented to the user as a hologram—ready for manipulation. The first designer places the hologram in the centre of the room and then, from a short distance back, scales the model up to full size. This hologram is shared with the other designer and both begin to work on the model through natural, touch-based interactions as illustrated in Figure 4. The model can seem to take on the physical properties of clay, allowing it to be worked on and manipulated in the same manner as with the original physical medium but with the added benefit of having full control over properties such as plasticity. After some time one designer decides to work on the model in another part of the studio and so they make a scaled-down copy of the hologram to bring with them. Both designers can now work on the model remotely and see each others manipulations as they happen, interacting, in effect, with two separate tangible holograms sharing the same underlying digital model.

External collaborators: Timothy J. Curtin