Monthly Archives: October 2014

FlexSense: Interactive, deforming overlay that tracks in 3D – and what it offers us

Microsoft's FlexSense , source: https://www.youtube.com/watch?v=3Jo9ww9cLzg

Microsoft’s FlexSense , source: https://www.youtube.com/watch?v=3Jo9ww9cLzg

Another recent innovation from Microsoft’s R&D labs has presented us with ‘FlexSense’, which you can see demonstrated in a video here. This was brought to my attention in an article posted on The Verge. FlexSense is a thin, transparent, flexible surface that tracks its own deformation in 3D. It can be used to control applications in a variety of ways, such as the masked display of certain areas of an underlying screen and manipulation of mesh surfaces in 3d programs. It owes its versatility to the high degrees of freedom afforded by the sensors embedded in the device, which are shown as having been calibrated and ground-truthed by visual markers.

The examples that Microsoft demonstrate enable you to picture a number of uses, of which I can imagine a rich range of applications to digital heritage across engagement and conservation. Just in its capacity as an ‘overlay’ alone, you could peel away an animated reconstruction of an archaeological site to see the underlying evidence on which it is based, for example. Also layering other information such as uncertainty of specific features of an archaeological interpretation, alternate reconstructions, lighting-only of such an environment (based on accurate calculations), and also annotations of a 3D environment or object.

As a controller, perhaps it could also offer the ability to more naturally virtually manipulate high resolution scans of ’2D’ artefacts such as scrolls, parchments, charters, etc, which would otherwise be far too delicate to handle in a way that could deform it as such (by conservation professionals and members of the public alike). This could enable a much more natural simulation, exploration and interpretation without touching the fragile source material. It will be interesting to see where the FlexSense development goes, and hopefully its offering to these areas will bear fruit, if it hasn’t already.