Mixed Reality with the XTH Sense

Mixed Reality with the XTH Sense

Last week we organized our first creative lab exclusively focused on using the new XTH Sense™. We gathered at the Integra Lab at the Birmingham Conservatory in UK with researchers in sound and technology to explore some creative applications of the XTH Sense™. Integra Lab is a music interaction design and research group. Their mission is very well aligned with our vision at XTH: to empower people to explore sound and music through easy-to-use technologies and tools. The lab team is an experienced group of artists, designers and developers with expertise in interaction design, digital musical instrument design, haptic interaction, gestural control, immersion and mixed reality, music composition and performance.

Together with Federico Visi (ICCMR, Plymouth University) and Balandino di Donato (Integra Lab), we set out for three intense days of programming, movement and sound explorations in mixed reality.


To create mixed reality environments means to enable virtual and physical worlds to interact with each other. An object in the physical world, like a cup for instance, can be linked to sounds in the virtual realms, so that the cup itself becomes augmented without losing its physical properties, but rather exploring them creatively with the body.

It is a truly interesting approach to body and technology, mixing methods from VR (Virtual Reality) and HCI (Human-Computer Interaction) to create experiences where the real and the virtual co-exist, rather than disrupt or undermine each other.

At XTH, we want to explore innovative ways of expression that merge our bodies with technology, and mixed reality is a fantastic test bed for this.

Federico and Balandino used the XTH Sense™ to create two separate projects where they augmented bodily interaction with everyday objects and virtual sound capabilities. Federico augmented a rain stick (yes, you read it right, a rain stick) and transformed it into a kind of drone-producing ultra-expressive flute interacting with his body movement and muscular tension. To do so, he interfaced the XTH Sense™ with Max/MSP, a popular graphical programming environment. What’s most interesting, Federico managed to make the XTH Sense™ learn a specific set of body gestures using machine learning, a series of algorithms that enable a computer to learn autonomously about the bodily movement of a player. First, Federico explored how to move with the rainstick in order to produce expressive virtual sounds, and then he taught those gestures to the computer. As he played the rainstick, the computer recognized his specific gestures through the XTH Sense™ data, and linked those gestures to complex variation in the sound synthesis.


Balandino, on the other hand, created an interactive tea cup capable of containing virtual water. Crazy, right?  As a result, you can see Balandino shaking the empty cup and yet hear the sound of the water in it! It created not only the illusion of the water inside the empty cup, it also produced different water behaviors; he could throw the cup around and the sound of the water would feel like it was spilling all over the place or he could gently move the cup and you would hear the water slowly moving inside the cup. Technically, Balandino created a software in Pd (Pure Data) that transformed the muscle sounds captured from the XTH Sense™ in the sound of water. He linked the raw data from the XTH Sense™ bioacoustic sensor to the playback of a water sound sample, so that specific amounts of muscular force would trigger the sound playback differently. Then, he used the motion data from the accelerometer and the compass embedded in the XTH Sense™ to link specific gestures to continuous changes of pitch and loudness of the water sound.


As we speak, we are editing a couple of short and sweet videos documenting the results of their project, stay tuned!

Very soon, Balandino and Federico will share their thoughts and secrets with our community through dedicated blog posts. We were pretty inspired by the projects emerging from just three days working together during our inaugural creative lab. Experiencing the XTH Sense™ experimentation with mixed reality projects like these enabled us to deepen our understanding about the potential of the XTH Sense™ for merging body and technology, the real and the virtual in innovative ways. That’s why we love creative labs. They are a great chance to learn how our community uses and experiences the XTH Sense™ through hands-on discovery, rather than surveys or questionnaires, being there in the moment, drinking lots of coffee, exploring movement and creating together. And yes, sometimes someone breaks out a rainstick in the studio for no apparent reason.

Make sure to check more of Balandino’s and Federico’s work here and here!

Submit a Comment

Your email address will not be published. Required fields are marked *

Pin It on Pinterest

Share This