Without a doubt the highlight of day 2 of the CTM 13 Hacklab was a talk by the Grammy Award-winning English singer, composer and songwriter from London Imogen Heap. She was specifically talking about a project she is currently working on involving the use of specially designed gloves with which she controls Ableton Live software through the Max/MSP. The gloves contain a range of sensors including magnetometers (direction), accelerometers (movement), bend sensors, microphones, transmitters, receivers as well as buzzers to give haptic feedback. The video below is of a talk she gave a few months back and although she doesn’t give as much detail of the technology being used as she did at CTM today it will certainly illustrate what is a unique performance much better than I can describe.
As well as using raw sensor data, her systems recognises a series of gestures. Some of the hand gestures she used are called Puppet Hand, Open Hand, Fist, One Finger Point (self explanatory) all probably differentiated primarily by the bend sensors in the fingers of the gloves. What she doesn’t mention in the video above is how they have integrated Microsoft’s Kinect and use it in combination with the gloves. “I wanted to not only have the possibilities in my hands but I also wanted to use the stage.” she explained. In practice this means that depending on where she is on the stage the music or effects available to her will change. For example, further back and the drums could be really loud and full, as she moves to the front of the stage nearer the audience and they lower, enhancing the feeling of intimacy. Same could work with different keys. The Kinect can also read gestures which load different samples and effects and allows specific areas of the stage to be “hot” areas where certain effects will automatically kick in (the choir effect in the video above). A particularly interesting part of this is how the gloves were designed so that she could also compose with them which means that the music and visual aspects of the performance are integral to one another rather than one being a reaction to the other as might usually be the case. The movement is the musical notation. What Imogen and her team seem to have created is the ultimate intuitive interface for performing music (for her a least). “It shouldn’t feel difficult to play, it should feel natural to play, you don’t have to think about it… becomes part of the performance” and she talked about feeling “like I’m actually touching the thing that I just sang” and cutting out the technology “middleman”. All in all it was a great talk and fascinating innovative use of technology to create music. The plan is to release the gloves to the wider music community. They may even become an open source project, here’s hoping. Looking forward to seeing her tour with it in 2013.