Thursday, January 12, 2006

real-time performance-driven animation

I've been thinking a bit about how motion capture and animation could be used in a musical performance context - I haven't seen this done (yet) but it seems like it could be really cool. It wouldn't necessarily require a 'rubber suit with sparkly bits glued on' either - it could be as simple as a couple of data gloves transmitting positional information wirelessly to a receiver that was used to control a real-time animation system. You could combine a number of 'pre-visualized' animation sequences with dynamically generated animation that was controlled through the data gloves, for example. The animation would need to be projected, either on a flat screen the way things are done currently, or on a curved translucent screen with the performer at its center. Ideally the screen would be some form of low-cost polymer LCD that would generate the images directly, but you could probably use 2 digital projectors that were projecting beams that intersected at a 90 degree angle on the display screen, and pre-distort the images they were projecting to account for the curvature of the cylindrical screen. I think the 'nightclub' sequence in Pattern Recognition could feature an animation depicting this kind of performance.

I've started hacking together a short story about a struggling rock group in the not too distant future, as well. Here's my first cut at a description of one of their gigs that uses this kind of technology:


The stage was dark, featureless, lightless. On the floor, a tiny point of light sparkled into being with a quiet [crack], lazily rising up, tracing a fading helix, a spark rising without a flame. As it rose out of view, all that was left behind was a thin gossamer strand of fading light. A second spark followed, weaving a helical counterpart upwards; The gossamer double helix began to slowly rotate, with a swelling sound of wind. Two other sparks crackled into life, closer together, weaving a complex shape - when they too had risen out of sight, they were followed by hundreds of sparks, rapidly tracing their way upwards, weaving the outline of a gossamer figure, crouched, fading into darkness. Thousands of sparks rose. She was clearer now, faintly painted by the rising light, her long hair stirred into life by the quiet, imagined wind. She started to radiate a light of her own, swirling blues and greens, a caftan of light, her hair gossamer black floated around her. The wind became rhythmic, breathlike, with a deeper sound beating slow as a heartbeat. She opened her eyes and suddently sunlight flooded the cylinder traced by the double helix, the helix itself began to pulse with light, revolving and widening, sheets of light forming gradients that swirled in time with the breath, their width modulated by the heartbeat. She reached upwards and shafts of orange and blue flew upwards accompanied by swirling sound, ethereally floating in the sonic space between the wind and the heartbeat. The heartbeat sped up, became polyrhythmic, an infectious rhtyhm and she started to sway, each movement catalysing a cascade of colors, rich and vibrant, full of images. Her breathing modulated the wind, became whispered words, breathily evocative. She started singing to the music, the melody modulating the colours of her caftan, the imagistic lyrics shaping the images that swirled upwards and around her, translucent imaginings mixed with the swirling colours. Other sounds joined her singing: a harmony voice, drums, a soaring guitar voice that morphed into a wailing sax line before splintering into a glorious cascade of colour. As the music grew louder, more and more sparks swirled up from the floor, swirling around her - she raised her eyes and looked upwards as she was engulfed in virtual flames, became flame herself and dissipated as she rose with the sparks, leaving only a translucent, fading afterimage.