real-time performance-driven animation

I've been thinking a bit about how motion capture and animation could be used in a musical performance context - I haven't seen this done (yet) but it seems like it could be really cool. It wouldn't necessarily require a 'rubber suit with sparkly bits glued on' either - it could be as simple as a couple of data gloves transmitting positional information wirelessly to a receiver that was used to control a real-time animation system. You could combine a number of 'pre-visualized' animation sequences with dynamically generated animation that was controlled through the data gloves, for example. The animation would need to be projected, either on a flat screen the way things are done currently, or on a curved translucent screen with the performer at its center. Ideally the screen would be some form of low-cost polymer LCD that would generate the images directly, but you could probably use 2 digital projectors that were projecting beams that intersected at a 90 degree angle on the display screen, and pre-distort the images they were projecting to account for the curvature of the cylindrical screen. I think the 'nightclub' sequence in Pattern Recognition could feature an animation depicting this kind of performance.

I've started hacking together a short story about a struggling rock group in the not too distant future, as well. Here's my first cut at a description of one of their gigs that uses this kind of technology:


The stage was dark, featureless, lightless. On the floor, a tiny point of light sparkled into being with a quiet [crack], lazily rising up, tracing a fading helix, a spark rising without a flame. As it rose out of view, all that was left behind was a thin gossamer strand of fading light. A second spark followed, weaving a helical counterpart upwards; The gossamer double helix began to slowly rotate, with a swelling sound of wind. Two other sparks crackled into life, closer together, weaving a complex shape - when they too had risen out of sight, they were followed by hundreds of sparks, rapidly tracing their way upwards, weaving the outline of a gossamer figure, crouched, fading into darkness. Thousands of sparks rose. She was clearer now, faintly painted by the rising light, her long hair stirred into life by the quiet, imagined wind. She started to radiate a light of her own, swirling blues and greens, a caftan of light, her hair gossamer black floated around her. The wind became rhythmic, breathlike, with a deeper sound beating slow as a heartbeat. She opened her eyes and suddently sunlight flooded the cylinder traced by the double helix, the helix itself began to pulse with light, revolving and widening, sheets of light forming gradients that swirled in time with the breath, their width modulated by the heartbeat. She reached upwards and shafts of orange and blue flew upwards accompanied by swirling sound, ethereally floating in the sonic space between the wind and the heartbeat. The heartbeat sped up, became polyrhythmic, an infectious rhtyhm and she started to sway, each movement catalysing a cascade of colors, rich and vibrant, full of images. Her breathing modulated the wind, became whispered words, breathily evocative. She started singing to the music, the melody modulating the colours of her caftan, the imagistic lyrics shaping the images that swirled upwards and around her, translucent imaginings mixed with the swirling colours. Other sounds joined her singing: a harmony voice, drums, a soaring guitar voice that morphed into a wailing sax line before splintering into a glorious cascade of colour. As the music grew louder, more and more sparks swirled up from the floor, swirling around her - she raised her eyes and looked upwards as she was engulfed in virtual flames, became flame herself and dissipated as she rose with the sparks, leaving only a translucent, fading afterimage.

All that's left is the retina burn and the sonic shock, just an afterimage...

One of the coolest things about animation is that you can do ANYTHING. But you need to speak a language that people can understand in order to convey any kind of meaning. In film, the standard ways of conveying a sense of 'looking back' is done through flashbacks, and I think that needs to be part of the animation - move away from a linear story narrative into a more complex time sense. There needs to be some back-story to flash-back to. Perhaps use muted washes to convey the misty sense of memory. These could be used in the satori sequence instead of just using random computation representations. I definitely want to get a sense of a fevered, electronically enhanced mind working away during this sequence as well; not sure how compatible the two things are. The real catalyst has to be emotionally evocative. I still like the ending: a memory of nature that is getting pixelated. Need to make it more personal, more powerful, more to the point.

I think one of the ways to convey the sense of 'realization' is to have a profile shot of the protagonist, eyes closed, head tilted up towards the ceiling, mouth open, tense, zoom out until the reflected image of the protagonist is just a black dot, and then hold that for a couple of beats, then slam back into a closeup of the eyes as they are opened wide, replacing the dot with the pupil and then zoom in like mad into the pupil. Perhaps echo the 'glowing white' pupil image here momentarily.

I've also been thinking about playing around with the whole eye imagery early on, perhaps using old TV test patterns (sweeping / color wheel) in place of the iris.

The trick in all of this is to keep it general enough so that it could mean many different things to many different people, but still keep it personal enough so that it has emotional resonance. I have no idea what I'm really getting into at this point, except that it's WAY over my head. Just need to start sketching and riffing on some of these ideas, find out what works and be open minded enough to let this thing breathe it's own imagery out of thin air. I consistently find that the less 'rational' I am about creating something, the better.

Trying out some ideas as drawings

Night Cityscape



This is just a rough sketch using mainly the Microsoft Acrylic 'palette knife' strokes.

Same drawing with motion and radial blurs applied (done in Photoshop). Not sure which one I like better:

>

"Burning bright like life..."



I was hoping that the simplicity of drawing edge lit faces would be both evocative and efficient. Well, it succeeds on both counts, but it's WAY too dark and sinister. NOT where I want this to go. Time to rethink things.

Sketch / Outline

Very rough sketch of some ideas for the story line for the Pattern Recognition video and the visuals used to convey it. Still a lot of work to do to refine this and get pacing, etc. but it's a start.

Visual themes: backgrounds are speed blurred: buildings are vertically speed blurred, street level is horizontally speed blurred. Superimposed on the blurred images of buildings and cars is a pencil sketch or high pass filtered outline of reality. People have the ghost-like presence of time-lapse photography, moving ethereally like partially transpartent shadows with blurred
details and outlines.

Intro:
perspective shot looking down the neck of a bass guitar.
The 4 strings dissolve into a speed-blurred vision of a highway at night, 2 strings glowing red, 2 white.

Zoom down the highway. As it dives into a tunnel, continue zoom into a shot of a man looking into a store-front window at night. Zoom in on the main character's reflection in a window of a shop that sells robotic parts and android accessories. His face is in darkness except for peripheral highlighting, edge lit jaw line, brow. Continuing the zoom, from the rear, we seem him slot a .5"x2" black cartridge into a slot near the base of his skull, and we zoom in front of the cartridge, racing along fibre optic pathways. As the cartridge engages, our point of view flips to his point of view. Zoom in on the reflection of shaded eyes as the pupils and sclera suddently glow white. (possibly from the reflected headlights of a passing car??). The gleam suffuses the character until his reflection is glowing like a lightbulb, as diffuse and semi-transparent as the greyish ghostlike people that stream past, but with an intense light radiating from him.

He/we turn down a back alley. A discarded android replicant / Asimo robot is lying broken against the left wall. On the right side of the alley is a fire-escape with a bare lightbulb suspended above the top landing. A vertical sign that reads "VIVID" in shades of neon purple and white identifies the door at the top as leading to a club of some form.

We ascend the stairs to the open door and the view is chaotic - a band is playing pattern recognition, shadowy movement swirls through the crowded dance floor, hexagonal tiling pulsing to the music. The scene zooms in on a light source and floods to white.

Sunlight streaming against a white wall. Backing up shows the shadows of the window, with a rear-view of a man (the protagonist??) sitting on the edge of a bed (slotting in a cartridge??) Zooming in, the view flips to his perspective.

The panes of the window multiply like cells, become the projections of a rapid cascade of image sections, speed blurred puzzle pieces that rearrange and snap into flashbacks of the preceding scenes, or into abstract geometric patterns, or into ??? Data readouts are superimposed on the display, as well as outlines in white - e.g. Bato's visual display in Ghost in the shell 2, or the terminator's visual display. Scenes of dna spiral helixes, neurons and glial cells, computer lights flashing, 7 segment number sequences or binary data flows against gradient backgrounds.

The eureka moment is a memory image. (Perhaps some photograph of an Aibo remote controlled robot dog in someone's arms??) The speed blurs snap out of existence. The sky is cloudy with sun rays streaming through. There are flowers and birds. We watch as it digitizes and pixelates into larger and larger blocks and dissolves to black.

Ideas and Inspiration...

Ideas and Inspiration can sometimes be quite annoying. They tend to come when you least need them, when you're overloaded with a billion other 'more important' things, when your brain is at a fever pitch whirling away in 10 different directions at once. Kinda makes sense, I suppose. So what happened? Well, I guess it started when Microsoft finally released a Beta of Expression v.4 (which they have code named 'Acrylic'). It's a vector graphics program that was created by a small company called Creature House a decade or so ago, which Microsoft wisely purchased in 2003. It's a totally sweet beautiful drawing application that I've used as my primary drawing application for a number of years. I used it to make all of the artwork for my first animation, a (still incomplete) 3 minute rock video for a song called "Secret Agent Girlfriend" that I did back in 2002.

Expression has a much more powerful approach to vector drawing than Flash does - it has B-splines with Nodes that act as anchor points for the curves you draw. Multiple views of a drawing can be defined by shifting the position of the nodes in the drawing. Expression can interpolate between the views, providing the basis for animating the sequence of views, and can export these animation sequences as a Flash .swf file. You have to really 'dumb down' the drawing to just vectors and gradients, however, losing much of the beauty that comes from using the natural media and photo integration techniques that Expression is so good at. (There's also the, uh, 'minor problem' that Acrylic no longer supports exporting animations to .swf format. More on that in another post...)

So what does this all have to do with ideas and inspiration? Well, Acrylic has added some pretty cool new features that I wanted to explore. It was the trigger that made me finally purchase a graphics tablet - I'd been holding off for a while to see what happened with the Tablet PC market, but the price premium over a normal laptop is still quite steep and the graphics tablet digitizer capabilities aren't as sophisticated as those offered by Wacom's Intuos line of graphics tablets. So I picked up an Intuos 3 4x5 tablet. Here's the first picture I've done with it:

I've been learning how to adjust tonal values in drawings to make light sources appear like they are glowing, and this opened me up to some ideas along this line of thought. I've also been writing a lot of music lately (e.g. see the sidebar for blogs on a number of these songs), including a song called 'Pattern Recognition'. This is one of the best songs I've written to date, and has some pretty good visual imagery in it. Other inspirations include the iPod commercials with the silhouetted dancers (now THAT's a nice way to make animations simpler!)

the graphic artist Jordi Labanda,

photographer André Gallant's Dreamscapes,

a sculpter named Giacometti,

a Google image search for the word "luminous",

Carlos Castenada's "luminous eggs", Edvard Munch's "The Scream", and probably a bunch of other stuff I'm not even aware of.

So, to make my next plunge into animation seem somewhat less pointless, I decided to blog the creation process. I'm hoping that I'll be able to catch the wave of 'Sparkle' technology as I slowly put together the story board, create multi-view strokes in Acrylic, learn about XAML, Avalon and Flash, refine my animation technique, learn more about visual screen writing, etc. It'll probably take a couple of years to do the 2:48 of animation, since i'll only be able to squeeze in a few hours per week. And I'd like to create some tutorials along the way so that I don't forget what I've learned and so that anyone else interested can also benefit from the effort.

Dreaming in Technicolor