Last fall when the Kinect first launched Jeremiah Johnson and I immediately started talking about how much sense it made to use it to make a video for him. Jeremiah records and performs as Nullsleep. His music is typically characterized as 8-bit (since he has a tendency to perform with Gameboys on stage), but, more broadly, his musical goal is to push technology beyond what it’s meant to do to the point where it breaks, producing interesting results and textures. Hence the complex, but also frequently quite glitchy, aesthetic of the images being produced in some of the early Kinect demo videos seemed like a perfect fit.
And from my point of view, obsessed with special effects and trying to figure out how to deploy them as an artistic medium, the prospect of making a video with the Kinect seems quite attractive. The Kinect provides a shortcut to working with some of the processes and techniques that were previously only available to high budget special effects movies: motion capture, 3D photography, integrating actors with digital sets, etc. While a small team working with the Kinect would produce results that are essentially glitchy unpolished versions of these techniques compared to the professional Hollywood version, those glitches would be a terrific match with Jeremiah’s aesthetic.
After a couple of weeks of pondering this prospect, I got an idea for what the video’s story should be. I thought back to the Giant’s Drink sequence from Orson Scott Card’s sci-fi masterpiece, Ender’s Game. In the novel, the Giant’s Drink is one level of a video game used to evaluate the psychological state of young children being trained as generals. It’s an impossible game where a giant offers the child two drinks and tell them to choose. Drinking either option causes the child’s avatar to die a horrible death. It’s a test to find out if they’re suicidal; do they return to the game over-frequently? Does their need to win extend past the rational? The main character eventually defeats the game by breaking the rules.
My concept was to have Jeremiah play both the child and giant. Since we’d be capturing his performance in 3D using the Kinect we could arbitrarily scale and render it within the 3D environment we create without a problem.
Also, I imagined adding an element not present in Card’s story: a disintegrating city through which Jeremiah has to run to reach the giant. Much of Jeremiah’s music is extremely intense and surrounding the atmosphere with a world in the process of tearing itself apart would create a fast moving chaotic visual surface to match: all flying building parts, shaking camera, glinting glass.
After coming up with this idea, I pitched it to Jeremiah and he agreed. He selected a song he’s working on for an upcoming album and I got to work. I started working in two basic directions: doing experiments with the visual techniques I planned to use and shaping the story more precisely.
On this first front, I came up with a rather clever way to do the disintegrating city: downloading 3D models of buildings from Google Warehouse and applying physics to them so that they fall apart. I was pleasantly surprised to find out how easy this was. Using Cinema 4D I loaded up a model, applied some basic dynamics, materials, and lighting, and very quickly had a test of the Empire State Building collapsing within a couple of hours’ work.
Another experiment I’ve been working on is going from captured Kinect data to functional 3D models that can be used in animation. So far I’ve only been experimenting with still images, but I’ve managed to create a mesh from the Kinect point cloud, clean it up with Meshlab so that it’s workable, edit it in Cinema 4D, and animate it in Open Frameworks.
I’m currently working on using skeleton detection with the Kinect to puppet 3D models, on recording depth data in order to play it back later as edited meshes, and a few other techniques.
I’ve also been collecting some visual references for what I want the video to look like in the end, but I’ll write-up a more extensive “mood board” post from those later.
The other big avenue of work I’ve been focusing on has been shaping the story of the video into something I can actually shoot. I started by writing a full ~1000 word treatment that expanded the concept I pitched to Jeremiah into a more complete story. I’m including the full text of the treatment at the bottom of this post.
After writing that, I recruited Liza Singer, a fellow ITP student who also works as a storyboard artist to help me convert the treatment into a visual format. After some discussions, we established a process where I produce rough shot sketches which I then hand over to her (with extensive verbal explanation) and she then converts into legible storyboards.
For example here were my sketches for the first few shots of the video:
And her reinterpretation of the same:
In addition to this visual work, I’ve also been working on planning the timing of the story so that it fits the key moments of the song and working out a production schedule so the video will be completed by the summer date that Jeremiah’s set for release. I’ll write further updates here as the project progresses.
Here’s the treatment, as promised:
Jeremiah runs through a city falling to pieces. He dodges bits of falling buildings, exploding passersby. He gets hit by a car and thrown wildly to the side and into a building. He blinks in and out of reality then brushes himself off, stands up, and runs off again, barely dodging a large chunk of falling debris.
After a few more near misses, Jeremiah descends into a round plaza. As he reaches its center, the plaza begins rumbling. Pavement flies from its edges as it tears itself free from the surrounding street and rises into the sky. Jeremiah struggles to keep his balance as the dish-shaped plaza soars into the sky, a crumbling pillar of earth, layers of plumbing, and a torn-out segment of subway car emerging beneath it.
As the end of a shaky vertical climb brings it level with the surrounding skyscrapers, the saucer tower rumbles to a stop.
Jeremiah wheels around to see a giant approaching. The giant bellies up to the side of the plate. He stares directly at Jeremiah. His features are familiar, almost like a giant version of Jeremiah himself.
Jeremiah stands stock still.
The giant sets down two glasses in front of Jeremiah. They are giant-scale shot glasses, coming up to Jeremiah’s waist. The one on the left contains some kind of green liquid; the one on the right a pink liquid.
“Drink!” the giant bellows.
Jeremiah steps forwards hesitatingly. He looks at each of the glasses. The green one seems to be bubbling violently. Evil-looking smoke is issuing from its top. The pink one is the color of kid’s bubblegum. It coats the sides of the glass thickly.
“Drink!” the giant bellows again. “Or you’ll never get to Fairyland.”
Jeremiah looks up at the giant and then back at the drinks. A bubble pops from the surface of the green liquid, hissing and releasing a curl of smoke.
Jeremiah steps up to the other glass. He looks back up at the giant who is watching him intently. He tilts the huge glass forward and leans over to drink. The pink liquid oozes towards his mouth. When it finally reaches the lip of the glass he drinks a small sludgy sip.
Nothing happens.
Suddenly Jeremiah’s eyes get wide. Something is happening to him. His head starts to swell up like a balloon. He raises his hands to feel it expanding into a huge sphere.
His face stretches out across his head as it continues expanding. Now enormous, his head lifts him off the ground and his feet dangle helplessly. He looks up and sees the giant cackling madly.
With a loud POP his head explodes, fragments flying everywhere, and his body falls into a heap on the ground.
Everything fades to black.
Fade up from black on Jeremiah back where he started, running in the exploding city. Jeremiah runs frantically down a street with buildings collapsing into flying debris all around him. The same car that hit him before swerves up, but this time he’s ready, jumping over its hood at the last second and continuing down the street.
He passes the same sights as in the first pass — the falling Empire State Building, the exploding people — but this time he moves past them more smoothly and after a few familiar flashes he reaches the plate-like plaza again. This time he runs straight to its center and stands bracing himself, solidly keeping his footing as the plaza rumbles and rises into the air.
Again the plaza settles in level with the top of the surrounding skyscrapers and the giant approaches its lip. He slams down two more waist-high shot glasses, one filled with a glowing hazard-orange liquid, the other with what looks like miniature clouds firing off tiny lightning bolts.
“Drink!” the giant bellows.
Jeremiah steps cautiously forward and regards each of the drinks skeptically in turn.
He hesitates.
The giant draws his face in close above the glasses and bellows again: “Drink!” Or you’ll never get to Fairlyand!”
Jeremiah looks between the glasses again and then up at the giant. He starts to back away slowly.
“Drink!” the giant bellows again, bringing his face in even closer.
Jeremiah leans back and then hurls himself forwards, running towards the giant and the glasses. He leaps up onto the rim of one of the glasses and uses it to launch himself up towards the giant’s face. He lands on the giant’s cheek and clings on. The giant roars in anger and tilts his head back causing Jeremiah to lurch forward onto the giant’s lower eyelid.
He looks up at the eye, which swivels over trying to see him. Jeremiah starts digging into the eye with both hands, tearing out handfuls of eyestuff. The giant roars in agony and starts falling backwards. Jeremiah keeps digging as together they fall faster and faster toward the city below.
The giant comes crashing to the ground, shaking the earth and leveling the last of the city’s buildings. A cloud of dust rises around its fallen body.
Once the giant’s body has come to rest, Jeremiah is nowhere to be seen. After a beat he crawls out of the giant’s hollowed out eye hole. He climbs down off of the giant’s face and stumbles back a few paces.
As he watches the giant’s body begins to transform, its clothing turning into grass and rocks, its skin dissolving into mud and dirt. Its limbs become green hills stretching off into the distance. The dust clouds from its fall cohere into white puffy clouds floating away gently into a sky rapidly clearing into a pure blue.
Jeremiah scrambles back up onto the giant and looks around the landscape. He sees a road leading away down the giant’s neck towards its sternum and he heads over to it. At the start of the road is a sign pointing down it labeled “Fairyland.”
Jeremiah starts down the road. Soon the road passes into a dark wood, trees forming up in walls nearly to its edge. Glowing eyes poke out from between the trees. The sky is rapidly reddening into a bloody sunset.
One-by-one creatures begin emerging from the trees. They are grey wolves with human faces, all resembling Jeremiah’s own. They gradually surround Jeremiah in an ever tighter pack. He seems unconcerned, barely noticing them.
Jeremiah and the wolves head off down the road as the sun goes down and the landscape disappears into darkness.